2024-04-21T23:43:29.643 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T23:43:29.643 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} --owner scheduled_teuthology@teuthology --verbose -- /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/orig.config.yaml 2024-04-21T23:43:29.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T23:43:29.805 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667643' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 676 sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b sleep_before_teardown: 0 subset: 46/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAIRsLEZF8bUPVTPVURDYHHkQajUyECvFcGO8+MK15+cEMCjqcMSMoNorGFFIKrZ125LRNZIk77xmy0O8nPbDOc= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+jWwHED4WcUBBHwUMDyaN+19fOeKAWQnoRKUP1yb3sPfDkS4GH9jpuO7e433M2WCIhvA+84e3CeLT26FMk9j0= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkEwQaAvdwi2zXiaRZXkS72nLZUnrLN4pyFgb8JROek53d1ipyt2oEPNZ+ZGVMCu0eVw7OzFF3uWSVNgfNgrgQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T23:43:29.805 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa; will attempt to use it 2024-04-21T23:43:29.806 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks 2024-04-21T23:43:29.806 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T23:43:29.808 INFO:teuthology.task.internal:Checking packages... 2024-04-21T23:43:29.832 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '820a2f8ced66152ed3d765c31444b8076ea28c6b' 2024-04-21T23:43:29.832 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T23:43:29.832 INFO:teuthology.packaging:ref: None 2024-04-21T23:43:29.832 INFO:teuthology.packaging:tag: None 2024-04-21T23:43:29.832 INFO:teuthology.packaging:branch: squid 2024-04-21T23:43:29.832 INFO:teuthology.packaging:sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:43:29.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-21T23:43:30.124 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2407.g430e09df 2024-04-21T23:43:30.126 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T23:43:30.134 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T23:43:30.134 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T23:43:30.145 INFO:teuthology.task.internal:Saving configuration 2024-04-21T23:43:30.175 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T23:43:30.183 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:43:30.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 23:38:10.806871', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAIRsLEZF8bUPVTPVURDYHHkQajUyECvFcGO8+MK15+cEMCjqcMSMoNorGFFIKrZ125LRNZIk77xmy0O8nPbDOc='} 2024-04-21T23:43:30.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 23:38:10.807731', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+jWwHED4WcUBBHwUMDyaN+19fOeKAWQnoRKUP1yb3sPfDkS4GH9jpuO7e433M2WCIhvA+84e3CeLT26FMk9j0='} 2024-04-21T23:43:30.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 23:38:10.805526', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkEwQaAvdwi2zXiaRZXkS72nLZUnrLN4pyFgb8JROek53d1ipyt2oEPNZ+ZGVMCu0eVw7OzFF3uWSVNgfNgrgQ='} 2024-04-21T23:43:30.246 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T23:43:30.253 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T23:43:30.253 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T23:43:30.254 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T23:43:30.254 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T23:43:30.360 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efcb25c7160>, signals=[15]) 2024-04-21T23:43:30.361 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T23:43:30.369 INFO:teuthology.task.internal:Opening connections... 2024-04-21T23:43:30.369 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-21T23:43:30.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:43:30.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-21T23:43:30.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:43:30.536 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-21T23:43:30.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:43:30.609 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T23:43:30.618 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-21T23:43:30.637 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-21T23:43:30.638 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:43:30.696 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:43:30.697 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:43:30.697 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:43:30.697 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:43:30.697 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-21T23:43:30.729 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-21T23:43:30.746 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-21T23:43:30.746 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:43:30.803 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:43:30.804 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:43:30.804 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-21T23:43:30.820 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-21T23:43:30.837 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-21T23:43:30.837 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T23:43:30.895 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-21T23:43:30.896 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T23:43:30.896 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T23:43:30.896 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T23:43:30.896 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-21T23:43:30.920 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T23:43:30.932 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T23:43:30.940 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T23:43:30.941 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:43:30.943 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:43:30.945 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-21T23:43:30.961 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T23:43:30.970 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T23:43:30.970 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:43:31.000 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:43:31.003 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-21T23:43:31.287 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T23:43:31.350 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T23:43:31.368 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-21T23:43:31.369 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T23:43:31.369 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T23:43:31.369 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:43:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:31.387 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-04-21T23:43:31.445 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-437.el9.x86_64 2024-04-21T23:43:31.445 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-437.el9.x86_64 2024-04-21T23:43:31.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-04-21T23:43:33.833 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-21T23:43:35.870 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-21T23:43:40.768 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.2 MB 00:01 2024-04-21T23:43:42.177 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-04-21T23:43:43.196 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-21T23:43:48.354 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-21T23:43:50.319 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T23:43:50.319 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T23:43:50.408 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-21T23:43:50.409 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-04-21T23:43:50.409 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-21T23:43:50.520 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2024-04-21T23:43:51.042 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 11:43:48 PM UTC. 2024-04-21T23:43:51.166 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T23:43:51.198 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-21T23:43:51.199 INFO:teuthology.orchestra.run.smithi038.stderr:Operation aborted. 2024-04-21T23:43:51.264 INFO:teuthology.orchestra.run.smithi038.stdout:Is this ok [y/N]: 2024-04-21T23:43:51.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2024-04-21T23:43:51.776 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 11:43:48 PM UTC. 2024-04-21T23:43:51.896 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 255 k 2024-04-21T23:43:51.898 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-04-21T23:43:51.898 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-21T23:43:52.165 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.2 MB/s | 255 kB 00:00 2024-04-21T23:43:52.165 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:43:52.165 INFO:teuthology.orchestra.run.smithi038.stdout:Total 953 kB/s | 255 kB 00:00 2024-04-21T23:43:52.184 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-21T23:43:52.374 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-21T23:43:52.374 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-21T23:43:52.379 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-21T23:43:52.379 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-21T23:43:52.521 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-21T23:43:52.607 INFO:teuthology.orchestra.run.smithi038.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T23:43:52.607 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:43:53.280 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:43:53.280 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalled: 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:43:53.944 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-21T23:43:54.148 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T23:43:54.195 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T23:43:54.196 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T23:43:54.196 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T23:43:54.197 DEBUG:teuthology.parallel:result is None 2024-04-21T23:43:54.197 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T23:43:54.206 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T23:43:54.207 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:43:54.209 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:43:54.211 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T23:43:54.229 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T23:43:54.240 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T23:43:54.254 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T23:43:54.254 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:43:54.264 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:43:54.269 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T23:43:54.370 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T23:43:54.377 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T23:43:54.377 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:43:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:54.395 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T23:43:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:54.411 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:43:54.413 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:43:54.438 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T23:43:54.468 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.473 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.482 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.488 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.508 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.513 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T23:43:54.514 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T23:43:54.522 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T23:43:54.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:43:54.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:43:54.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T23:43:54.586 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T23:43:54.594 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T23:43:54.595 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:43:54.597 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:43:54.623 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T23:43:54.642 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:54.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:54.740 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:54.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:54.860 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:43:54.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:43:54.925 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:54.965 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:55.019 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:55.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:55.142 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:43:55.142 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:43:55.206 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:55.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T23:43:55.307 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:55.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T23:43:55.433 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:43:55.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T23:43:55.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-21T23:43:55.502 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-21T23:43:55.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-21T23:43:55.540 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:43:55.542 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:43:55.579 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:43:56.068 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T23:43:56.077 INFO:teuthology.task.internal:Starting timer... 2024-04-21T23:43:56.077 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T23:43:56.167 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T23:43:56.176 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-21T23:43:56.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-21T23:43:56.232 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-21T23:43:56.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-21T23:43:56.300 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-21T23:43:56.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-21T23:43:56.364 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-21T23:43:56.367 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T23:43:56.367 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-21T23:43:56.390 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-21T23:43:56.390 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-21T23:43:56.419 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-21T23:43:56.419 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-21T23:43:56.496 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-21T23:43:56.496 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-21T23:43:56.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:43:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:56.522 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:43:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:56.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:43:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:43:56.575 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T23:43:56.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:43:56.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:43:56.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-21T23:43:56.656 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T23:43:56.665 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-21T23:43:56.671 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-21T23:43:56.795 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T23:43:56.809 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T23:43:56.810 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T23:47:59.125 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-21T23:47:59.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-21T23:47:59.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:47:59.205 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-21T23:47:59.287 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-21T23:47:59.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T23:47:59.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:47:59.360 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-21T23:47:59.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-21T23:47:59.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-21T23:47:59.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T23:47:59.527 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-21T23:47:59.610 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-21T23:47:59.611 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T23:47:59.624 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T23:47:59.624 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:47:59.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:47:59.628 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:47:59.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:47:59.630 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T23:47:59.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:47:59.656 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:47:59.659 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:47:59.669 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:47:59.674 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:47:59.679 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T23:47:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-21T23:47:59.692 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T23:47:59.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-21T23:47:59.701 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-21T23:47:59.711 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-21T23:47:59.714 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:47:59.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-21T23:47:59.725 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:47:59.726 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:47:59.727 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-21T23:47:59.740 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:47:59.740 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T23:47:59.752 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T23:47:59.787 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.790 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.794 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.804 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:47:59.809 INFO:teuthology.run_tasks:Running task install... 2024-04-21T23:47:59.818 DEBUG:teuthology.task.install:project ceph 2024-04-21T23:47:59.818 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}} 2024-04-21T23:47:59.818 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'} 2024-04-21T23:47:59.818 INFO:teuthology.task.install:Using flavor: default 2024-04-21T23:47:59.823 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T23:47:59.823 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T23:47:59.824 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:47:59.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:47:59.826 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:47:59.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:47:59.827 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T23:47:59.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:48:00.048 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:48:00.048 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:48:00.058 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:48:00.058 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:48:00.065 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T23:48:00.065 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T23:48:00.161 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:48:00.161 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:48:00.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:48:00.172 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:48:00.172 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:48:00.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:48:00.177 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T23:48:00.177 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:48:00.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T23:48:00.189 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:48:00.189 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:48:00.197 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:48:00.198 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:48:00.201 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T23:48:00.201 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T23:48:00.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:48:00.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:48:00.265 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T23:48:00.331 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:48:00.335 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:48:00.338 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T23:48:00.396 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-21T23:48:00.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-21T23:48:00.398 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-21T23:48:00.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-21T23:48:00.400 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-04-21T23:48:00.401 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-04-21T23:48:00.625 INFO:teuthology.orchestra.run.smithi203.stdout:48 files removed 2024-04-21T23:48:00.625 INFO:teuthology.orchestra.run.smithi111.stdout:48 files removed 2024-04-21T23:48:00.634 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-21T23:48:00.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:48:00.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:48:00.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T23:48:01.316 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-04-21T23:48:01.320 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 214 kB/s | 83 kB 00:00 2024-04-21T23:48:01.351 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 189 kB/s | 83 kB 00:00 2024-04-21T23:48:01.719 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-21T23:48:01.733 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-04-21T23:48:01.754 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-21T23:48:02.042 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.7 kB/s | 1.8 kB 00:00 2024-04-21T23:48:02.130 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 5.4 kB/s | 1.8 kB 00:00 2024-04-21T23:48:02.136 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 4.8 kB/s | 1.8 kB 00:00 2024-04-21T23:48:02.568 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-21T23:48:02.602 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-04-21T23:48:03.916 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-21T23:48:04.660 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-21T23:48:08.923 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 19 MB 00:03 2024-04-21T23:48:09.186 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.2 MB 00:00 2024-04-21T23:48:10.762 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-21T23:48:13.424 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.2 MB 00:00 2024-04-21T23:48:14.833 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-04-21T23:48:14.930 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 21 MB 00:04 2024-04-21T23:48:19.862 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-04-21T23:48:20.030 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-21T23:48:22.197 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:22.198 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:22.209 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:48:22.209 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:48:22.210 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:48:22.211 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:48:22.212 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:48:22.214 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:48:22.215 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-21T23:48:22.291 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:48:22.292 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:48:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:48:22.294 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:48:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:48:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:48:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:48:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:48:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:48:22.300 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:48:22.301 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:48:22.302 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:48:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout:Install 148 Packages 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-04-21T23:48:22.304 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:48:22.305 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 216 M 2024-04-21T23:48:22.305 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-21T23:48:23.648 INFO:teuthology.orchestra.run.smithi203.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-21T23:48:24.015 INFO:teuthology.orchestra.run.smithi203.stdout:(2/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 8.0 MB/s | 5.4 MB 00:00 2024-04-21T23:48:24.074 INFO:teuthology.orchestra.run.smithi203.stdout:(3/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 2.0 MB/s | 889 kB 00:00 2024-04-21T23:48:24.107 INFO:teuthology.orchestra.run.smithi203.stdout:(4/150): ceph-immutable-object-cache-19.0.0-240 1.5 MB/s | 145 kB 00:00 2024-04-21T23:48:24.274 INFO:teuthology.orchestra.run.smithi203.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-04-21T23:48:24.650 INFO:teuthology.orchestra.run.smithi203.stdout:(6/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-21T23:48:24.708 INFO:teuthology.orchestra.run.smithi203.stdout:(7/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 3.7 MB/s | 2.4 MB 00:00 2024-04-21T23:48:24.818 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 920 kB/s | 19 MB 00:21 2024-04-21T23:48:25.068 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-04-21T23:48:25.900 INFO:teuthology.orchestra.run.smithi203.stdout:(8/150): ceph-common-19.0.0-2405.g820a2f8c.el9. 8.2 MB/s | 21 MB 00:02 2024-04-21T23:48:26.133 INFO:teuthology.orchestra.run.smithi203.stdout:(9/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 11 MB/s | 17 MB 00:01 2024-04-21T23:48:26.159 INFO:teuthology.orchestra.run.smithi203.stdout:(10/150): ceph-selinux-19.0.0-2405.g820a2f8c.el 97 kB/s | 25 kB 00:00 2024-04-21T23:48:26.242 INFO:teuthology.orchestra.run.smithi203.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 2.6 MB/s | 222 kB 00:00 2024-04-21T23:48:26.376 INFO:teuthology.orchestra.run.smithi203.stdout:(12/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el 6.1 MB/s | 10 MB 00:01 2024-04-21T23:48:26.402 INFO:teuthology.orchestra.run.smithi203.stdout:(13/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 170 kB/s | 27 kB 00:00 2024-04-21T23:48:26.435 INFO:teuthology.orchestra.run.smithi203.stdout:(14/150): libcephfs-devel-19.0.0-2405.g820a2f8c 543 kB/s | 32 kB 00:00 2024-04-21T23:48:26.519 INFO:teuthology.orchestra.run.smithi203.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.9 MB/s | 164 kB 00:00 2024-04-21T23:48:26.544 INFO:teuthology.orchestra.run.smithi203.stdout:(16/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 5.1 MB/s | 745 kB 00:00 2024-04-21T23:48:26.594 INFO:teuthology.orchestra.run.smithi203.stdout:(17/150): librados-devel-19.0.0-2405.g820a2f8c. 1.6 MB/s | 127 kB 00:00 2024-04-21T23:48:26.636 INFO:teuthology.orchestra.run.smithi203.stdout:(18/150): libradosstriper1-19.0.0-2405.g820a2f8 5.3 MB/s | 501 kB 00:00 2024-04-21T23:48:26.711 INFO:teuthology.orchestra.run.smithi203.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 601 kB/s | 45 kB 00:00 2024-04-21T23:48:26.803 INFO:teuthology.orchestra.run.smithi203.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.4 MB/s | 133 kB 00:00 2024-04-21T23:48:26.896 INFO:teuthology.orchestra.run.smithi203.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.7 MB/s | 161 kB 00:00 2024-04-21T23:48:27.005 INFO:teuthology.orchestra.run.smithi203.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 2.9 MB/s | 321 kB 00:00 2024-04-21T23:48:27.208 INFO:teuthology.orchestra.run.smithi203.stdout:(23/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 11 MB/s | 5.3 MB 00:00 2024-04-21T23:48:27.209 INFO:teuthology.orchestra.run.smithi203.stdout:(24/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.9 MB/s | 298 kB 00:00 2024-04-21T23:48:27.209 INFO:teuthology.orchestra.run.smithi203.stdout:(25/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.7 MB/s | 99 kB 00:00 2024-04-21T23:48:27.209 INFO:teuthology.orchestra.run.smithi203.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-21T23:48:27.273 INFO:teuthology.orchestra.run.smithi203.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-21T23:48:27.373 INFO:teuthology.orchestra.run.smithi203.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 317 kB/s | 32 kB 00:00 2024-04-21T23:48:27.395 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:27.395 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:27.407 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:48:27.408 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:48:27.409 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:48:27.410 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:48:27.411 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:48:27.413 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:48:27.414 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:48:27.432 INFO:teuthology.orchestra.run.smithi203.stdout:(29/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 11 MB/s | 3.1 MB 00:00 2024-04-21T23:48:27.449 INFO:teuthology.orchestra.run.smithi203.stdout:(30/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.8 MB/s | 141 kB 00:00 2024-04-21T23:48:27.478 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-21T23:48:27.488 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-21T23:48:27.488 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-21T23:48:27.488 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-21T23:48:27.488 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-21T23:48:27.488 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:48:27.489 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:48:27.490 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:48:27.491 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:48:27.492 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:48:27.493 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:48:27.493 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:48:27.493 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:48:27.494 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:48:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:48:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:48:27.497 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:48:27.498 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:48:27.499 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:48:27.500 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:48:27.501 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-21T23:48:27.502 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-04-21T23:48:27.503 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:48:27.504 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 216 M 2024-04-21T23:48:27.504 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-21T23:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 11 MB/s | 3.5 MB 00:00 2024-04-21T23:48:27.816 INFO:teuthology.orchestra.run.smithi203.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 3.2 MB/s | 249 kB 00:00 2024-04-21T23:48:27.892 INFO:teuthology.orchestra.run.smithi203.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 659 kB/s | 49 kB 00:00 2024-04-21T23:48:27.959 INFO:teuthology.orchestra.run.smithi203.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 230 kB/s | 15 kB 00:00 2024-04-21T23:48:28.042 INFO:teuthology.orchestra.run.smithi203.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.1 MB/s | 264 kB 00:00 2024-04-21T23:48:28.159 INFO:teuthology.orchestra.run.smithi203.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-21T23:48:28.235 INFO:teuthology.orchestra.run.smithi203.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 243 kB/s | 18 kB 00:00 2024-04-21T23:48:28.378 INFO:teuthology.orchestra.run.smithi038.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-04-21T23:48:28.451 INFO:teuthology.orchestra.run.smithi203.stdout:(38/150): ceph-mgr-diskprediction-local-19.0.0- 7.4 MB/s | 7.4 MB 00:01 2024-04-21T23:48:28.652 INFO:teuthology.orchestra.run.smithi203.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 193 kB/s | 80 kB 00:00 2024-04-21T23:48:28.803 INFO:teuthology.orchestra.run.smithi038.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 2.0 MB/s | 889 kB 00:00 2024-04-21T23:48:28.844 INFO:teuthology.orchestra.run.smithi203.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-21T23:48:28.911 INFO:teuthology.orchestra.run.smithi038.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 1.3 MB/s | 145 kB 00:00 2024-04-21T23:48:29.036 INFO:teuthology.orchestra.run.smithi203.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-04-21T23:48:29.228 INFO:teuthology.orchestra.run.smithi203.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-21T23:48:29.278 INFO:teuthology.orchestra.run.smithi038.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-04-21T23:48:29.336 INFO:teuthology.orchestra.run.smithi203.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-04-21T23:48:29.445 INFO:teuthology.orchestra.run.smithi203.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-21T23:48:29.487 INFO:teuthology.orchestra.run.smithi203.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 2.7 MB/s | 1.2 MB 00:00 2024-04-21T23:48:29.520 INFO:teuthology.orchestra.run.smithi038.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-04-21T23:48:29.546 INFO:teuthology.orchestra.run.smithi203.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-21T23:48:29.596 INFO:teuthology.orchestra.run.smithi203.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-04-21T23:48:29.820 INFO:teuthology.orchestra.run.smithi038.stdout:(6/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-04-21T23:48:29.838 INFO:teuthology.orchestra.run.smithi203.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 689 kB 00:00 2024-04-21T23:48:29.881 INFO:teuthology.orchestra.run.smithi203.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-04-21T23:48:29.922 INFO:teuthology.orchestra.run.smithi203.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 107 kB 00:00 2024-04-21T23:48:29.953 INFO:teuthology.orchestra.run.smithi203.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 320 kB/s | 21 kB 00:00 2024-04-21T23:48:30.006 INFO:teuthology.orchestra.run.smithi203.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 563 kB/s | 33 kB 00:00 2024-04-21T23:48:30.037 INFO:teuthology.orchestra.run.smithi203.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T23:48:30.227 INFO:teuthology.orchestra.run.smithi038.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 6.8 MB/s | 4.7 MB 00:00 2024-04-21T23:48:30.508 INFO:teuthology.orchestra.run.smithi203.stdout:(54/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 11 MB/s | 48 MB 00:04 2024-04-21T23:48:30.533 INFO:teuthology.orchestra.run.smithi203.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 36 kB/s | 18 kB 00:00 2024-04-21T23:48:30.575 INFO:teuthology.orchestra.run.smithi203.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-04-21T23:48:30.658 INFO:teuthology.orchestra.run.smithi203.stdout:(57/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-04-21T23:48:30.700 INFO:teuthology.orchestra.run.smithi203.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 460 kB/s | 45 kB 00:00 2024-04-21T23:48:30.747 INFO:teuthology.orchestra.run.smithi038.stdout:(8/150): ceph-common-19.0.0-2405.g820a2f8c.el9. 8.2 MB/s | 21 MB 00:02 2024-04-21T23:48:30.750 INFO:teuthology.orchestra.run.smithi203.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 662 kB 00:00 2024-04-21T23:48:30.792 INFO:teuthology.orchestra.run.smithi203.stdout:(60/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 618 kB/s | 160 kB 00:00 2024-04-21T23:48:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 505 kB/s | 25 kB 00:00 2024-04-21T23:48:30.818 INFO:teuthology.orchestra.run.smithi203.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-04-21T23:48:30.853 INFO:teuthology.orchestra.run.smithi203.stdout:(62/150): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-21T23:48:30.876 INFO:teuthology.orchestra.run.smithi203.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 569 kB/s | 47 kB 00:00 2024-04-21T23:48:30.903 INFO:teuthology.orchestra.run.smithi203.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-21T23:48:30.927 INFO:teuthology.orchestra.run.smithi203.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-21T23:48:30.953 INFO:teuthology.orchestra.run.smithi203.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-21T23:48:31.077 INFO:teuthology.orchestra.run.smithi203.stdout:(67/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-21T23:48:31.169 INFO:teuthology.orchestra.run.smithi203.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-21T23:48:31.236 INFO:teuthology.orchestra.run.smithi203.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-21T23:48:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:(10/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el 9.8 MB/s | 10 MB 00:01 2024-04-21T23:48:31.311 INFO:teuthology.orchestra.run.smithi203.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-04-21T23:48:31.347 INFO:teuthology.orchestra.run.smithi038.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 2.4 MB/s | 222 kB 00:00 2024-04-21T23:48:31.353 INFO:teuthology.orchestra.run.smithi203.stdout:(71/150): postgresql-13.14-1.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-04-21T23:48:31.410 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.2 MB 00:02 2024-04-21T23:48:31.430 INFO:teuthology.orchestra.run.smithi038.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 325 kB/s | 27 kB 00:00 2024-04-21T23:48:31.471 INFO:teuthology.orchestra.run.smithi203.stdout:(72/150): postgresql-server-13.14-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-04-21T23:48:31.496 INFO:teuthology.orchestra.run.smithi203.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 615 kB/s | 87 kB 00:00 2024-04-21T23:48:31.505 INFO:teuthology.orchestra.run.smithi038.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 423 kB/s | 32 kB 00:00 2024-04-21T23:48:31.529 INFO:teuthology.orchestra.run.smithi203.stdout:(74/150): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-04-21T23:48:31.580 INFO:teuthology.orchestra.run.smithi203.stdout:(75/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-21T23:48:31.648 INFO:teuthology.orchestra.run.smithi038.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 5.1 MB/s | 745 kB 00:00 2024-04-21T23:48:31.664 INFO:teuthology.orchestra.run.smithi203.stdout:(76/150): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-04-21T23:48:31.689 INFO:teuthology.orchestra.run.smithi203.stdout:(77/150): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-21T23:48:31.714 INFO:teuthology.orchestra.run.smithi203.stdout:(78/150): python3-jsonpatch-1.21-16.el9.noarch. 191 kB/s | 26 kB 00:00 2024-04-21T23:48:31.739 INFO:teuthology.orchestra.run.smithi203.stdout:(79/150): python3-jsonpointer-2.0-4.el9.noarch. 254 kB/s | 19 kB 00:00 2024-04-21T23:48:31.740 INFO:teuthology.orchestra.run.smithi038.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.7 MB/s | 164 kB 00:00 2024-04-21T23:48:31.765 INFO:teuthology.orchestra.run.smithi203.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-21T23:48:31.790 INFO:teuthology.orchestra.run.smithi203.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-04-21T23:48:31.823 INFO:teuthology.orchestra.run.smithi038.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.5 MB/s | 127 kB 00:00 2024-04-21T23:48:31.840 INFO:teuthology.orchestra.run.smithi203.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-21T23:48:31.891 INFO:teuthology.orchestra.run.smithi203.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-21T23:48:31.915 INFO:teuthology.orchestra.run.smithi038.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 5.4 MB/s | 501 kB 00:00 2024-04-21T23:48:31.916 INFO:teuthology.orchestra.run.smithi203.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-21T23:48:32.008 INFO:teuthology.orchestra.run.smithi203.stdout:(85/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-04-21T23:48:32.033 INFO:teuthology.orchestra.run.smithi203.stdout:(86/150): python3-prettytable-0.7.2-27.el9.noar 297 kB/s | 42 kB 00:00 2024-04-21T23:48:32.058 INFO:teuthology.orchestra.run.smithi203.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 1.8 MB/s | 269 kB 00:00 2024-04-21T23:48:32.084 INFO:teuthology.orchestra.run.smithi203.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-21T23:48:32.109 INFO:teuthology.orchestra.run.smithi203.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-21T23:48:32.134 INFO:teuthology.orchestra.run.smithi203.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-04-21T23:48:32.160 INFO:teuthology.orchestra.run.smithi203.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-04-21T23:48:32.210 INFO:teuthology.orchestra.run.smithi203.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-04-21T23:48:32.235 INFO:teuthology.orchestra.run.smithi203.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-04-21T23:48:32.294 INFO:teuthology.orchestra.run.smithi203.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-21T23:48:32.319 INFO:teuthology.orchestra.run.smithi203.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 429 kB/s | 36 kB 00:00 2024-04-21T23:48:32.428 INFO:teuthology.orchestra.run.smithi203.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-04-21T23:48:32.453 INFO:teuthology.orchestra.run.smithi203.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-21T23:48:32.495 INFO:teuthology.orchestra.run.smithi203.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 863 kB/s | 58 kB 00:00 2024-04-21T23:48:32.499 INFO:teuthology.orchestra.run.smithi038.stdout:(18/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 9.1 MB/s | 5.3 MB 00:00 2024-04-21T23:48:32.521 INFO:teuthology.orchestra.run.smithi203.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-21T23:48:32.563 INFO:teuthology.orchestra.run.smithi203.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 946 kB/s | 64 kB 00:00 2024-04-21T23:48:32.582 INFO:teuthology.orchestra.run.smithi038.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 543 kB/s | 45 kB 00:00 2024-04-21T23:48:32.588 INFO:teuthology.orchestra.run.smithi203.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 669 kB/s | 45 kB 00:00 2024-04-21T23:48:32.658 INFO:teuthology.orchestra.run.smithi038.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.7 MB/s | 133 kB 00:00 2024-04-21T23:48:32.735 INFO:teuthology.orchestra.run.smithi038.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 2.0 MB/s | 161 kB 00:00 2024-04-21T23:48:32.763 INFO:teuthology.orchestra.run.smithi203.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-21T23:48:32.789 INFO:teuthology.orchestra.run.smithi203.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-04-21T23:48:32.827 INFO:teuthology.orchestra.run.smithi038.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 3.4 MB/s | 321 kB 00:00 2024-04-21T23:48:32.839 INFO:teuthology.orchestra.run.smithi203.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-04-21T23:48:32.906 INFO:teuthology.orchestra.run.smithi203.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-04-21T23:48:32.910 INFO:teuthology.orchestra.run.smithi038.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 3.5 MB/s | 298 kB 00:00 2024-04-21T23:48:32.928 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-04-21T23:48:33.023 INFO:teuthology.orchestra.run.smithi203.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-04-21T23:48:33.065 INFO:teuthology.orchestra.run.smithi203.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-04-21T23:48:33.099 INFO:teuthology.orchestra.run.smithi203.stdout:(108/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-21T23:48:33.103 INFO:teuthology.orchestra.run.smithi038.stdout:(24/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x8 5.1 MB/s | 17 MB 00:03 2024-04-21T23:48:33.115 INFO:teuthology.orchestra.run.smithi203.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-04-21T23:48:33.128 INFO:teuthology.orchestra.run.smithi038.stdout:(25/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 458 kB/s | 99 kB 00:00 2024-04-21T23:48:33.141 INFO:teuthology.orchestra.run.smithi203.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-21T23:48:33.170 INFO:teuthology.orchestra.run.smithi038.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.2 MB/s | 86 kB 00:00 2024-04-21T23:48:33.208 INFO:teuthology.orchestra.run.smithi203.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-21T23:48:33.246 INFO:teuthology.orchestra.run.smithi038.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-21T23:48:33.304 INFO:teuthology.orchestra.run.smithi038.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 544 kB/s | 32 kB 00:00 2024-04-21T23:48:33.342 INFO:teuthology.orchestra.run.smithi203.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-21T23:48:33.371 INFO:teuthology.orchestra.run.smithi038.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 2.1 MB/s | 141 kB 00:00 2024-04-21T23:48:33.375 INFO:teuthology.orchestra.run.smithi203.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-21T23:48:33.400 INFO:teuthology.orchestra.run.smithi203.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-04-21T23:48:33.426 INFO:teuthology.orchestra.run.smithi203.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-04-21T23:48:33.451 INFO:teuthology.orchestra.run.smithi203.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-21T23:48:33.455 INFO:teuthology.orchestra.run.smithi038.stdout:(30/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 9.5 MB/s | 3.1 MB 00:00 2024-04-21T23:48:33.501 INFO:teuthology.orchestra.run.smithi203.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-21T23:48:33.568 INFO:teuthology.orchestra.run.smithi203.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-21T23:48:33.594 INFO:teuthology.orchestra.run.smithi203.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-21T23:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-04-21T23:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 1.2 MB/s | 203 kB 00:00 2024-04-21T23:48:33.811 INFO:teuthology.orchestra.run.smithi203.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-21T23:48:33.853 INFO:teuthology.orchestra.run.smithi203.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-21T23:48:33.937 INFO:teuthology.orchestra.run.smithi203.stdout:(124/150): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-04-21T23:48:33.962 INFO:teuthology.orchestra.run.smithi203.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-04-21T23:48:33.988 INFO:teuthology.orchestra.run.smithi203.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-21T23:48:34.013 INFO:teuthology.orchestra.run.smithi203.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-04-21T23:48:34.038 INFO:teuthology.orchestra.run.smithi203.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-21T23:48:34.080 INFO:teuthology.orchestra.run.smithi203.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-21T23:48:34.122 INFO:teuthology.orchestra.run.smithi203.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-21T23:48:34.164 INFO:teuthology.orchestra.run.smithi203.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-21T23:48:34.256 INFO:teuthology.orchestra.run.smithi203.stdout:(132/150): python3-grpcio-1.46.7-10.el9.x86_64. 3.2 MB/s | 2.0 MB 00:00 2024-04-21T23:48:34.281 INFO:teuthology.orchestra.run.smithi203.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-21T23:48:34.307 INFO:teuthology.orchestra.run.smithi203.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-21T23:48:34.332 INFO:teuthology.orchestra.run.smithi203.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-21T23:48:34.357 INFO:teuthology.orchestra.run.smithi203.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-04-21T23:48:34.373 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-21T23:48:34.383 INFO:teuthology.orchestra.run.smithi203.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-04-21T23:48:34.417 INFO:teuthology.orchestra.run.smithi203.stdout:(138/150): python3-kubernetes-26.1.0-2.el9.noar 2.5 MB/s | 1.0 MB 00:00 2024-04-21T23:48:34.442 INFO:teuthology.orchestra.run.smithi203.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-21T23:48:34.464 INFO:teuthology.orchestra.run.smithi038.stdout:(31/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 13 MB/s | 48 MB 00:03 2024-04-21T23:48:34.467 INFO:teuthology.orchestra.run.smithi203.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-21T23:48:34.492 INFO:teuthology.orchestra.run.smithi203.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-04-21T23:48:34.509 INFO:teuthology.orchestra.run.smithi203.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 760 kB/s | 51 kB 00:00 2024-04-21T23:48:34.531 INFO:teuthology.orchestra.run.smithi038.stdout:(32/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 3.0 MB/s | 3.5 MB 00:01 2024-04-21T23:48:34.534 INFO:teuthology.orchestra.run.smithi203.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-21T23:48:34.556 INFO:teuthology.orchestra.run.smithi038.stdout:(33/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.7 MB/s | 249 kB 00:00 2024-04-21T23:48:34.560 INFO:teuthology.orchestra.run.smithi203.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-21T23:48:34.577 INFO:teuthology.orchestra.run.smithi203.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 476 kB/s | 20 kB 00:00 2024-04-21T23:48:34.581 INFO:teuthology.orchestra.run.smithi038.stdout:(34/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 984 kB/s | 49 kB 00:00 2024-04-21T23:48:34.619 INFO:teuthology.orchestra.run.smithi203.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-21T23:48:34.644 INFO:teuthology.orchestra.run.smithi203.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-21T23:48:34.681 INFO:teuthology.orchestra.run.smithi038.stdout:(35/150): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-04-21T23:48:34.707 INFO:teuthology.orchestra.run.smithi038.stdout:(36/150): ceph-prometheus-alerts-19.0.0-2405.g8 102 kB/s | 15 kB 00:00 2024-04-21T23:48:34.732 INFO:teuthology.orchestra.run.smithi038.stdout:(37/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 1.7 MB/s | 264 kB 00:00 2024-04-21T23:48:34.765 INFO:teuthology.orchestra.run.smithi038.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 311 kB/s | 18 kB 00:00 2024-04-21T23:48:34.869 INFO:teuthology.orchestra.run.smithi203.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-21T23:48:34.881 INFO:teuthology.orchestra.run.smithi038.stdout:(39/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 3.7 MB/s | 762 kB 00:00 2024-04-21T23:48:35.020 INFO:teuthology.orchestra.run.smithi203.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-04-21T23:48:35.041 INFO:teuthology.orchestra.run.smithi038.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 260 kB/s | 80 kB 00:00 2024-04-21T23:48:35.086 INFO:teuthology.orchestra.run.smithi203.stdout:(150/150): librados2-19.0.0-2405.g820a2f8c.el9. 7.2 MB/s | 3.4 MB 00:00 2024-04-21T23:48:35.092 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:48:35.092 INFO:teuthology.orchestra.run.smithi203.stdout:Total 17 MB/s | 216 MB 00:12 2024-04-21T23:48:35.183 INFO:teuthology.orchestra.run.smithi038.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-04-21T23:48:35.208 INFO:teuthology.orchestra.run.smithi038.stdout:(42/150): libquadmath-11.4.1-3.el9.x86_64.rpm 601 kB/s | 195 kB 00:00 2024-04-21T23:48:35.484 INFO:teuthology.orchestra.run.smithi038.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 573 kB/s | 253 kB 00:00 2024-04-21T23:48:35.525 INFO:teuthology.orchestra.run.smithi038.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 336 kB/s | 106 kB 00:00 2024-04-21T23:48:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 2.6 MB/s | 1.2 MB 00:00 2024-04-21T23:48:35.793 INFO:teuthology.orchestra.run.smithi038.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 437 kB/s | 135 kB 00:00 2024-04-21T23:48:35.860 INFO:teuthology.orchestra.run.smithi038.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 406 kB/s | 125 kB 00:00 2024-04-21T23:48:35.977 INFO:teuthology.orchestra.run.smithi038.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 676 kB/s | 215 kB 00:00 2024-04-21T23:48:36.121 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-21T23:48:36.244 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-21T23:48:36.244 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-21T23:48:36.719 INFO:teuthology.orchestra.run.smithi038.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 144 kB/s | 107 kB 00:00 2024-04-21T23:48:37.278 INFO:teuthology.orchestra.run.smithi038.stdout:(50/150): automake-1.16.2-8.el9.noarch.rpm 464 kB/s | 689 kB 00:01 2024-04-21T23:48:37.303 INFO:teuthology.orchestra.run.smithi038.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-04-21T23:48:37.595 INFO:teuthology.orchestra.run.smithi038.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 544 kB/s | 943 kB 00:01 2024-04-21T23:48:37.920 INFO:teuthology.orchestra.run.smithi038.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-04-21T23:48:37.938 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-21T23:48:37.940 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-21T23:48:38.287 INFO:teuthology.orchestra.run.smithi038.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 311 kB/s | 313 kB 00:01 2024-04-21T23:48:38.437 INFO:teuthology.orchestra.run.smithi038.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-04-21T23:48:38.804 INFO:teuthology.orchestra.run.smithi038.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-04-21T23:48:39.305 INFO:teuthology.orchestra.run.smithi038.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 185 kB/s | 160 kB 00:00 2024-04-21T23:48:39.429 INFO:teuthology.orchestra.run.smithi038.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-04-21T23:48:39.729 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 4.9 kB/s | 1.7 kB 00:00 2024-04-21T23:48:40.747 INFO:teuthology.orchestra.run.smithi038.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 442 kB/s | 582 kB 00:01 2024-04-21T23:48:40.805 INFO:teuthology.orchestra.run.smithi038.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 442 kB/s | 662 kB 00:01 2024-04-21T23:48:41.507 INFO:teuthology.orchestra.run.smithi038.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 33 kB/s | 25 kB 00:00 2024-04-21T23:48:41.539 INFO:teuthology.orchestra.run.smithi038.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-04-21T23:48:41.889 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:41.889 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T23:48:41.900 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T23:48:41.901 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T23:48:41.901 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T23:48:41.902 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T23:48:41.905 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T23:48:41.906 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T23:48:41.907 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T23:48:41.966 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-21T23:48:41.982 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:48:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-21T23:48:41.982 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:48:41.982 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T23:48:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T23:48:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T23:48:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T23:48:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T23:48:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T23:48:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T23:48:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T23:48:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T23:48:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T23:48:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T23:48:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T23:48:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T23:48:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T23:48:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T23:48:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T23:48:41.997 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout:Install 148 Packages 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-04-21T23:48:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:48:41.999 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 216 M 2024-04-21T23:48:41.999 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-21T23:48:42.098 INFO:teuthology.orchestra.run.smithi038.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 680 kB/s | 3.0 MB 00:04 2024-04-21T23:48:42.456 INFO:teuthology.orchestra.run.smithi038.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 206 kB/s | 188 kB 00:00 2024-04-21T23:48:42.540 INFO:teuthology.orchestra.run.smithi038.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 283 kB/s | 292 kB 00:01 2024-04-21T23:48:42.639 INFO:teuthology.orchestra.run.smithi111.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-21T23:48:42.740 INFO:teuthology.orchestra.run.smithi038.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 55 kB/s | 35 kB 00:00 2024-04-21T23:48:42.889 INFO:teuthology.orchestra.run.smithi111.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 3.5 MB/s | 889 kB 00:00 2024-04-21T23:48:42.964 INFO:teuthology.orchestra.run.smithi111.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 1.9 MB/s | 145 kB 00:00 2024-04-21T23:48:43.040 INFO:teuthology.orchestra.run.smithi111.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 9.4 MB/s | 5.4 MB 00:00 2024-04-21T23:48:43.173 INFO:teuthology.orchestra.run.smithi111.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 11 MB/s | 1.5 MB 00:00 2024-04-21T23:48:43.249 INFO:teuthology.orchestra.run.smithi111.stdout:(6/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 8.3 MB/s | 2.4 MB 00:00 2024-04-21T23:48:43.524 INFO:teuthology.orchestra.run.smithi111.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-21T23:48:43.914 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-21T23:48:43.990 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:48:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:48:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:48:44.389 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:48:44.516 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:48:44.662 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:48:44.692 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:48:44.767 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:48:44.808 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:48:44.838 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:48:44.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:48:44.941 INFO:teuthology.orchestra.run.smithi111.stdout:(8/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 7.2 MB/s | 10 MB 00:01 2024-04-21T23:48:45.013 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:48:45.061 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:48:45.089 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:48:45.174 INFO:teuthology.orchestra.run.smithi111.stdout:(9/150): ceph-common-19.0.0-2405.g820a2f8c.el9. 7.8 MB/s | 21 MB 00:02 2024-04-21T23:48:45.193 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:48:45.200 INFO:teuthology.orchestra.run.smithi111.stdout:(10/150): ceph-selinux-19.0.0-2405.g820a2f8c.el 97 kB/s | 25 kB 00:00 2024-04-21T23:48:45.224 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:48:45.257 INFO:teuthology.orchestra.run.smithi038.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 646 kB/s | 1.6 MB 00:02 2024-04-21T23:48:45.287 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:48:45.308 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:48:45.331 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:48:45.367 INFO:teuthology.orchestra.run.smithi111.stdout:(11/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x8 7.9 MB/s | 17 MB 00:02 2024-04-21T23:48:45.392 INFO:teuthology.orchestra.run.smithi111.stdout:(12/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 1.1 MB/s | 222 kB 00:00 2024-04-21T23:48:45.426 INFO:teuthology.orchestra.run.smithi111.stdout:(13/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 462 kB/s | 27 kB 00:00 2024-04-21T23:48:45.467 INFO:teuthology.orchestra.run.smithi111.stdout:(14/150): libcephfs-devel-19.0.0-2405.g820a2f8c 422 kB/s | 32 kB 00:00 2024-04-21T23:48:45.525 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:48:45.543 INFO:teuthology.orchestra.run.smithi111.stdout:(15/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 6.2 MB/s | 745 kB 00:00 2024-04-21T23:48:45.558 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:48:45.568 INFO:teuthology.orchestra.run.smithi111.stdout:(16/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.6 MB/s | 164 kB 00:00 2024-04-21T23:48:45.602 INFO:teuthology.orchestra.run.smithi111.stdout:(17/150): librados-devel-19.0.0-2405.g820a2f8c. 2.1 MB/s | 127 kB 00:00 2024-04-21T23:48:45.665 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:48:45.702 INFO:teuthology.orchestra.run.smithi111.stdout:(18/150): libradosstriper1-19.0.0-2405.g820a2f8 3.7 MB/s | 501 kB 00:00 2024-04-21T23:48:45.754 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:48:45.786 INFO:teuthology.orchestra.run.smithi111.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 542 kB/s | 45 kB 00:00 2024-04-21T23:48:45.791 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:48:45.826 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:48:45.877 INFO:teuthology.orchestra.run.smithi111.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.4 MB/s | 133 kB 00:00 2024-04-21T23:48:45.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:48:45.925 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:48:45.968 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:48:45.970 INFO:teuthology.orchestra.run.smithi111.stdout:(21/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.7 MB/s | 161 kB 00:00 2024-04-21T23:48:46.001 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:48:46.029 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:48:46.070 INFO:teuthology.orchestra.run.smithi111.stdout:(22/150): python3-rados-19.0.0-2405.g820a2f8c.e 3.1 MB/s | 321 kB 00:00 2024-04-21T23:48:46.098 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:48:46.152 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:48:46.171 INFO:teuthology.orchestra.run.smithi111.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.9 MB/s | 298 kB 00:00 2024-04-21T23:48:46.182 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:48:46.218 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:48:46.246 INFO:teuthology.orchestra.run.smithi111.stdout:(24/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.3 MB/s | 99 kB 00:00 2024-04-21T23:48:46.295 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:48:46.321 INFO:teuthology.orchestra.run.smithi111.stdout:(25/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-21T23:48:46.329 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:48:46.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:48:46.401 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:48:46.438 INFO:teuthology.orchestra.run.smithi111.stdout:(26/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 6.4 MB/s | 5.3 MB 00:00 2024-04-21T23:48:46.473 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:48:46.516 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:48:46.530 INFO:teuthology.orchestra.run.smithi111.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-21T23:48:46.592 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:48:46.614 INFO:teuthology.orchestra.run.smithi111.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 382 kB/s | 32 kB 00:00 2024-04-21T23:48:46.728 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:48:46.730 INFO:teuthology.orchestra.run.smithi111.stdout:(29/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 7.6 MB/s | 3.1 MB 00:00 2024-04-21T23:48:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:(30/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 996 kB/s | 141 kB 00:00 2024-04-21T23:48:46.778 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:48:47.015 INFO:teuthology.orchestra.run.smithi038.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 518 kB/s | 911 kB 00:01 2024-04-21T23:48:47.215 INFO:teuthology.orchestra.run.smithi111.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 7.3 MB/s | 3.5 MB 00:00 2024-04-21T23:48:47.273 INFO:teuthology.orchestra.run.smithi111.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 4.2 MB/s | 249 kB 00:00 2024-04-21T23:48:47.345 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:48:47.348 INFO:teuthology.orchestra.run.smithi111.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 658 kB/s | 49 kB 00:00 2024-04-21T23:48:47.407 INFO:teuthology.orchestra.run.smithi111.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 261 kB/s | 15 kB 00:00 2024-04-21T23:48:47.426 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:48:47.462 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:48:47.483 INFO:teuthology.orchestra.run.smithi111.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.4 MB/s | 264 kB 00:00 2024-04-21T23:48:47.513 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:48:47.551 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:48:47.584 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:48:47.633 INFO:teuthology.orchestra.run.smithi111.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-21T23:48:47.640 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:48:47.700 INFO:teuthology.orchestra.run.smithi111.stdout:(37/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 274 kB/s | 18 kB 00:00 2024-04-21T23:48:47.714 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:48:47.752 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:48:47.790 INFO:teuthology.orchestra.run.smithi038.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 172 kB/s | 133 kB 00:00 2024-04-21T23:48:47.797 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:48:47.844 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:48:47.858 INFO:teuthology.orchestra.run.smithi111.stdout:(38/150): ceph-mgr-diskprediction-local-19.0.0- 6.7 MB/s | 7.4 MB 00:01 2024-04-21T23:48:47.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:48:47.883 INFO:teuthology.orchestra.run.smithi111.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 437 kB/s | 80 kB 00:00 2024-04-21T23:48:47.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:48:47.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:48:48.111 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:48:48.154 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:48:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:48:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:48:48.321 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:48:48.334 INFO:teuthology.orchestra.run.smithi111.stdout:(40/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 15 MB/s | 48 MB 00:03 2024-04-21T23:48:48.359 INFO:teuthology.orchestra.run.smithi111.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 411 kB/s | 195 kB 00:00 2024-04-21T23:48:48.393 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:48:48.407 INFO:teuthology.orchestra.run.smithi038.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 721 kB/s | 4.1 MB 00:05 2024-04-21T23:48:48.433 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:48:48.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:48:48.492 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:48:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-21T23:48:48.518 INFO:teuthology.orchestra.run.smithi111.stdout:(43/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-21T23:48:48.543 INFO:teuthology.orchestra.run.smithi111.stdout:(44/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-21T23:48:48.567 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:48:48.577 INFO:teuthology.orchestra.run.smithi111.stdout:(45/150): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-04-21T23:48:48.602 INFO:teuthology.orchestra.run.smithi111.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-21T23:48:48.627 INFO:teuthology.orchestra.run.smithi111.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-21T23:48:48.661 INFO:teuthology.orchestra.run.smithi111.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-21T23:48:48.694 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:48:48.733 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:48:48.761 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:48:48.796 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:48:48.840 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:48:48.878 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:48:48.918 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:48:48.947 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:48:48.982 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:48:49.015 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:48:49.057 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:48:49.074 INFO:teuthology.orchestra.run.smithi038.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 733 kB/s | 4.7 MB 00:06 2024-04-21T23:48:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:48:49.103 INFO:teuthology.orchestra.run.smithi111.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 242 kB/s | 107 kB 00:00 2024-04-21T23:48:49.179 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:48:49.220 INFO:teuthology.orchestra.run.smithi111.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 184 kB/s | 21 kB 00:00 2024-04-21T23:48:49.222 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:48:49.378 INFO:teuthology.orchestra.run.smithi111.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 888 kB/s | 689 kB 00:00 2024-04-21T23:48:49.421 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:48:49.429 INFO:teuthology.orchestra.run.smithi111.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-21T23:48:49.470 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:48:49.479 INFO:teuthology.orchestra.run.smithi111.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-04-21T23:48:49.506 INFO:teuthology.orchestra.run.smithi111.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-21T23:48:49.578 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:48:49.588 INFO:teuthology.orchestra.run.smithi111.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 167 kB/s | 18 kB 00:00 2024-04-21T23:48:49.613 INFO:teuthology.orchestra.run.smithi111.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 11 kB 00:00 2024-04-21T23:48:49.630 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:48:49.697 INFO:teuthology.orchestra.run.smithi111.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-21T23:48:49.722 INFO:teuthology.orchestra.run.smithi111.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-04-21T23:48:49.783 INFO:teuthology.orchestra.run.smithi038.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 124 kB/s | 87 kB 00:00 2024-04-21T23:48:49.830 INFO:teuthology.orchestra.run.smithi111.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-04-21T23:48:49.922 INFO:teuthology.orchestra.run.smithi111.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 582 kB 00:00 2024-04-21T23:48:49.948 INFO:teuthology.orchestra.run.smithi111.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 218 kB/s | 25 kB 00:00 2024-04-21T23:48:49.981 INFO:teuthology.orchestra.run.smithi111.stdout:(62/150): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-04-21T23:48:50.023 INFO:teuthology.orchestra.run.smithi111.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-04-21T23:48:50.065 INFO:teuthology.orchestra.run.smithi111.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-21T23:48:50.091 INFO:teuthology.orchestra.run.smithi111.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-21T23:48:50.117 INFO:teuthology.orchestra.run.smithi111.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 35 kB 00:00 2024-04-21T23:48:50.300 INFO:teuthology.orchestra.run.smithi038.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 546 kB/s | 1.0 MB 00:01 2024-04-21T23:48:50.362 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:48:50.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:48:50.416 INFO:teuthology.orchestra.run.smithi111.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-04-21T23:48:50.445 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:48:50.517 INFO:teuthology.orchestra.run.smithi111.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 4.1 MB 00:00 2024-04-21T23:48:50.546 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:48:50.550 INFO:teuthology.orchestra.run.smithi111.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-21T23:48:50.596 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:48:50.617 INFO:teuthology.orchestra.run.smithi111.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-21T23:48:50.684 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:48:50.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:48:50.743 INFO:teuthology.orchestra.run.smithi111.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-21T23:48:50.851 INFO:teuthology.orchestra.run.smithi111.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 808 kB/s | 87 kB 00:00 2024-04-21T23:48:50.901 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:48:50.937 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:48:51.014 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:48:51.192 INFO:teuthology.orchestra.run.smithi038.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 274 kB/s | 244 kB 00:00 2024-04-21T23:48:51.675 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:48:51.788 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:48:52.125 INFO:teuthology.orchestra.run.smithi038.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 266 kB/s | 248 kB 00:00 2024-04-21T23:48:52.709 INFO:teuthology.orchestra.run.smithi038.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 44 kB/s | 26 kB 00:00 2024-04-21T23:48:53.043 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:48:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:48:53.180 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:48:53.213 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:48:53.259 INFO:teuthology.orchestra.run.smithi038.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-21T23:48:53.517 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:48:53.559 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:48:53.650 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:48:53.681 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:48:53.717 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:48:54.101 INFO:teuthology.orchestra.run.smithi038.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 205 kB/s | 172 kB 00:00 2024-04-21T23:48:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:48:54.221 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:48:54.292 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:48:54.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:48:54.709 INFO:teuthology.orchestra.run.smithi038.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-04-21T23:48:55.585 INFO:teuthology.orchestra.run.smithi038.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 759 kB/s | 5.8 MB 00:07 2024-04-21T23:48:55.817 INFO:teuthology.orchestra.run.smithi111.stdout:(73/150): postgresql-server-13.14-1.el9.x86_64. 1.1 MB/s | 5.8 MB 00:05 2024-04-21T23:48:55.959 INFO:teuthology.orchestra.run.smithi111.stdout:(74/150): openblas-openmp-0.3.21-2.el9.x86_64.r 823 kB/s | 4.7 MB 00:05 2024-04-21T23:48:56.067 INFO:teuthology.orchestra.run.smithi111.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-21T23:48:56.168 INFO:teuthology.orchestra.run.smithi111.stdout:(76/150): python3-devel-3.9.18-3.el9.x86_64.rpm 697 kB/s | 244 kB 00:00 2024-04-21T23:48:56.185 INFO:teuthology.orchestra.run.smithi111.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-21T23:48:56.261 INFO:teuthology.orchestra.run.smithi111.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-04-21T23:48:56.294 INFO:teuthology.orchestra.run.smithi111.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-04-21T23:48:56.361 INFO:teuthology.orchestra.run.smithi111.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-04-21T23:48:56.476 INFO:teuthology.orchestra.run.smithi038.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 281 kB/s | 250 kB 00:00 2024-04-21T23:48:56.544 INFO:teuthology.orchestra.run.smithi111.stdout:(81/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-21T23:48:56.653 INFO:teuthology.orchestra.run.smithi111.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-21T23:48:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:48:56.694 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:48:56.736 INFO:teuthology.orchestra.run.smithi111.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 667 kB/s | 250 kB 00:00 2024-04-21T23:48:56.761 INFO:teuthology.orchestra.run.smithi111.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 712 kB/s | 77 kB 00:00 2024-04-21T23:48:56.878 INFO:teuthology.orchestra.run.smithi111.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 298 kB/s | 42 kB 00:00 2024-04-21T23:48:56.904 INFO:teuthology.orchestra.run.smithi111.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-21T23:48:57.012 INFO:teuthology.orchestra.run.smithi111.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-21T23:48:57.088 INFO:teuthology.orchestra.run.smithi111.stdout:(88/150): python3-babel-2.9.1-2.el9.noarch.rpm 979 kB/s | 6.0 MB 00:06 2024-04-21T23:48:57.121 INFO:teuthology.orchestra.run.smithi111.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-04-21T23:48:57.171 INFO:teuthology.orchestra.run.smithi111.stdout:(90/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 542 kB/s | 159 kB 00:00 2024-04-21T23:48:57.197 INFO:teuthology.orchestra.run.smithi111.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 493 kB/s | 54 kB 00:00 2024-04-21T23:48:57.262 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:48:57.273 INFO:teuthology.orchestra.run.smithi111.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-04-21T23:48:57.297 INFO:teuthology.orchestra.run.smithi111.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 440 kB/s | 44 kB 00:00 2024-04-21T23:48:57.298 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:48:57.368 INFO:teuthology.orchestra.run.smithi038.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 249 kB/s | 222 kB 00:00 2024-04-21T23:48:57.392 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:48:57.393 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:48:57.393 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:48:57.393 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:48:57.406 INFO:teuthology.orchestra.run.smithi111.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 331 kB/s | 36 kB 00:00 2024-04-21T23:48:57.425 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:48:57.515 INFO:teuthology.orchestra.run.smithi111.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-04-21T23:48:57.623 INFO:teuthology.orchestra.run.smithi111.stdout:(96/150): qatlib-24.02.0-1.el9.x86_64.rpm 629 kB/s | 221 kB 00:00 2024-04-21T23:48:57.791 INFO:teuthology.orchestra.run.smithi111.stdout:(97/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-21T23:48:57.794 INFO:teuthology.orchestra.run.smithi038.stdout:(83/150): python3-babel-2.9.1-2.el9.noarch.rpm 762 kB/s | 6.0 MB 00:08 2024-04-21T23:48:57.815 INFO:teuthology.orchestra.run.smithi111.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 301 kB/s | 58 kB 00:00 2024-04-21T23:48:57.899 INFO:teuthology.orchestra.run.smithi111.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 867 kB/s | 94 kB 00:00 2024-04-21T23:48:58.000 INFO:teuthology.orchestra.run.smithi111.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-21T23:48:58.025 INFO:teuthology.orchestra.run.smithi111.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 358 kB/s | 45 kB 00:00 2024-04-21T23:48:58.050 INFO:teuthology.orchestra.run.smithi111.stdout:(102/150): socat-1.7.4.1-5.el9.x86_64.rpm 570 kB/s | 305 kB 00:00 2024-04-21T23:48:58.125 INFO:teuthology.orchestra.run.smithi111.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-04-21T23:48:58.169 INFO:teuthology.orchestra.run.smithi038.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 96 kB/s | 77 kB 00:00 2024-04-21T23:48:58.184 INFO:teuthology.orchestra.run.smithi111.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-04-21T23:48:58.277 INFO:teuthology.orchestra.run.smithi111.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-04-21T23:48:58.301 INFO:teuthology.orchestra.run.smithi111.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-04-21T23:48:58.318 INFO:teuthology.orchestra.run.smithi111.stdout:(107/150): grpc-data-1.46.7-10.el9.noarch.rpm 145 kB/s | 19 kB 00:00 2024-04-21T23:48:58.435 INFO:teuthology.orchestra.run.smithi111.stdout:(108/150): libarrow-9.0.0-10.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-04-21T23:48:58.444 INFO:teuthology.orchestra.run.smithi038.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 65 kB/s | 42 kB 00:00 2024-04-21T23:48:58.460 INFO:teuthology.orchestra.run.smithi111.stdout:(109/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 185 kB/s | 26 kB 00:00 2024-04-21T23:48:58.485 INFO:teuthology.orchestra.run.smithi111.stdout:(110/150): liboath-2.6.7-2.el9.x86_64.rpm 294 kB/s | 49 kB 00:00 2024-04-21T23:48:58.502 INFO:teuthology.orchestra.run.smithi111.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-21T23:48:58.536 INFO:teuthology.orchestra.run.smithi111.stdout:(112/150): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-04-21T23:48:58.578 INFO:teuthology.orchestra.run.smithi111.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-04-21T23:48:58.611 INFO:teuthology.orchestra.run.smithi111.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-21T23:48:58.628 INFO:teuthology.orchestra.run.smithi111.stdout:(115/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 485 kB/s | 43 kB 00:00 2024-04-21T23:48:58.645 INFO:teuthology.orchestra.run.smithi111.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 971 kB/s | 32 kB 00:00 2024-04-21T23:48:58.662 INFO:teuthology.orchestra.run.smithi111.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-04-21T23:48:58.696 INFO:teuthology.orchestra.run.smithi111.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-21T23:48:58.737 INFO:teuthology.orchestra.run.smithi111.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-21T23:48:58.763 INFO:teuthology.orchestra.run.smithi111.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-21T23:48:58.788 INFO:teuthology.orchestra.run.smithi111.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 388 kB/s | 35 kB 00:00 2024-04-21T23:48:58.813 INFO:teuthology.orchestra.run.smithi111.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-21T23:48:58.855 INFO:teuthology.orchestra.run.smithi111.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-04-21T23:48:58.880 INFO:teuthology.orchestra.run.smithi111.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 1.6 MB/s | 144 kB 00:00 2024-04-21T23:48:58.905 INFO:teuthology.orchestra.run.smithi111.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 120 kB/s | 11 kB 00:00 2024-04-21T23:48:58.933 INFO:teuthology.orchestra.run.smithi111.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 444 kB/s | 23 kB 00:00 2024-04-21T23:48:58.958 INFO:teuthology.orchestra.run.smithi111.stdout:(127/150): python3-jaraco-classes-3.2.1-5.el9.n 228 kB/s | 18 kB 00:00 2024-04-21T23:48:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 226 kB/s | 19 kB 00:00 2024-04-21T23:48:59.017 INFO:teuthology.orchestra.run.smithi111.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-04-21T23:48:59.059 INFO:teuthology.orchestra.run.smithi111.stdout:(130/150): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-21T23:48:59.084 INFO:teuthology.orchestra.run.smithi111.stdout:(131/150): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-21T23:48:59.109 INFO:teuthology.orchestra.run.smithi111.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 868 kB/s | 79 kB 00:00 2024-04-21T23:48:59.128 INFO:teuthology.orchestra.run.smithi038.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 281 kB/s | 269 kB 00:00 2024-04-21T23:48:59.135 INFO:teuthology.orchestra.run.smithi111.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-04-21T23:48:59.160 INFO:teuthology.orchestra.run.smithi111.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-21T23:48:59.185 INFO:teuthology.orchestra.run.smithi111.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-21T23:48:59.211 INFO:teuthology.orchestra.run.smithi111.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-21T23:48:59.228 INFO:teuthology.orchestra.run.smithi038.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 203 kB/s | 159 kB 00:00 2024-04-21T23:48:59.235 INFO:teuthology.orchestra.run.smithi111.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 568 kB/s | 28 kB 00:00 2024-04-21T23:48:59.286 INFO:teuthology.orchestra.run.smithi111.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 308 kB/s | 31 kB 00:00 2024-04-21T23:48:59.311 INFO:teuthology.orchestra.run.smithi111.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-21T23:48:59.336 INFO:teuthology.orchestra.run.smithi111.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-21T23:48:59.362 INFO:teuthology.orchestra.run.smithi111.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 478 kB/s | 36 kB 00:00 2024-04-21T23:48:59.387 INFO:teuthology.orchestra.run.smithi111.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-04-21T23:48:59.412 INFO:teuthology.orchestra.run.smithi111.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-21T23:48:59.437 INFO:teuthology.orchestra.run.smithi111.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-21T23:48:59.462 INFO:teuthology.orchestra.run.smithi111.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-04-21T23:48:59.488 INFO:teuthology.orchestra.run.smithi111.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 407 kB/s | 20 kB 00:00 2024-04-21T23:48:59.513 INFO:teuthology.orchestra.run.smithi111.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T23:48:59.555 INFO:teuthology.orchestra.run.smithi111.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-21T23:48:59.865 INFO:teuthology.orchestra.run.smithi111.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 9.0 MB/s | 3.1 MB 00:00 2024-04-21T23:48:59.870 INFO:teuthology.orchestra.run.smithi038.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-04-21T23:48:59.922 INFO:teuthology.orchestra.run.smithi111.stdout:(150/150): librados2-19.0.0-2405.g820a2f8c.el9. 8.4 MB/s | 3.4 MB 00:00 2024-04-21T23:48:59.932 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:48:59.932 INFO:teuthology.orchestra.run.smithi111.stdout:Total 12 MB/s | 216 MB 00:17 2024-04-21T23:49:00.097 INFO:teuthology.orchestra.run.smithi038.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 288 kB/s | 279 kB 00:00 2024-04-21T23:49:00.521 INFO:teuthology.orchestra.run.smithi038.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 82 kB/s | 54 kB 00:00 2024-04-21T23:49:00.985 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-21T23:49:01.115 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-21T23:49:01.115 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-21T23:49:01.173 INFO:teuthology.orchestra.run.smithi038.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-04-21T23:49:01.796 INFO:teuthology.orchestra.run.smithi038.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 71 kB/s | 44 kB 00:00 2024-04-21T23:49:01.963 INFO:teuthology.orchestra.run.smithi038.stdout:(93/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 720 kB/s | 5.1 MB 00:07 2024-04-21T23:49:02.597 INFO:teuthology.orchestra.run.smithi038.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 57 kB/s | 36 kB 00:00 2024-04-21T23:49:02.680 INFO:teuthology.orchestra.run.smithi038.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 250 kB/s | 221 kB 00:00 2024-04-21T23:49:02.817 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-21T23:49:02.817 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-21T23:49:03.239 INFO:teuthology.orchestra.run.smithi038.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-04-21T23:49:03.722 INFO:teuthology.orchestra.run.smithi038.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 293 kB/s | 305 kB 00:01 2024-04-21T23:49:03.915 INFO:teuthology.orchestra.run.smithi038.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-21T23:49:04.432 INFO:teuthology.orchestra.run.smithi038.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 132 kB/s | 94 kB 00:00 2024-04-21T23:49:04.582 INFO:teuthology.orchestra.run.smithi038.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-04-21T23:49:04.657 INFO:teuthology.orchestra.run.smithi038.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2024-04-21T23:49:04.791 INFO:teuthology.orchestra.run.smithi038.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-04-21T23:49:04.900 INFO:teuthology.orchestra.run.smithi038.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 5.0 MB/s | 548 kB 00:00 2024-04-21T23:49:04.941 INFO:teuthology.orchestra.run.smithi038.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-04-21T23:49:04.975 INFO:teuthology.orchestra.run.smithi038.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 583 kB/s | 19 kB 00:00 2024-04-21T23:49:05.109 INFO:teuthology.orchestra.run.smithi038.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-04-21T23:49:05.126 INFO:teuthology.orchestra.run.smithi038.stdout:(107/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 65 kB/s | 45 kB 00:00 2024-04-21T23:49:05.151 INFO:teuthology.orchestra.run.smithi038.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 619 kB/s | 26 kB 00:00 2024-04-21T23:49:05.185 INFO:teuthology.orchestra.run.smithi038.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-04-21T23:49:05.210 INFO:teuthology.orchestra.run.smithi038.stdout:(110/150): liboath-2.6.7-2.el9.x86_64.rpm 580 kB/s | 49 kB 00:00 2024-04-21T23:49:05.244 INFO:teuthology.orchestra.run.smithi038.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-21T23:49:05.311 INFO:teuthology.orchestra.run.smithi038.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-04-21T23:49:05.352 INFO:teuthology.orchestra.run.smithi038.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-21T23:49:05.378 INFO:teuthology.orchestra.run.smithi038.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 649 kB/s | 43 kB 00:00 2024-04-21T23:49:05.403 INFO:teuthology.orchestra.run.smithi038.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-04-21T23:49:05.428 INFO:teuthology.orchestra.run.smithi038.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-04-21T23:49:05.454 INFO:teuthology.orchestra.run.smithi038.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-21T23:49:05.479 INFO:teuthology.orchestra.run.smithi038.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 6.9 MB/s | 358 kB 00:00 2024-04-21T23:49:05.504 INFO:teuthology.orchestra.run.smithi038.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-21T23:49:05.538 INFO:teuthology.orchestra.run.smithi038.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 603 kB/s | 35 kB 00:00 2024-04-21T23:49:05.563 INFO:teuthology.orchestra.run.smithi038.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-21T23:49:05.605 INFO:teuthology.orchestra.run.smithi038.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-21T23:49:05.647 INFO:teuthology.orchestra.run.smithi038.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-21T23:49:05.673 INFO:teuthology.orchestra.run.smithi038.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-04-21T23:49:05.698 INFO:teuthology.orchestra.run.smithi038.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 349 kB/s | 18 kB 00:00 2024-04-21T23:49:05.724 INFO:teuthology.orchestra.run.smithi038.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 458 kB/s | 23 kB 00:00 2024-04-21T23:49:05.749 INFO:teuthology.orchestra.run.smithi038.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-21T23:49:05.774 INFO:teuthology.orchestra.run.smithi038.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-04-21T23:49:05.824 INFO:teuthology.orchestra.run.smithi038.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-04-21T23:49:05.850 INFO:teuthology.orchestra.run.smithi038.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-21T23:49:05.875 INFO:teuthology.orchestra.run.smithi038.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-21T23:49:05.900 INFO:teuthology.orchestra.run.smithi038.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-21T23:49:05.926 INFO:teuthology.orchestra.run.smithi038.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-21T23:49:05.951 INFO:teuthology.orchestra.run.smithi038.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-21T23:49:05.976 INFO:teuthology.orchestra.run.smithi038.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-21T23:49:06.002 INFO:teuthology.orchestra.run.smithi038.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-21T23:49:06.027 INFO:teuthology.orchestra.run.smithi038.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-04-21T23:49:06.052 INFO:teuthology.orchestra.run.smithi038.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-21T23:49:06.077 INFO:teuthology.orchestra.run.smithi038.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-21T23:49:06.094 INFO:teuthology.orchestra.run.smithi038.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 853 kB/s | 36 kB 00:00 2024-04-21T23:49:06.111 INFO:teuthology.orchestra.run.smithi038.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-04-21T23:49:06.145 INFO:teuthology.orchestra.run.smithi038.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-21T23:49:06.187 INFO:teuthology.orchestra.run.smithi038.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-21T23:49:06.221 INFO:teuthology.orchestra.run.smithi038.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-21T23:49:06.238 INFO:teuthology.orchestra.run.smithi038.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 401 kB/s | 20 kB 00:00 2024-04-21T23:49:06.263 INFO:teuthology.orchestra.run.smithi038.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-04-21T23:49:06.322 INFO:teuthology.orchestra.run.smithi038.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-21T23:49:06.688 INFO:teuthology.orchestra.run.smithi038.stdout:(148/150): librados2-19.0.0-2405.g820a2f8c.el9. 8.0 MB/s | 3.4 MB 00:00 2024-04-21T23:49:06.764 INFO:teuthology.orchestra.run.smithi038.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-04-21T23:49:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-21T23:49:08.790 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:49:08.824 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:49:08.867 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-21T23:49:09.021 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-21T23:49:09.022 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-21T23:49:09.022 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-21T23:49:09.022 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:09.215 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:09.215 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:49:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:09.259 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:49:09.317 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:49:09.439 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:49:09.468 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:09.559 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:09.601 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:49:09.631 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:49:09.657 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:49:09.807 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:49:09.871 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:49:09.899 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:09.986 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:10.008 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:49:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:49:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:49:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:49:10.317 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:49:10.350 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:49:10.458 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:49:10.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:49:10.591 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:49:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:49:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:49:10.711 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:10.736 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:10.737 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:10.737 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:49:10.737 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:10.737 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:10.737 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:49:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:49:10.819 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:49:10.847 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:49:10.917 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:49:10.962 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:10.978 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:49:10.983 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:10.984 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:10.984 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:49:10.984 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:10.984 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:10.984 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:11.007 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:49:11.051 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:49:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:49:11.162 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:49:11.206 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:49:11.234 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:49:11.305 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:49:11.349 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:49:11.421 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:11.424 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:49:11.443 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:11.443 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:11.443 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:49:11.443 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:11.443 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:11.444 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:11.551 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:49:11.560 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:49:11.610 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:49:11.618 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:49:11.663 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:49:11.849 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:49:11.897 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:49:11.966 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:49:12.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:49:12.148 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:49:12.196 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:49:12.231 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:49:12.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:49:12.259 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:49:12.271 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:49:12.299 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:49:12.302 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:49:12.339 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:49:12.392 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:49:12.649 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:49:12.722 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:49:12.743 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:49:12.758 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:49:12.773 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:49:12.795 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:49:12.804 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:49:12.842 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:49:12.875 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:49:12.877 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:49:12.905 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:49:12.920 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:49:12.963 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:49:13.099 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:49:13.151 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:49:13.180 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:49:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:49:13.318 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:49:13.355 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:49:13.396 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:49:13.421 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:49:13.446 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:49:13.521 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:49:13.649 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:49:13.687 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:49:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:49:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:49:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:49:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:49:13.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:49:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:49:13.950 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:49:13.982 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:49:14.101 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:49:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:49:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:49:14.278 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:49:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:49:14.408 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:49:14.439 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:49:14.478 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:49:14.577 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:49:14.624 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:49:14.945 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:49:14.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:49:15.074 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:49:15.238 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:49:15.268 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:49:15.290 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:49:15.290 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:15.290 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:49:15.290 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:49:15.291 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:49:15.291 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:15.335 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:49:15.346 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:49:15.349 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:49:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:49:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:49:15.443 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:49:15.492 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:49:15.530 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:49:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:49:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:49:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:49:15.893 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:49:15.930 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:49:16.015 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:49:16.391 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:49:16.421 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:49:16.444 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:49:16.444 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:16.444 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:49:16.445 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:49:16.445 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:49:16.445 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:16.489 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:49:16.510 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:49:16.510 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:16.510 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:49:16.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:16.556 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:49:16.578 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:49:16.578 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:16.578 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:49:16.578 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:49:16.578 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:49:16.579 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:16.667 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:49:16.778 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:49:16.882 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:49:16.904 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:49:16.904 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:16.905 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:49:16.905 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:49:16.905 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:49:16.905 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:18.005 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:49:18.101 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:49:18.128 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:49:18.161 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:49:18.477 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:49:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:49:18.641 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:49:18.671 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:49:18.706 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:49:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:49:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:49:19.282 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:49:19.312 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:49:21.394 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:49:21.444 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:49:21.485 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:49:21.517 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:49:21.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:49:21.596 INFO:teuthology.orchestra.run.smithi038.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 775 kB/s | 16 MB 00:21 2024-04-21T23:49:21.600 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:49:21.600 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.0 MB/s | 216 MB 00:54 2024-04-21T23:49:21.623 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:21.659 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:21.724 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:49:21.766 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:49:21.796 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:49:22.212 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:22.245 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:49:22.337 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:49:22.337 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:49:22.337 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:49:22.337 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:22.371 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:49:22.418 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:22.462 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:22.481 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:22.677 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-21T23:49:22.817 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-21T23:49:22.818 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-21T23:49:23.261 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:49:23.341 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:49:23.382 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:49:23.462 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:49:23.503 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:49:23.504 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:49:23.587 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:49:23.587 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:24.640 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-21T23:49:24.641 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-21T23:49:29.597 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:29.597 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:49:29.597 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:49:29.598 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:49:29.599 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:49:29.611 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:49:29.612 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:49:29.613 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:49:29.614 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:49:29.615 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:49:29.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:49:29.617 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:49:29.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:49:29.619 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:49:29.620 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:49:29.621 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:49:30.706 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-21T23:49:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T23:49:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T23:49:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T23:49:31.200 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T23:49:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:49:31.441 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T23:49:31.467 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:31.610 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:49:31.639 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T23:49:31.664 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:49:31.686 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.687 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:49:31.688 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.689 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:49:31.690 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:49:31.691 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:49:31.692 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:49:31.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:49:31.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:49:31.696 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:49:31.697 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-21T23:49:31.826 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T23:49:31.879 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T23:49:31.906 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:32.012 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:32.041 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:49:32.067 DEBUG:teuthology.parallel:result is None 2024-04-21T23:49:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T23:49:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:49:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T23:49:32.353 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:49:32.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T23:49:32.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T23:49:32.592 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T23:49:32.625 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T23:49:32.661 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T23:49:32.738 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T23:49:32.776 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T23:49:32.820 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T23:49:32.853 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T23:49:32.880 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T23:49:32.945 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T23:49:33.005 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T23:49:33.133 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T23:49:33.178 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T23:49:33.265 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T23:49:33.296 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T23:49:33.340 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T23:49:33.368 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T23:49:33.445 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T23:49:33.492 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T23:49:33.579 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T23:49:33.728 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T23:49:33.775 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:49:33.775 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-04-21T23:49:33.775 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-04-21T23:49:33.775 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-04-21T23:49:33.776 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-04-21T23:49:33.776 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-04-21T23:49:33.776 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-04-21T23:49:33.776 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-04-21T23:49:33.776 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:33.789 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T23:49:33.950 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:33.967 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:33.967 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:33.967 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:49:33.967 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T23:49:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T23:49:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T23:49:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T23:49:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T23:49:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T23:49:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T23:49:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T23:49:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T23:49:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T23:49:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T23:49:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T23:49:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T23:49:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T23:49:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T23:49:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T23:49:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T23:49:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T23:49:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T23:49:35.453 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T23:49:35.479 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:49:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:35.480 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T23:49:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T23:49:35.574 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T23:49:35.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T23:49:35.704 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:35.726 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:35.727 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:35.727 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:49:35.727 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:35.727 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:35.727 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:35.785 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T23:49:35.824 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T23:49:35.861 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T23:49:35.895 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T23:49:35.940 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T23:49:35.978 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T23:49:36.018 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T23:49:36.047 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T23:49:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T23:49:36.115 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T23:49:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T23:49:36.188 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:36.191 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T23:49:36.210 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:36.210 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:36.210 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:49:36.211 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:36.211 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:36.211 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:36.275 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T23:49:36.318 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:49:36.330 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T23:49:36.361 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T23:49:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:49:36.410 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T23:49:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:49:36.522 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T23:49:36.573 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T23:49:36.615 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:49:36.663 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:49:36.732 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:49:36.821 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:49:36.951 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:49:36.998 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:49:37.029 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:49:37.057 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:49:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T23:49:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T23:49:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T23:49:37.501 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:49:37.530 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:49:37.553 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:49:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T23:49:37.607 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:49:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T23:49:37.636 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:49:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T23:49:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T23:49:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T23:49:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T23:49:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T23:49:38.669 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:49:38.705 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:49:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T23:49:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T23:49:39.675 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:49:39.706 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:49:39.808 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:49:39.971 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:49:40.008 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:49:40.031 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:49:40.031 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:40.032 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:49:40.032 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:49:40.032 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:49:40.032 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:40.075 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:49:40.086 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:49:40.184 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:49:40.231 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:49:40.421 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T23:49:40.500 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T23:49:40.528 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T23:49:40.561 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T23:49:40.890 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T23:49:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:49:41.024 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T23:49:41.054 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T23:49:41.091 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T23:49:41.141 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:49:41.169 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:49:41.192 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:41.237 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:49:41.258 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:49:41.259 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:41.259 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:49:41.259 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:49:41.326 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:49:41.326 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:41.327 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:49:41.327 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:49:41.327 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:49:41.327 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:41.589 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T23:49:41.618 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:49:41.630 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:49:41.652 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:41.673 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T23:49:41.703 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T23:49:44.031 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:44.059 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:44.633 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T23:49:44.671 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:49:44.768 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T23:49:44.768 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T23:49:44.768 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T23:49:44.768 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:44.806 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:49:46.217 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:49:46.258 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:49:46.298 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:49:46.331 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:49:46.358 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:49:46.540 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:49:46.588 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:49:46.618 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:49:47.254 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:47.298 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:47.318 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:49:48.089 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:49:48.178 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:49:48.218 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:49:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:49:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:49:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:49:48.399 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:49:48.399 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:54.479 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:54.479 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:49:54.480 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:49:54.481 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:49:54.482 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:49:54.482 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:49:54.482 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:49:54.482 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:49:54.493 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:49:54.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:49:54.495 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:49:54.496 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:49:54.497 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:49:54.498 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:49:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:49:54.500 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:49:54.501 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:49:54.502 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:49:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:49:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:49:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:49:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:49:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:49:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:49:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:49:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:49:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-21T23:49:56.543 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-21T23:49:56.805 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T23:49:56.805 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-21T23:49:56.806 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:56.896 DEBUG:teuthology.parallel:result is None 2024-04-21T23:49:56.987 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:57.002 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T23:49:57.003 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:57.003 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T23:49:57.003 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:58.539 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:58.566 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T23:49:58.567 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:58.567 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T23:49:58.567 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:58.567 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T23:49:58.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:58.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:58.832 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T23:49:58.832 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:58.832 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T23:49:58.832 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:58.833 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T23:49:58.833 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:59.301 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T23:49:59.325 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:49:59.432 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T23:49:59.491 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T23:49:59.554 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T23:49:59.751 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T23:49:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T23:50:00.009 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T23:50:00.099 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T23:50:00.253 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T23:50:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T23:50:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T23:50:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T23:50:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:50:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T23:50:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T23:50:00.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T23:50:00.984 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:50:02.082 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T23:50:02.111 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:50:03.133 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T23:50:03.162 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:50:03.273 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T23:50:03.441 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T23:50:03.464 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T23:50:03.488 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:03.540 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:50:03.552 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T23:50:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T23:50:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T23:50:04.665 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T23:50:04.701 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:50:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T23:50:04.725 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:50:04.725 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T23:50:04.726 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:50:04.726 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T23:50:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:50:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T23:50:04.792 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:50:04.792 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T23:50:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:04.869 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:50:04.892 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T23:50:04.892 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:50:04.893 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T23:50:04.893 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:50:04.893 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T23:50:04.893 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:05.211 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T23:50:05.235 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:10.019 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T23:50:10.066 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T23:50:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T23:50:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T23:50:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T23:50:10.365 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T23:50:10.404 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T23:50:10.434 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T23:50:11.007 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:50:11.040 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:50:11.061 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T23:50:11.921 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T23:50:12.012 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T23:50:12.054 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T23:50:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T23:50:12.183 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T23:50:12.183 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:50:12.267 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T23:50:12.267 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:50:18.303 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T23:50:18.304 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T23:50:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T23:50:18.306 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T23:50:18.306 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T23:50:18.306 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T23:50:18.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T23:50:18.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T23:50:18.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T23:50:18.321 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T23:50:18.322 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T23:50:18.323 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T23:50:18.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T23:50:18.325 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T23:50:18.326 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T23:50:18.327 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T23:50:20.364 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.365 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T23:50:20.366 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T23:50:20.367 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T23:50:20.368 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T23:50:20.369 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T23:50:20.370 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T23:50:20.371 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T23:50:20.372 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T23:50:20.373 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T23:50:20.374 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:50:20.375 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-21T23:50:20.743 DEBUG:teuthology.parallel:result is None 2024-04-21T23:50:20.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:20.885 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:50:20.907 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:20.908 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:20.908 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:50:20.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:21.056 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:50:21.083 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:21.083 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:21.083 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:50:21.085 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:21.226 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T23:50:21.252 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:21.252 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T23:50:21.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T23:50:21.254 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T23:50:21.255 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:50:21.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:50:21.281 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:50:21.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:50:21.313 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:50:21.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T23:50:21.343 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T23:50:21.344 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:50:21.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:50:21.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:50:21.434 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:50:21.434 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:50:21.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:50:21.531 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:50:21.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T23:50:21.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T23:50:21.627 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T23:50:21.628 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:50:21.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:50:21.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:50:21.718 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:50:21.718 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:50:21.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:50:21.806 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:50:21.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T23:50:21.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T23:50:21.903 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T23:50:21.903 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:50:21.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:50:21.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:50:21.992 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:50:21.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:50:22.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:50:22.081 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:50:22.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T23:50:22.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T23:50:22.177 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T23:50:22.276 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-21T23:50:22.276 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:22.277 INFO:tasks.cephadm:Cluster fsid is eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:50:22.277 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T23:50:22.277 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.38', 'mon.b': '172.21.15.111', 'mon.c': '172.21.15.203'} 2024-04-21T23:50:22.277 INFO:tasks.cephadm:First mon is mon.a on smithi038 2024-04-21T23:50:22.277 INFO:tasks.cephadm:First mgr is x 2024-04-21T23:50:22.278 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T23:50:22.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-21T23:50:22.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-04-21T23:50:22.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-04-21T23:50:22.353 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T23:50:22.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:22.575 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54945/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2405-g820a2f8c', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2405.g820a2f8c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-20 17:52:04.986240', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-04-21T23:50:22.721 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 820a2f8ced66152ed3d765c31444b8076ea28c6b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:50:22.723 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm 2024-04-21T23:50:22.725 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:23.776 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:50 /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:23.776 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:24.462 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:50 /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:24.463 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:25.729 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 23:50 /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:25.729 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:25.749 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:25.768 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T23:50:25.804 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b on all hosts... 2024-04-21T23:50:25.804 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:50:25.806 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:50:25.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T23:50:27.331 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:50:27.389 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:50:27.529 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:51:05.486 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-21T23:51:05.487 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-21T23:51:14.148 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-04-21T23:51:14.149 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-21T23:51:21.595 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-21T23:51:21.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-21T23:51:21.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-04-21T23:51:21.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-04-21T23:51:21.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-21T23:51:21.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-04-21T23:51:21.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-04-21T23:51:21.809 INFO:tasks.cephadm:Writing seed config... 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T23:51:21.811 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T23:51:21.812 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T23:51:21.813 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T23:51:21.814 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T23:51:21.815 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:51:21.815 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T23:51:21.830 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eab6f840-0039-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T23:51:21.831 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service 2024-04-21T23:51:21.873 DEBUG:teuthology.orchestra.run.smithi038:mgr.x> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x.service 2024-04-21T23:51:21.915 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T23:51:21.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b -v bootstrap --fsid eab6f840-0039-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:51:22.111 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-21T23:51:22.111 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b', '-v', 'bootstrap', '--fsid', 'eab6f840-0039-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-04-21T23:51:22.112 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T23:51:22.112 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-04-21T23:51:22.139 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T23:51:22.139 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-04-21T23:51:22.139 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-04-21T23:51:22.146 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T23:51:22.146 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T23:51:22.153 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T23:51:22.153 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:51:22.159 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-21T23:51:22.166 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-21T23:51:22.166 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-21T23:51:22.166 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-04-21T23:51:22.189 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T23:51:22.189 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T23:51:22.189 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-04-21T23:51:22.189 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-04-21T23:51:22.196 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T23:51:22.196 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T23:51:22.202 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T23:51:22.202 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:51:22.208 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 140382776546208 on /run/cephadm/eab6f840-0039-11ef-bc93-c7b262605968.lock 2024-04-21T23:51:22.215 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 140382776546208 acquired on /run/cephadm/eab6f840-0039-11ef-bc93-c7b262605968.lock 2024-04-21T23:51:22.216 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2024-04-21T23:51:22.216 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-04-21T23:51:22.216 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-21T23:51:22.219 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-04-21T23:51:22.219 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-04-21T23:51:22.222 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T23:51:22.222 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T23:51:22.222 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T23:51:22.224 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T23:51:22.224 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T23:51:22.224 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T23:51:22.225 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T23:51:22.225 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-21T23:51:22.225 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T23:51:22.225 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-21T23:51:22.225 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-04-21T23:51:22.226 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T23:51:22.226 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T23:51:22.226 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:51:22.706 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:4706d97462bc6e0886c6ef3c0b1ac67e3e6a7e63a606e29958996d1fc15dafd8 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T23:51:22.707 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T23:51:23.995 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T23:51:23.995 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T23:51:23.995 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T23:51:24.925 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-04-21T23:51:24.926 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-04-21T23:51:25.821 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQB9piVmOwT7DxAAZ09BYDlLoSNWqAhlw0Coag== 2024-04-21T23:51:26.791 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQB+piVmEsC5DBAAjEW7OYM7aner4IdHiKs6BQ== 2024-04-21T23:51:27.771 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQB/piVm9h7fBxAAVreZ9iZTL4eutKrnGM62nA== 2024-04-21T23:51:27.772 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-04-21T23:51:28.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T23:51:28.762 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:51:28.763 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-04-21T23:51:29.697 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2024-04-21T23:51:29.914 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T23:51:30.091 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T23:51:30.288 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target. 2024-04-21T23:51:30.288 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target. 2024-04-21T23:51:30.532 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a 2024-04-21T23:51:30.532 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T23:51:30.694 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target.wants/ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968@.service. 2024-04-21T23:51:30.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:30 smithi038 podman[27583]: 2024-04-21 23:51:30.839279628 +0000 UTC m=+0.018773365 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:51:31.172 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:51:31.172 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-04-21T23:51:31.180 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 podman[27583]: 2024-04-21 23:51:31.089294202 +0000 UTC m=+0.268787922 container init 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 podman[27583]: 2024-04-21 23:51:31.092287049 +0000 UTC m=+0.271780767 container start 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 bash[27583]: 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 systemd[1]: Started Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 ceph-mon[27604]: mkfs eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:31.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:31 smithi038 ceph-mon[27604]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.582192s) 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:32.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ef36700 1 Processor -- start 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ef36700 1 -- start start 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ef36700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ef36700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee98107710 con 0x7fee98106d20 2024-04-21T23:51:32.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ccd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ccd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50556/0 (socket says 172.21.15.38:50556) 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.652+0000 7fee9ccd2700 1 -- 172.21.15.38:0/3686590030 learned_addr learned my addr 172.21.15.38:0/3686590030 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ccd2700 1 -- 172.21.15.38:0/3686590030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee98107ed0 con 0x7fee98106d20 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ccd2700 1 --2- 172.21.15.38:0/3686590030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0098a0 tx=0x7fee8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe6e850f9a36858 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee977fe700 1 -- 172.21.15.38:0/3686590030 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee8c004160 con 0x7fee98106d20 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee977fe700 1 -- 172.21.15.38:0/3686590030 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8c004300 con 0x7fee98106d20 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee977fe700 1 -- 172.21.15.38:0/3686590030 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee8c0036a0 con 0x7fee98106d20 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ef36700 1 -- 172.21.15.38:0/3686590030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 msgr2=0x7fee98107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ef36700 1 --2- 172.21.15.38:0/3686590030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0098a0 tx=0x7fee8c009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ef36700 1 -- 172.21.15.38:0/3686590030 shutdown_connections 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ef36700 1 --2- 172.21.15.38:0/3686590030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee98107140 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.653+0000 7fee9ef36700 1 -- 172.21.15.38:0/3686590030 >> 172.21.15.38:0/3686590030 conn(0x7fee981022e0 msgr2=0x7fee98104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:32.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 -- 172.21.15.38:0/3686590030 shutdown_connections 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 -- 172.21.15.38:0/3686590030 wait complete. 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 Processor -- start 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 -- start start 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ef36700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee98108400 con 0x7fee98106d20 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ccd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ccd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50570/0 (socket says 172.21.15.38:50570) 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.654+0000 7fee9ccd2700 1 -- 172.21.15.38:0/1057008962 learned_addr learned my addr 172.21.15.38:0/1057008962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee9ccd2700 1 -- 172.21.15.38:0/1057008962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9819bab0 con 0x7fee98106d20 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee9ccd2700 1 --2- 172.21.15.38:0/1057008962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee8c003db0 con 0x7fee98106d20 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee8c011630 con 0x7fee98106d20 2024-04-21T23:51:32.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9819bd40 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee8c011950 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9819ea30 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fee8c011af0 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.655+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fee8c01c450 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.656+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee98063890 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.659+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fee8c032020 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.732+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fee9802dd40 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.732+0000 7fee95ffb700 1 -- 172.21.15.38:0/1057008962 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fee8c004260 con 0x7fee98106d20 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 msgr2=0x7fee9819b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 --2- 172.21.15.38:0/1057008962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee8c000c00 tx=0x7fee8c004750 comp rx=0 tx=0).stop 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 shutdown_connections 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 --2- 172.21.15.38:0/1057008962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee98106d20 0x7fee9819b570 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 >> 172.21.15.38:0/1057008962 conn(0x7fee981022e0 msgr2=0x7fee98192190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:32.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 shutdown_connections 2024-04-21T23:51:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:31.733+0000 7fee9ef36700 1 -- 172.21.15.38:0/1057008962 wait complete. 2024-04-21T23:51:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-04-21T23:51:32.312 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T23:51:32.416 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: monmap epoch 1 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: last_changed 2024-04-21T23:51:28.138627+0000 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: min_mon_release 19 (squid) 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: election_strategy: 1 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: fsmap 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: mgrmap e1: no daemons active 2024-04-21T23:51:32.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:32 smithi038 ceph-mon[27604]: from='client.? 172.21.15.38:0/1057008962' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T23:51:33.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:33 smithi038 ceph-mon[27604]: from='client.? 172.21.15.38:0/1120674324' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T23:51:33.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:33 smithi038 ceph-mon[27604]: from='client.? 172.21.15.38:0/1120674324' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T23:51:33.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T23:51:33.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9889148700 1 Processor -- start 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9889148700 1 -- start start 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9889148700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9889148700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f988407d090 con 0x7f988407e6a0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9882d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9882d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50576/0 (socket says 172.21.15.38:50576) 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.750+0000 7f9882d9d700 1 -- 172.21.15.38:0/2470294498 learned_addr learned my addr 172.21.15.38:0/2470294498 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9882d9d700 1 -- 172.21.15.38:0/2470294498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988407d210 con 0x7f988407e6a0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9882d9d700 1 --2- 172.21.15.38:0/2470294498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f98740098a0 tx=0x7f9874009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1b898d815c9eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9881d9b700 1 -- 172.21.15.38:0/2470294498 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9874004160 con 0x7f988407e6a0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9881d9b700 1 -- 172.21.15.38:0/2470294498 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9874004300 con 0x7f988407e6a0 2024-04-21T23:51:33.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9881d9b700 1 -- 172.21.15.38:0/2470294498 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98740036a0 con 0x7f988407e6a0 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9889148700 1 -- 172.21.15.38:0/2470294498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 msgr2=0x7f988407cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.751+0000 7f9889148700 1 --2- 172.21.15.38:0/2470294498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f98740098a0 tx=0x7f9874009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 -- 172.21.15.38:0/2470294498 shutdown_connections 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 --2- 172.21.15.38:0/2470294498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988407cb50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 -- 172.21.15.38:0/2470294498 >> 172.21.15.38:0/2470294498 conn(0x7f9884103410 msgr2=0x7f9884105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 -- 172.21.15.38:0/2470294498 shutdown_connections 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 -- 172.21.15.38:0/2470294498 wait complete. 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.752+0000 7f9889148700 1 Processor -- start 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9889148700 1 -- start start 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9889148700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9889148700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f988407d740 con 0x7f988407e6a0 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9882d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:33.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9882d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50588/0 (socket says 172.21.15.38:50588) 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9882d9d700 1 -- 172.21.15.38:0/1120674324 learned_addr learned my addr 172.21.15.38:0/1120674324 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9882d9d700 1 -- 172.21.15.38:0/1120674324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988419dc50 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.753+0000 7f9882d9d700 1 --2- 172.21.15.38:0/1120674324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9874000c00 tx=0x7f9874004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9874003db0 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9874011630 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988419dee0 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9874011950 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98841a0bd0 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9874011af0 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.754+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f987401c450 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.755+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9884063890 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.757+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9874032020 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.828+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f988402dcb0 con 0x7f988407e6a0 2024-04-21T23:51:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.831+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f9874004260 con 0x7f988407e6a0 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.832+0000 7f986bfff700 1 -- 172.21.15.38:0/1120674324 <== mon.0 v2:172.21.15.38:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f987401cb60 con 0x7f988407e6a0 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 msgr2=0x7f988419d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 --2- 172.21.15.38:0/1120674324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9874000c00 tx=0x7f9874004750 comp rx=0 tx=0).stop 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 shutdown_connections 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 --2- 172.21.15.38:0/1120674324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f988407e6a0 0x7f988419d710 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 >> 172.21.15.38:0/1120674324 conn(0x7f9884103410 msgr2=0x7f9884105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 shutdown_connections 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:32.833+0000 7f9889148700 1 -- 172.21.15.38:0/1120674324 wait complete. 2024-04-21T23:51:33.388 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-04-21T23:51:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:34 smithi038 ceph-mon[27604]: from='client.? 172.21.15.38:0/1963048560' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1912dcf700 1 Processor -- start 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1912dcf700 1 -- start start 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1912dcf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1912dcf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f190c108810 con 0x7f190c107eb0 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1910b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:34.466 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1910b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39662/0 (socket says 172.21.15.38:39662) 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.799+0000 7f1910b6b700 1 -- 172.21.15.38:0/507672593 learned_addr learned my addr 172.21.15.38:0/507672593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f1910b6b700 1 -- 172.21.15.38:0/507672593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c108f70 con 0x7f190c107eb0 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f1910b6b700 1 --2- 172.21.15.38:0/507672593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19000098a0 tx=0x7f1900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3641ca6fd99c8f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f190b7fe700 1 -- 172.21.15.38:0/507672593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1900004160 con 0x7f190c107eb0 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f190b7fe700 1 -- 172.21.15.38:0/507672593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1900004300 con 0x7f190c107eb0 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f190b7fe700 1 -- 172.21.15.38:0/507672593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1900011b50 con 0x7f190c107eb0 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f1912dcf700 1 -- 172.21.15.38:0/507672593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 msgr2=0x7f190c1082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f1912dcf700 1 --2- 172.21.15.38:0/507672593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f19000098a0 tx=0x7f1900009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.800+0000 7f1912dcf700 1 -- 172.21.15.38:0/507672593 shutdown_connections 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 --2- 172.21.15.38:0/507672593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c1082d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 -- 172.21.15.38:0/507672593 >> 172.21.15.38:0/507672593 conn(0x7f190c103470 msgr2=0x7f190c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 -- 172.21.15.38:0/507672593 shutdown_connections 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 -- 172.21.15.38:0/507672593 wait complete. 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 Processor -- start 2024-04-21T23:51:34.467 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 -- start start 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1912dcf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f190c10c3b0 con 0x7f190c107eb0 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1910b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1910b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39664/0 (socket says 172.21.15.38:39664) 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.801+0000 7f1910b6b700 1 -- 172.21.15.38:0/1963048560 learned_addr learned my addr 172.21.15.38:0/1963048560 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1910b6b700 1 -- 172.21.15.38:0/1963048560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c19c2f0 con 0x7f190c107eb0 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1910b6b700 1 --2- 172.21.15.38:0/1963048560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1900011670 tx=0x7f1900011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19000039c0 con 0x7f190c107eb0 2024-04-21T23:51:34.468 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190c19c580 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1900011e50 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f190001a5c0 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190c19f270 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f190001ac50 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.802+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1900019700 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.803+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190c063890 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.805+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1900003b60 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.876+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f190c19f7c0 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.877+0000 7f1909ffb700 1 -- 172.21.15.38:0/1963048560 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f19000199a0 con 0x7f190c107eb0 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 msgr2=0x7f190c19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 --2- 172.21.15.38:0/1963048560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1900011670 tx=0x7f1900011b60 comp rx=0 tx=0).stop 2024-04-21T23:51:34.469 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 shutdown_connections 2024-04-21T23:51:34.470 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 --2- 172.21.15.38:0/1963048560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f190c107eb0 0x7f190c19bdb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:34.470 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 >> 172.21.15.38:0/1963048560 conn(0x7f190c103470 msgr2=0x7f190c103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:34.470 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 shutdown_connections 2024-04-21T23:51:34.470 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:33.878+0000 7f1912dcf700 1 -- 172.21.15.38:0/1963048560 wait complete. 2024-04-21T23:51:34.470 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-04-21T23:51:34.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:34 smithi038 systemd[1]: Stopping Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:51:34.984 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:34 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a[27600]: 2024-04-21T23:51:34.735+0000 7f1b61ce1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:51:34.985 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:34 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a[27600]: 2024-04-21T23:51:34.735+0000 7f1b61ce1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T23:51:35.279 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:34 smithi038 podman[27849]: 2024-04-21 23:51:34.991737996 +0000 UTC m=+0.351732533 container died 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T23:51:35.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 podman[27849]: 2024-04-21 23:51:35.394218893 +0000 UTC m=+0.754213425 container cleanup 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:51:35.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 bash[27849]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a 2024-04-21T23:51:35.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 podman[27861]: 2024-04-21 23:51:35.594948409 +0000 UTC m=+0.606414251 container remove 95d4281fb2746d809be5166f04bfd2ea932ca33238ea038b08d932e67ffa94f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:51:35.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 systemd[1]: ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T23:51:35.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 systemd[1]: Stopped Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:51:35.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 systemd[1]: Starting Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:51:36.161 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 podman[27953]: 2024-04-21 23:51:35.92066124 +0000 UTC m=+0.083871057 container create 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:35 smithi038 podman[27953]: 2024-04-21 23:51:35.855415987 +0000 UTC m=+0.018625824 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 podman[27953]: 2024-04-21 23:51:36.098027029 +0000 UTC m=+0.261236855 container init 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 podman[27953]: 2024-04-21 23:51:36.10120563 +0000 UTC m=+0.264415450 container start 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: pidfile_write: ignore empty --pid-file 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: load: jerasure load: lrc 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: RocksDB version: 7.9.2 2024-04-21T23:51:36.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Git sha 0 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: DB SUMMARY 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: DB Session ID: F4K1RG60T05DYEP104VK 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: CURRENT file: CURRENT 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105778 ; 2024-04-21T23:51:36.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.error_if_exists: 0 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.create_if_missing: 0 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.paranoid_checks: 1 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.env: 0x56015b295c60 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.info_log: 0x56015ce37aa0 2024-04-21T23:51:36.193 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.statistics: (nil) 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.use_fsync: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_log_file_size: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_fallocate: 1 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.use_direct_reads: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T23:51:36.194 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.db_log_dir: 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.wal_dir: 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.write_buffer_manager: 0x56015ce0a1e0 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T23:51:36.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.unordered_write: 0 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.row_cache: None 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.wal_filter: None 2024-04-21T23:51:36.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.two_write_queues: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.wal_compression: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.atomic_flush: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.log_readahead_size: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T23:51:36.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_background_jobs: 2 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_background_compactions: -1 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_subcompactions: 1 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T23:51:36.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_open_files: -1 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_background_flushes: -1 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Compression algorithms supported: 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kZSTD supported: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kXpressCompression supported: 0 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kZlibCompression supported: 1 2024-04-21T23:51:36.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kSnappyCompression supported: 1 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kLZ4Compression supported: 1 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: kBZip2Compression supported: 0 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.merge_operator: 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_filter: None 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T23:51:36.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56015ce37ba0) 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x56015ce3b090 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2024-04-21T23:51:36.201 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2024-04-21T23:51:36.202 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2024-04-21T23:51:36.203 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2024-04-21T23:51:36.203 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2024-04-21T23:51:36.203 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2024-04-21T23:51:36.203 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression: NoCompression 2024-04-21T23:51:36.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.num_levels: 7 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T23:51:36.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T23:51:36.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T23:51:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T23:51:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.table_properties_collectors: 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.inplace_update_support: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.bloom_locality: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.max_successive_merges: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T23:51:36.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.ttl: 2592000 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enable_blob_files: false 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.min_blob_size: 0 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T23:51:36.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 902229ac-0012-4abc-9923-1d93f29d0b0b 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713743496130195, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713743496131293, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100762, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98683, "index_size": 297, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14425, "raw_average_key_size": 49, "raw_value_size": 90878, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713743496, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "902229ac-0012-4abc-9923-1d93f29d0b0b", "db_session_id": "F4K1RG60T05DYEP104VK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713743496131378, "job": 1, "event": "recovery_finished"} 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56015cf2c000 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: DB pointer 0x56015cf16000 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T23:51:36.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 100.24 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:51:36.212 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 100.24 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T23:51:36.213 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 12.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 12.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x56015ce3b090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: starting mon.a rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???) e1 preinit fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).mds e1 new map 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).mds e1 print_map 2024-04-21T23:51:36.214 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 bash[27953]: 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 systemd[1]: Started Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T23:51:36.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: monmap epoch 1 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: last_changed 2024-04-21T23:51:28.138627+0000 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: min_mon_release 19 (squid) 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: election_strategy: 1 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: fsmap 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T23:51:36.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:36 smithi038 ceph-mon[27972]: mgrmap e1: no daemons active 2024-04-21T23:51:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.597+0000 7fa24a43a700 1 Processor -- start 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.597+0000 7fa24a43a700 1 -- start start 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa24a43a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa24a43a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa24407d0b0 con 0x7fa24407e670 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa243fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa243fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39672/0 (socket says 172.21.15.38:39672) 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa243fff700 1 -- 172.21.15.38:0/3618231448 learned_addr learned my addr 172.21.15.38:0/3618231448 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.598+0000 7fa243fff700 1 -- 172.21.15.38:0/3618231448 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24407d230 con 0x7fa24407e670 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa243fff700 1 --2- 172.21.15.38:0/3618231448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa2340098a0 tx=0x7fa234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6eb46bfc57435d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:37.213 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa242ffd700 1 -- 172.21.15.38:0/3618231448 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa234004160 con 0x7fa24407e670 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa242ffd700 1 -- 172.21.15.38:0/3618231448 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa234004300 con 0x7fa24407e670 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa242ffd700 1 -- 172.21.15.38:0/3618231448 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa234011b50 con 0x7fa24407e670 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 -- 172.21.15.38:0/3618231448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 msgr2=0x7fa24407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 --2- 172.21.15.38:0/3618231448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa2340098a0 tx=0x7fa234009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 -- 172.21.15.38:0/3618231448 shutdown_connections 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 --2- 172.21.15.38:0/3618231448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24407cb70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 -- 172.21.15.38:0/3618231448 >> 172.21.15.38:0/3618231448 conn(0x7fa244103470 msgr2=0x7fa244105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 -- 172.21.15.38:0/3618231448 shutdown_connections 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.599+0000 7fa24a43a700 1 -- 172.21.15.38:0/3618231448 wait complete. 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa24a43a700 1 Processor -- start 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa24a43a700 1 -- start start 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa24a43a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa24a43a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa24410e460 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa243fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa243fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39688/0 (socket says 172.21.15.38:39688) 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.600+0000 7fa243fff700 1 -- 172.21.15.38:0/739844662 learned_addr learned my addr 172.21.15.38:0/739844662 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa243fff700 1 -- 172.21.15.38:0/739844662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24419e4f0 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa243fff700 1 --2- 172.21.15.38:0/739844662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa234011670 tx=0x7fa234011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2340039c0 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa234011e50 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24419e780 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa23401a5c0 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2441a1470 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa23401ac50 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.601+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa234019700 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.602+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa244063890 con 0x7fa24407e670 2024-04-21T23:51:37.215 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.605+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa234003b60 con 0x7fa24407e670 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.675+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa2441a1640 con 0x7fa24407e670 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.676+0000 7fa2417fa700 1 -- 172.21.15.38:0/739844662 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa234019900 con 0x7fa24407e670 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 msgr2=0x7fa24419dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 --2- 172.21.15.38:0/739844662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa234011670 tx=0x7fa234011b60 comp rx=0 tx=0).stop 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 shutdown_connections 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 --2- 172.21.15.38:0/739844662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa24407e670 0x7fa24419dfb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 >> 172.21.15.38:0/739844662 conn(0x7fa244103470 msgr2=0x7fa244103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 shutdown_connections 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:36.677+0000 7fa24a43a700 1 -- 172.21.15.38:0/739844662 wait complete. 2024-04-21T23:51:37.216 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T23:51:37.238 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:51:37.239 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-04-21T23:51:37.239 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T23:51:37.240 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T23:51:37.504 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x 2024-04-21T23:51:37.504 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T23:51:37.662 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968.target.wants/ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-eab6f840-0039-11ef-bc93-c7b262605968@.service. 2024-04-21T23:51:37.874 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:37 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/739844662' entity='client.admin' 2024-04-21T23:51:37.874 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:37 smithi038 systemd[1]: Starting Ceph mgr.x for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:51:38.168 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:37 smithi038 podman[28182]: 2024-04-21 23:51:37.881158642 +0000 UTC m=+0.093003999 container create aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True) 2024-04-21T23:51:38.169 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:37 smithi038 podman[28182]: 2024-04-21 23:51:37.806566957 +0000 UTC m=+0.018412282 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T23:51:38.169 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 podman[28182]: 2024-04-21 23:51:38.092055689 +0000 UTC m=+0.303901001 container init aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, ceph=True) 2024-04-21T23:51:38.169 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 podman[28182]: 2024-04-21 23:51:38.094707996 +0000 UTC m=+0.306553312 container start aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T23:51:38.169 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 bash[28182]: aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c 2024-04-21T23:51:38.183 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:51:38.183 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-21T23:51:38.191 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:51:38.191 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:51:38.191 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-21T23:51:38.191 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T23:51:38.200 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:51:38.200 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-21T23:51:38.208 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:51:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:51:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-21T23:51:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T23:51:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-04-21T23:51:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-04-21T23:51:38.431 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 systemd[1]: Started Ceph mgr.x for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:51:38.431 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:38.260+0000 7fdf58f39200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:51:38.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:38.455+0000 7fdf58f39200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:51:38.709 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:38.561+0000 7fdf58f39200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:51:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2186147369' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T23:51:39.209 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:38.893+0000 7fdf58f39200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:51:39.209 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:38 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:38.981+0000 7fdf58f39200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:51:40.585 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:40 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:40.239+0000 7fdf58f39200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:51:40.585 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:40 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:40.414+0000 7fdf58f39200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:51:40.585 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:40 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:40.590+0000 7fdf58f39200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:51:41.190 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:40 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:40.887+0000 7fdf58f39200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:51:41.190 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:40 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:40.984+0000 7fdf58f39200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:51:41.458 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:41 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:41.195+0000 7fdf58f39200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:51:41.459 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:41 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:41.394+0000 7fdf58f39200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:51:42.454 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:42.022+0000 7fdf58f39200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-21T23:51:42.519 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T23:51:42.520 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T23:51:42.521 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T23:51:31.160221+0000", 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:51:42.523 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.671+0000 7ff0c9b9b700 1 Processor -- start 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.671+0000 7ff0c9b9b700 1 -- start start 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.671+0000 7ff0c9b9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.671+0000 7ff0c9b9b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c4108790 con 0x7ff0c4107e30 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.671+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39710/0 (socket says 172.21.15.38:39710) 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c37fe700 1 -- 172.21.15.38:0/700922393 learned_addr learned my addr 172.21.15.38:0/700922393 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c37fe700 1 -- 172.21.15.38:0/700922393 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c4108910 con 0x7ff0c4107e30 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c37fe700 1 --2- 172.21.15.38:0/700922393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40098a0 tx=0x7ff0b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc6a87423a163e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c27fc700 1 -- 172.21.15.38:0/700922393 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0b4004160 con 0x7ff0c4107e30 2024-04-21T23:51:42.524 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c27fc700 1 -- 172.21.15.38:0/700922393 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff0b4004300 con 0x7ff0c4107e30 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.672+0000 7ff0c27fc700 1 -- 172.21.15.38:0/700922393 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0b4011b50 con 0x7ff0c4107e30 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/700922393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 msgr2=0x7ff0c4108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 --2- 172.21.15.38:0/700922393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40098a0 tx=0x7ff0b4009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/700922393 shutdown_connections 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 --2- 172.21.15.38:0/700922393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c4108250 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/700922393 >> 172.21.15.38:0/700922393 conn(0x7ff0c4103450 msgr2=0x7ff0c41058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/700922393 shutdown_connections 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.673+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/700922393 wait complete. 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c9b9b700 1 Processor -- start 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c9b9b700 1 -- start start 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c9b9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c9b9b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410c2a0 con 0x7ff0c4107e30 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:42.525 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39726/0 (socket says 172.21.15.38:39726) 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.674+0000 7ff0c37fe700 1 -- 172.21.15.38:0/2186147369 learned_addr learned my addr 172.21.15.38:0/2186147369 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c37fe700 1 -- 172.21.15.38:0/2186147369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c419c2d0 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c37fe700 1 --2- 172.21.15.38:0/2186147369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4011b60 tx=0x7ff0b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0b4003b20 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff0b4003cc0 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c419c560 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0b4024840 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c419f210 con 0x7ff0c4107e30 2024-04-21T23:51:42.526 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff0b4022020 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.675+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff0b4011e00 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.676+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c4063890 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.678+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff0b402d3f0 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.752+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff0c419f7d0 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.753+0000 7ff0c0ff9700 1 -- 172.21.15.38:0/2186147369 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff0b4032030 con 0x7ff0c4107e30 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 msgr2=0x7ff0c419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 --2- 172.21.15.38:0/2186147369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4011b60 tx=0x7ff0b4011c40 comp rx=0 tx=0).stop 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 shutdown_connections 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 --2- 172.21.15.38:0/2186147369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0c4107e30 0x7ff0c419bd90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 >> 172.21.15.38:0/2186147369 conn(0x7ff0c4103450 msgr2=0x7ff0c4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 shutdown_connections 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:38.754+0000 7ff0c9b9b700 1 -- 172.21.15.38:0/2186147369 wait complete. 2024-04-21T23:51:42.527 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-04-21T23:51:42.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:42.460+0000 7fdf58f39200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:51:42.709 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:42.560+0000 7fdf58f39200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:51:43.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:42.769+0000 7fdf58f39200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:51:43.677 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:43 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:43.336+0000 7fdf58f39200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:51:43.677 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:43 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:43.431+0000 7fdf58f39200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:51:43.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:43 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:43.683+0000 7fdf58f39200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:51:43.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:43 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:43.823+0000 7fdf58f39200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:51:43.959 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:43 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:43.921+0000 7fdf58f39200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:51:44.458 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:44 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:44.040+0000 7fdf58f39200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:51:44.458 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:44 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:44.134+0000 7fdf58f39200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:51:45.248 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:44 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:44.981+0000 7fdf58f39200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:51:45.248 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:45.202+0000 7fdf58f39200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:51:45.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: Activating manager daemon x 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: mgrmap e2: x(active, starting, since 0.00315837s) 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: Manager daemon x is now available 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' 2024-04-21T23:51:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' 2024-04-21T23:51:45.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:45 smithi038 ceph-mon[27972]: from='mgr.14100 172.21.15.38:0/268054774' entity='mgr.x' 2024-04-21T23:51:47.283 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:47 smithi038 ceph-mon[27972]: mgrmap e3: x(active, since 1.00772s) 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T23:51:48.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T23:51:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T23:51:48.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T23:51:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T23:51:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T23:51:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T23:51:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T23:51:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T23:51:48.190 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T23:51:31.160221+0000", 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.326+0000 7efef0c0e700 1 Processor -- start 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.326+0000 7efef0c0e700 1 -- start start 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efef0c0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efef0c0e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeec07d090 con 0x7efeec07e650 2024-04-21T23:51:48.191 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efeea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efeea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39778/0 (socket says 172.21.15.38:39778) 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efeea59c700 1 -- 172.21.15.38:0/3607940688 learned_addr learned my addr 172.21.15.38:0/3607940688 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efeea59c700 1 -- 172.21.15.38:0/3607940688 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec07d210 con 0x7efeec07e650 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efeea59c700 1 --2- 172.21.15.38:0/3607940688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efee00098a0 tx=0x7efee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2924c607b6a2ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efee959a700 1 -- 172.21.15.38:0/3607940688 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee0004160 con 0x7efeec07e650 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efee959a700 1 -- 172.21.15.38:0/3607940688 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efee0004300 con 0x7efeec07e650 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.327+0000 7efee959a700 1 -- 172.21.15.38:0/3607940688 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee0011b50 con 0x7efeec07e650 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 -- 172.21.15.38:0/3607940688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 msgr2=0x7efeec07cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3607940688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efee00098a0 tx=0x7efee0009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 -- 172.21.15.38:0/3607940688 shutdown_connections 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3607940688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec07cb50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:48.192 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 -- 172.21.15.38:0/3607940688 >> 172.21.15.38:0/3607940688 conn(0x7efeec103470 msgr2=0x7efeec105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 -- 172.21.15.38:0/3607940688 shutdown_connections 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.328+0000 7efef0c0e700 1 -- 172.21.15.38:0/3607940688 wait complete. 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.329+0000 7efef0c0e700 1 Processor -- start 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.329+0000 7efef0c0e700 1 -- start start 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.329+0000 7efef0c0e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.329+0000 7efef0c0e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeec10e420 con 0x7efeec07e650 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.329+0000 7efeea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efeea59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39792/0 (socket says 172.21.15.38:39792) 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efeea59c700 1 -- 172.21.15.38:0/3613635202 learned_addr learned my addr 172.21.15.38:0/3613635202 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efeea59c700 1 -- 172.21.15.38:0/3613635202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec19e4d0 con 0x7efeec07e650 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efeea59c700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efee0011670 tx=0x7efee0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:48.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee00039c0 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efee0011e50 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeec19e760 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee001a5c0 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.330+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeec1a1450 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.331+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7efee001a8c0 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.331+0000 7efedb7fe700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 0x7efed403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.331+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efee004d4b0 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.331+0000 7efee9d9b700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 0x7efed403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.331+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeec063890 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.332+0000 7efee9d9b700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 0x7efed403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efedc006fd0 tx=0x7efedc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.337+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efee0019c70 con 0x7efeec07e650 2024-04-21T23:51:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.626+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efeec1a1620 con 0x7efeec07e650 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.627+0000 7efedb7fe700 1 -- 172.21.15.38:0/3613635202 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efee0032030 con 0x7efeec07e650 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 msgr2=0x7efed403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 0x7efed403b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efedc006fd0 tx=0x7efedc006e40 comp rx=0 tx=0).stop 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 msgr2=0x7efeec19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efee0011670 tx=0x7efee0011b60 comp rx=0 tx=0).stop 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 shutdown_connections 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7efed4038fb0 0x7efed403b470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 --2- 172.21.15.38:0/3613635202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeec07e650 0x7efeec19df90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 >> 172.21.15.38:0/3613635202 conn(0x7efeec103470 msgr2=0x7efeec103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 shutdown_connections 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:47.630+0000 7efef0c0e700 1 -- 172.21.15.38:0/3613635202 wait complete. 2024-04-21T23:51:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-04-21T23:51:48.539 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:48 smithi038 ceph-mon[27972]: mgrmap e4: x(active, since 2s) 2024-04-21T23:51:48.540 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:48 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3613635202' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T23:51:49.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:49.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T23:51:49.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:51:49.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T23:51:49.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T23:51:49.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd82d5cd700 1 Processor -- start 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd82d5cd700 1 -- start start 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd82d5cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd82d5cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd828108810 con 0x7fd828107eb0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd826ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd826ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39800/0 (socket says 172.21.15.38:39800) 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.569+0000 7fd826ffd700 1 -- 172.21.15.38:0/1001986792 learned_addr learned my addr 172.21.15.38:0/1001986792 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.570+0000 7fd826ffd700 1 -- 172.21.15.38:0/1001986792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd828108f70 con 0x7fd828107eb0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.570+0000 7fd826ffd700 1 --2- 172.21.15.38:0/1001986792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8180098a0 tx=0x7fd818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e05b11cf75f32c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.570+0000 7fd825ffb700 1 -- 172.21.15.38:0/1001986792 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd818004160 con 0x7fd828107eb0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.570+0000 7fd825ffb700 1 -- 172.21.15.38:0/1001986792 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd818004300 con 0x7fd828107eb0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.570+0000 7fd825ffb700 1 -- 172.21.15.38:0/1001986792 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd818011b50 con 0x7fd828107eb0 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1001986792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 msgr2=0x7fd8281082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:49.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1001986792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8180098a0 tx=0x7fd818009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1001986792 shutdown_connections 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1001986792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd8281082d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1001986792 >> 172.21.15.38:0/1001986792 conn(0x7fd828103470 msgr2=0x7fd828105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1001986792 shutdown_connections 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1001986792 wait complete. 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.571+0000 7fd82d5cd700 1 Processor -- start 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd82d5cd700 1 -- start start 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd82d5cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd82d5cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd82810c3b0 con 0x7fd828107eb0 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd826ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd826ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39808/0 (socket says 172.21.15.38:39808) 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd826ffd700 1 -- 172.21.15.38:0/1154297758 learned_addr learned my addr 172.21.15.38:0/1154297758 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd826ffd700 1 -- 172.21.15.38:0/1154297758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82819c2f0 con 0x7fd828107eb0 2024-04-21T23:51:49.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd826ffd700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd818011670 tx=0x7fd818011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.572+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8180039c0 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.573+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82819c580 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.573+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd818011e50 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.573+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd81801a5c0 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.573+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd82819f270 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.573+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fd81801a8c0 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.574+0000 7fd80ffff700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 0x7fd81003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.574+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd81804d420 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.574+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd828063890 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.574+0000 7fd8267fc700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 0x7fd81003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.574+0000 7fd8267fc700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 0x7fd81003b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd81c006fd0 tx=0x7fd81c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.579+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd818019c70 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.782+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd82819f440 con 0x7fd828107eb0 2024-04-21T23:51:49.408 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.783+0000 7fd80ffff700 1 -- 172.21.15.38:0/1154297758 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7fd81802cc00 con 0x7fd828107eb0 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 msgr2=0x7fd81003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 0x7fd81003b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd81c006fd0 tx=0x7fd81c006e40 comp rx=0 tx=0).stop 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 msgr2=0x7fd82819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd818011670 tx=0x7fd818011b60 comp rx=0 tx=0).stop 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 shutdown_connections 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fd810038fb0 0x7fd81003b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 --2- 172.21.15.38:0/1154297758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd828107eb0 0x7fd82819bdb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 >> 172.21.15.38:0/1154297758 conn(0x7fd828103470 msgr2=0x7fd828103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 shutdown_connections 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:48.785+0000 7fd82d5cd700 1 -- 172.21.15.38:0/1154297758 wait complete. 2024-04-21T23:51:49.409 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-04-21T23:51:49.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:49 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1154297758' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T23:51:50.394 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3817386727' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T23:51:50.394 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setuser ceph since I am not root 2024-04-21T23:51:50.394 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setgroup ceph since I am not root 2024-04-21T23:51:50.394 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:50.400+0000 7f97d5920200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:51:50.696 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:50.596+0000 7f97d5920200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:51:50.736 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.837+0000 7fccbf814700 1 Processor -- start 2024-04-21T23:51:50.741 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.837+0000 7fccbf814700 1 -- start start 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.837+0000 7fccbf814700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.837+0000 7fccbf814700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fccb807d0d0 con 0x7fccb807e690 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccbd5b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccbd5b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39818/0 (socket says 172.21.15.38:39818) 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccbd5b0700 1 -- 172.21.15.38:0/2183306840 learned_addr learned my addr 172.21.15.38:0/2183306840 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccbd5b0700 1 -- 172.21.15.38:0/2183306840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb807d250 con 0x7fccb807e690 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccbd5b0700 1 --2- 172.21.15.38:0/2183306840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009e80 tx=0x7fccb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99da796345476a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccb7fff700 1 -- 172.21.15.38:0/2183306840 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0017070 con 0x7fccb807e690 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccb7fff700 1 -- 172.21.15.38:0/2183306840 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccb000b8c0 con 0x7fccb807e690 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.838+0000 7fccb7fff700 1 -- 172.21.15.38:0/2183306840 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb00045c0 con 0x7fccb807e690 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 -- 172.21.15.38:0/2183306840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 msgr2=0x7fccb807cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:50.742 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 --2- 172.21.15.38:0/2183306840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009e80 tx=0x7fccb000b330 comp rx=0 tx=0).stop 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 -- 172.21.15.38:0/2183306840 shutdown_connections 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 --2- 172.21.15.38:0/2183306840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb807cb90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 -- 172.21.15.38:0/2183306840 >> 172.21.15.38:0/2183306840 conn(0x7fccb8103470 msgr2=0x7fccb8105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 -- 172.21.15.38:0/2183306840 shutdown_connections 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.839+0000 7fccbf814700 1 -- 172.21.15.38:0/2183306840 wait complete. 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbf814700 1 Processor -- start 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbf814700 1 -- start start 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbf814700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbf814700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fccb810e440 con 0x7fccb807e690 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbd5b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbd5b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39822/0 (socket says 172.21.15.38:39822) 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbd5b0700 1 -- 172.21.15.38:0/3817386727 learned_addr learned my addr 172.21.15.38:0/3817386727 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:50.743 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbd5b0700 1 -- 172.21.15.38:0/3817386727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb819e4f0 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.840+0000 7fccbd5b0700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccb00072a0 tx=0x7fccb000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0020020 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb819e780 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccb0003b50 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0017040 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb81a1420 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.841+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fccb0023020 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.842+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb8063890 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.842+0000 7fccb67fc700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 0x7fcca003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.842+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fccb000f050 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.842+0000 7fccbcdaf700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 0x7fcca003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.842+0000 7fccbcdaf700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 0x7fcca003b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fccac006fd0 tx=0x7fccac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:49.847+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fccb002a020 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.112+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fccb81a15f0 con 0x7fccb807e690 2024-04-21T23:51:50.744 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.226+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fccb81a15f0 con 0x7fccb807e690 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.228+0000 7fccb67fc700 1 -- 172.21.15.38:0/3817386727 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fccb0011390 con 0x7fccb807e690 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 msgr2=0x7fcca003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 0x7fcca003b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fccac006fd0 tx=0x7fccac006e40 comp rx=0 tx=0).stop 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 msgr2=0x7fccb819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccb00072a0 tx=0x7fccb000bd50 comp rx=0 tx=0).stop 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 shutdown_connections 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fcca0038fb0 0x7fcca003b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 --2- 172.21.15.38:0/3817386727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccb807e690 0x7fccb819dfb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 >> 172.21.15.38:0/3817386727 conn(0x7fccb8103470 msgr2=0x7fccb8103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 shutdown_connections 2024-04-21T23:51:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:50.229+0000 7fccbf814700 1 -- 172.21.15.38:0/3817386727 wait complete. 2024-04-21T23:51:50.959 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:50.708+0000 7f97d5920200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:51:51.222 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:51 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:51.048+0000 7f97d5920200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:51:51.223 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:51 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:51.137+0000 7f97d5920200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:51:51.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3817386727' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T23:51:51.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:51 smithi038 ceph-mon[27972]: mgrmap e5: x(active, since 5s) 2024-04-21T23:51:52.140 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:51:52.140 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.239+0000 7fef3cc62700 1 Processor -- start 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.239+0000 7fef3cc62700 1 -- start start 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3cc62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3cc62700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef3807c800 con 0x7fef3807dd50 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39848/0 (socket says 172.21.15.38:39848) 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3659c700 1 -- 172.21.15.38:0/1634180204 learned_addr learned my addr 172.21.15.38:0/1634180204 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3659c700 1 -- 172.21.15.38:0/1634180204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3807c980 con 0x7fef3807dd50 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.240+0000 7fef3659c700 1 --2- 172.21.15.38:0/1634180204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fef280098a0 tx=0x7fef28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0d3e6a6ec538b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3559a700 1 -- 172.21.15.38:0/1634180204 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef28004160 con 0x7fef3807dd50 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3559a700 1 -- 172.21.15.38:0/1634180204 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef28004300 con 0x7fef3807dd50 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3559a700 1 -- 172.21.15.38:0/1634180204 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef28011b50 con 0x7fef3807dd50 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 -- 172.21.15.38:0/1634180204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 msgr2=0x7fef3807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 --2- 172.21.15.38:0/1634180204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fef280098a0 tx=0x7fef28009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 -- 172.21.15.38:0/1634180204 shutdown_connections 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 --2- 172.21.15.38:0/1634180204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3807c2c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 -- 172.21.15.38:0/1634180204 >> 172.21.15.38:0/1634180204 conn(0x7fef381034a0 msgr2=0x7fef381058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.241+0000 7fef3cc62700 1 -- 172.21.15.38:0/1634180204 shutdown_connections 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.242+0000 7fef3cc62700 1 -- 172.21.15.38:0/1634180204 wait complete. 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.242+0000 7fef3cc62700 1 Processor -- start 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.242+0000 7fef3cc62700 1 -- start start 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.242+0000 7fef3cc62700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3cc62700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef3810e310 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39852/0 (socket says 172.21.15.38:39852) 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3659c700 1 -- 172.21.15.38:0/839800885 learned_addr learned my addr 172.21.15.38:0/839800885 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3659c700 1 -- 172.21.15.38:0/839800885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3819e4c0 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3659c700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef28011b60 tx=0x7fef28011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef28003b20 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef28003cc0 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3819e750 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef28024840 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.243+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef381a1440 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.244+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef38063890 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.245+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fef28022020 con 0x7fef3807dd50 2024-04-21T23:51:52.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.245+0000 7fef277fe700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 0x7fef2003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.245+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fef28011e00 con 0x7fef3807dd50 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.245+0000 7fef35d9b700 1 -- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 msgr2=0x7fef2003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.245+0000 7fef35d9b700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 0x7fef2003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.250+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fef2800ba00 con 0x7fef3807dd50 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.446+0000 7fef35d9b700 1 -- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 msgr2=0x7fef2003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.446+0000 7fef35d9b700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 0x7fef2003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.513+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fef381a1610 con 0x7fef3807dd50 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.513+0000 7fef277fe700 1 -- 172.21.15.38:0/839800885 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fef2800bc60 con 0x7fef3807dd50 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.515+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 msgr2=0x7fef2003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.515+0000 7fef3cc62700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 0x7fef2003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.515+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 msgr2=0x7fef3819df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.515+0000 7fef3cc62700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef28011b60 tx=0x7fef28011c40 comp rx=0 tx=0).stop 2024-04-21T23:51:52.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 shutdown_connections 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fef20039050 0x7fef2003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 --2- 172.21.15.38:0/839800885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef3807dd50 0x7fef3819df80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 >> 172.21.15.38:0/839800885 conn(0x7fef381034a0 msgr2=0x7fef38104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 shutdown_connections 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:51.516+0000 7fef3cc62700 1 -- 172.21.15.38:0/839800885 wait complete. 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-21T23:51:52.145 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-04-21T23:51:52.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/839800885' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T23:51:52.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:52 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:52.405+0000 7f97d5920200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:51:52.709 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:52 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:52.582+0000 7f97d5920200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:51:53.052 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:52 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:52.761+0000 7f97d5920200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:51:53.052 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:53.058+0000 7f97d5920200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:51:53.431 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:53.154+0000 7f97d5920200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:51:53.697 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:53.364+0000 7f97d5920200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:51:53.697 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:53.562+0000 7f97d5920200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:51:54.458 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:54 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:54.195+0000 7f97d5920200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:51:54.933 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:54 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:54.631+0000 7f97d5920200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:51:54.933 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:54 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:54.731+0000 7f97d5920200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:51:54.934 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:54 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:54.939+0000 7f97d5920200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:51:55.845 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:55 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:55.504+0000 7f97d5920200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:51:55.845 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:55 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:55.598+0000 7f97d5920200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:51:56.203 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:55 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:55.851+0000 7f97d5920200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:51:56.203 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:55 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:55.989+0000 7f97d5920200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:51:56.203 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:56 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:56.089+0000 7f97d5920200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:51:56.203 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:56 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:56.209+0000 7f97d5920200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:51:56.458 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:56 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:56.303+0000 7f97d5920200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:51:57.414 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:57.145+0000 7f97d5920200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:51:57.414 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:51:57.367+0000 7f97d5920200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:51:57.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: Active manager daemon x restarted 2024-04-21T23:51:57.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: Activating manager daemon x 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: mgrmap e6: x(active, starting, since 0.0051308s) 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:51:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: Manager daemon x is now available 2024-04-21T23:51:57.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:51:57.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:51:57.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:51:57.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:57 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:51:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: Found migration_current of "None". Setting to last migration. 2024-04-21T23:51:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:51:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:51:58.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:51:58.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:51:58.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:58 smithi038 ceph-mon[27972]: mgrmap e7: x(active, since 1.00826s) 2024-04-21T23:51:59.080 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:51:59.080 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T23:51:59.080 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9c97a700 1 Processor -- start 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9c97a700 1 -- start start 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9c97a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9c97a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf98108cb0 con 0x7fbf981082c0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59702/0 (socket says 172.21.15.38:59702) 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9659c700 1 -- 172.21.15.38:0/1842833890 learned_addr learned my addr 172.21.15.38:0/1842833890 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.703+0000 7fbf9659c700 1 -- 172.21.15.38:0/1842833890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf98108e30 con 0x7fbf981082c0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9659c700 1 --2- 172.21.15.38:0/1842833890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf880098a0 tx=0x7fbf88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6bec3b5b18815f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9559a700 1 -- 172.21.15.38:0/1842833890 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf88004160 con 0x7fbf981082c0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9559a700 1 -- 172.21.15.38:0/1842833890 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbf88004300 con 0x7fbf981082c0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9559a700 1 -- 172.21.15.38:0/1842833890 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf88011b50 con 0x7fbf981082c0 2024-04-21T23:51:59.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9c97a700 1 -- 172.21.15.38:0/1842833890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 msgr2=0x7fbf981086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.704+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/1842833890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf880098a0 tx=0x7fbf88009bb0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.705+0000 7fbf9c97a700 1 -- 172.21.15.38:0/1842833890 shutdown_connections 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.705+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/1842833890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf981086e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.705+0000 7fbf9c97a700 1 -- 172.21.15.38:0/1842833890 >> 172.21.15.38:0/1842833890 conn(0x7fbf98101a90 msgr2=0x7fbf98103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.705+0000 7fbf9c97a700 1 -- 172.21.15.38:0/1842833890 shutdown_connections 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.705+0000 7fbf9c97a700 1 -- 172.21.15.38:0/1842833890 wait complete. 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9c97a700 1 Processor -- start 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9c97a700 1 -- start start 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9c97a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:59.082 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9c97a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810b240 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59718/0 (socket says 172.21.15.38:59718) 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9659c700 1 -- 172.21.15.38:0/869577387 learned_addr learned my addr 172.21.15.38:0/869577387 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9659c700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9819c940 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.706+0000 7fbf9659c700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf88011670 tx=0x7fbf88011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.707+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf880039c0 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.707+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbf88011e50 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.707+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf981a0f50 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.707+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf8801a5c0 con 0x7fbf981082c0 2024-04-21T23:51:59.083 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.707+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf981a12f0 con 0x7fbf981082c0 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fbf8801a8c0 con 0x7fbf981082c0 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf8f7fe700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf98069390 con 0x7fbf84039000 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbf8804d490 con 0x7fbf981082c0 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf95d9b700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.708+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.909+0000 7fbf95d9b700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:53.909+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:54.310+0000 7fbf95d9b700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:54.310+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:55.111+0000 7fbf95d9b700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:55.111+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T23:51:59.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:56.712+0000 7fbf95d9b700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/71250327 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:56.712+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:57.372+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fbf8802d910 con 0x7fbf981082c0 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:57.372+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 msgr2=0x7fbf8403b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:57.372+0000 7fbf8f7fe700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.375+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fbf8804f880 con 0x7fbf981082c0 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.375+0000 7fbf8f7fe700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 0x7fbf8403eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.375+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf88021e70 con 0x7fbf8403c740 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.375+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 0x7fbf8403eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.376+0000 7fbf95d9b700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 0x7fbf8403eb30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf80003a10 tx=0x7fbf800092b0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.376+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fbf88021e70 con 0x7fbf8403c740 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbf9810b3c0 con 0x7fbf8403c740 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf8f7fe700 1 -- 172.21.15.38:0/869577387 <== mgr.14116 v2:172.21.15.38:6800/2824973701 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbf9810b3c0 con 0x7fbf8403c740 2024-04-21T23:51:59.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 msgr2=0x7fbf8403eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 0x7fbf8403eb30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf80003a10 tx=0x7fbf800092b0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 msgr2=0x7fbf98106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbf88011670 tx=0x7fbf88011b60 comp rx=0 tx=0).stop 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 shutdown_connections 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fbf8403c740 0x7fbf8403eb30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf981082c0 0x7fbf98106970 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 --2- 172.21.15.38:0/869577387 >> [v2:172.21.15.38:6800/71250327,v1:172.21.15.38:6801/71250327] conn(0x7fbf84039000 0x7fbf8403b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 >> 172.21.15.38:0/869577387 conn(0x7fbf98101a90 msgr2=0x7fbf981025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 shutdown_connections 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:58.388+0000 7fbf9c97a700 1 -- 172.21.15.38:0/869577387 wait complete. 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-04-21T23:51:59.086 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-04-21T23:51:59.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:59 smithi038 ceph-mon[27972]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T23:51:59.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:51:59 smithi038 ceph-mon[27972]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: [21/Apr/2024:23:51:59] ENGINE Bus STARTING 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: mgrmap e8: x(active, since 2s) 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: [21/Apr/2024:23:51:59] ENGINE Serving on https://172.21.15.38:7150 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: [21/Apr/2024:23:51:59] ENGINE Serving on http://172.21.15.38:8765 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: [21/Apr/2024:23:51:59] ENGINE Bus STARTED 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:00.683 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:00 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51fb76700 1 Processor -- start 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51fb76700 1 -- start start 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51fb76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51fb76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb518108810 con 0x7fb518107eb0 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59786/0 (socket says 172.21.15.38:59786) 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.884+0000 7fb51d912700 1 -- 172.21.15.38:0/3839130368 learned_addr learned my addr 172.21.15.38:0/3839130368 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51d912700 1 -- 172.21.15.38:0/3839130368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb518108f70 con 0x7fb518107eb0 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51d912700 1 --2- 172.21.15.38:0/3839130368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5140098a0 tx=0x7fb514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5664597bd2cdb10f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:00.702 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51c910700 1 -- 172.21.15.38:0/3839130368 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb514004160 con 0x7fb518107eb0 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51c910700 1 -- 172.21.15.38:0/3839130368 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb514004300 con 0x7fb518107eb0 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51c910700 1 -- 172.21.15.38:0/3839130368 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb514011b50 con 0x7fb518107eb0 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51fb76700 1 -- 172.21.15.38:0/3839130368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 msgr2=0x7fb5181082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51fb76700 1 --2- 172.21.15.38:0/3839130368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5140098a0 tx=0x7fb514009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51fb76700 1 -- 172.21.15.38:0/3839130368 shutdown_connections 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51fb76700 1 --2- 172.21.15.38:0/3839130368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb5181082d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.885+0000 7fb51fb76700 1 -- 172.21.15.38:0/3839130368 >> 172.21.15.38:0/3839130368 conn(0x7fb518103470 msgr2=0x7fb518105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 -- 172.21.15.38:0/3839130368 shutdown_connections 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 -- 172.21.15.38:0/3839130368 wait complete. 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 Processor -- start 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 -- start start 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51fb76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb51810c3b0 con 0x7fb518107eb0 2024-04-21T23:52:00.703 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51d912700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59792/0 (socket says 172.21.15.38:59792) 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.886+0000 7fb51d912700 1 -- 172.21.15.38:0/799162903 learned_addr learned my addr 172.21.15.38:0/799162903 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb51d912700 1 -- 172.21.15.38:0/799162903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51819c2f0 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb51d912700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb514011670 tx=0x7fb514011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5140039c0 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb514011e50 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51819c580 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb51401a5c0 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.887+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51819f270 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.888+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb518063890 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.888+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb514024430 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.888+0000 7fb50effd700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 0x7fb50403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.888+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb51404d390 con 0x7fb518107eb0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.888+0000 7fb51d111700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 0x7fb50403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:00.704 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.889+0000 7fb51d111700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 0x7fb50403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb508006fd0 tx=0x7fb508006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:51:59.894+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb51402d3f0 con 0x7fb518107eb0 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.120+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb5181046b0 con 0x7fb504039050 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.126+0000 7fb50effd700 1 -- 172.21.15.38:0/799162903 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5181046b0 con 0x7fb504039050 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 msgr2=0x7fb50403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 0x7fb50403b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb508006fd0 tx=0x7fb508006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 msgr2=0x7fb51819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb514011670 tx=0x7fb514011b60 comp rx=0 tx=0).stop 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 shutdown_connections 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fb504039050 0x7fb50403b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 --2- 172.21.15.38:0/799162903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb518107eb0 0x7fb51819bdb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 >> 172.21.15.38:0/799162903 conn(0x7fb518103470 msgr2=0x7fb518103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 shutdown_connections 2024-04-21T23:52:00.705 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:00.129+0000 7fb51fb76700 1 -- 172.21.15.38:0/799162903 wait complete. 2024-04-21T23:52:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T23:52:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.099+0000 7f14cb7eb700 1 Processor -- start 2024-04-21T23:52:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.099+0000 7f14cb7eb700 1 -- start start 2024-04-21T23:52:01.852 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.099+0000 7f14cb7eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.099+0000 7f14cb7eb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14c407c670 con 0x7f14c41088f0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14c9587700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14c9587700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59800/0 (socket says 172.21.15.38:59800) 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14c9587700 1 -- 172.21.15.38:0/2500734240 learned_addr learned my addr 172.21.15.38:0/2500734240 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14c9587700 1 -- 172.21.15.38:0/2500734240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c410b220 con 0x7f14c41088f0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14c9587700 1 --2- 172.21.15.38:0/2500734240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14c00098a0 tx=0x7f14c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93fba7e88de6575 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14bbfff700 1 -- 172.21.15.38:0/2500734240 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14c0004160 con 0x7f14c41088f0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14bbfff700 1 -- 172.21.15.38:0/2500734240 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14c0004300 con 0x7f14c41088f0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.100+0000 7f14bbfff700 1 -- 172.21.15.38:0/2500734240 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14c0011b50 con 0x7f14c41088f0 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 -- 172.21.15.38:0/2500734240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 msgr2=0x7f14c410ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:01.853 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/2500734240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f14c00098a0 tx=0x7f14c0009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 -- 172.21.15.38:0/2500734240 shutdown_connections 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/2500734240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c410ace0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 -- 172.21.15.38:0/2500734240 >> 172.21.15.38:0/2500734240 conn(0x7f14c41022e0 msgr2=0x7f14c4104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 -- 172.21.15.38:0/2500734240 shutdown_connections 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.101+0000 7f14cb7eb700 1 -- 172.21.15.38:0/2500734240 wait complete. 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14cb7eb700 1 Processor -- start 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14cb7eb700 1 -- start start 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14cb7eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14cb7eb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14c410c1e0 con 0x7f14c41088f0 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14c9587700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:01.854 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14c9587700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59810/0 (socket says 172.21.15.38:59810) 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14c9587700 1 -- 172.21.15.38:0/3040303440 learned_addr learned my addr 172.21.15.38:0/3040303440 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14c9587700 1 -- 172.21.15.38:0/3040303440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c419c370 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.102+0000 7f14c9587700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14c0011b60 tx=0x7f14c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14c0003970 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14c0003b10 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14c0024610 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c419c600 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c419f2f0 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f14c001a430 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.103+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c4069390 con 0x7f14c41088f0 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.104+0000 7f14ba7fc700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 0x7f14b003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:01.855 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.104+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f14c0011e00 con 0x7f14c41088f0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.104+0000 7f14c8d86700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 0x7f14b003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.104+0000 7f14c8d86700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 0x7f14b003b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006fd0 tx=0x7f14b4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.109+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f14c00212a0 con 0x7f14c41088f0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.317+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f14c4103660 con 0x7f14b00390a0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.318+0000 7f14ba7fc700 1 -- 172.21.15.38:0/3040303440 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f14c4103660 con 0x7f14b00390a0 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 msgr2=0x7f14b003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 0x7f14b003b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006fd0 tx=0x7f14b4006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 msgr2=0x7f14c419be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14c0011b60 tx=0x7f14c0011c40 comp rx=0 tx=0).stop 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 shutdown_connections 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f14b00390a0 0x7f14b003b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:01.856 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 --2- 172.21.15.38:0/3040303440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14c41088f0 0x7f14c419be30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:01.857 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 >> 172.21.15.38:0/3040303440 conn(0x7f14c41022e0 msgr2=0x7f14c4102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:01.857 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 shutdown_connections 2024-04-21T23:52:01.857 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:01.320+0000 7f14cb7eb700 1 -- 172.21.15.38:0/3040303440 wait complete. 2024-04-21T23:52:01.857 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-04-21T23:52:02.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-mon[27972]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: Generating public/private rsa key pair. 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: Your identification has been saved in /tmp/tmpa0x0uyqn/key. 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: Your public key has been saved in /tmp/tmpa0x0uyqn/key.pub. 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: The key fingerprint is: 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: SHA256:MFO8q72+pu8oMHeDnvvgQEQJ+jJmJAYaDcv0aN+fV9Y ceph-eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: The key's randomart image is: 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: +---[RSA 3072]----+ 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |==.. .. | 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |*+= .. | 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |=*.. + . | 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |=o. . +. . | 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |ooo. o S. o E | 2024-04-21T23:52:02.948 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: |o+o o + o o | 2024-04-21T23:52:02.949 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: | .=.o * . | 2024-04-21T23:52:02.949 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: | o+...+ | 2024-04-21T23:52:02.949 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: | o++**o | 2024-04-21T23:52:02.949 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: +----[SHA256]-----+ 2024-04-21T23:52:03.163 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f2438e5b700 1 Processor -- start 2024-04-21T23:52:03.163 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f2438e5b700 1 -- start start 2024-04-21T23:52:03.163 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f2438e5b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f2438e5b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2434106b00 con 0x7f2434106110 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f243259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f243259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59818/0 (socket says 172.21.15.38:59818) 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f243259c700 1 -- 172.21.15.38:0/2497613209 learned_addr learned my addr 172.21.15.38:0/2497613209 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.327+0000 7f243259c700 1 -- 172.21.15.38:0/2497613209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24341090a0 con 0x7f2434106110 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.328+0000 7f243259c700 1 --2- 172.21.15.38:0/2497613209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f24240098a0 tx=0x7f2424009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3ab620a82eaaec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f243159a700 1 -- 172.21.15.38:0/2497613209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2424004160 con 0x7f2434106110 2024-04-21T23:52:03.164 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f243159a700 1 -- 172.21.15.38:0/2497613209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2424004300 con 0x7f2434106110 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f243159a700 1 -- 172.21.15.38:0/2497613209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2424011b50 con 0x7f2434106110 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 -- 172.21.15.38:0/2497613209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 msgr2=0x7f2434106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 --2- 172.21.15.38:0/2497613209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f24240098a0 tx=0x7f2424009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 -- 172.21.15.38:0/2497613209 shutdown_connections 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 --2- 172.21.15.38:0/2497613209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f2434106530 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 -- 172.21.15.38:0/2497613209 >> 172.21.15.38:0/2497613209 conn(0x7f2434101a60 msgr2=0x7f2434103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 -- 172.21.15.38:0/2497613209 shutdown_connections 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.330+0000 7f2438e5b700 1 -- 172.21.15.38:0/2497613209 wait complete. 2024-04-21T23:52:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.331+0000 7f2438e5b700 1 Processor -- start 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.331+0000 7f2438e5b700 1 -- start start 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.331+0000 7f2438e5b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.331+0000 7f2438e5b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f243410c450 con 0x7f2434106110 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.331+0000 7f243259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f243259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59822/0 (socket says 172.21.15.38:59822) 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f243259c700 1 -- 172.21.15.38:0/1036363961 learned_addr learned my addr 172.21.15.38:0/1036363961 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f243259c700 1 -- 172.21.15.38:0/1036363961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243419c360 con 0x7f2434106110 2024-04-21T23:52:03.166 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f243259c700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2424000c00 tx=0x7f2424011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2424003b20 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f243419c5f0 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2424003cc0 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2424024910 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.332+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f243419f2e0 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.333+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2424022020 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.333+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2434063890 con 0x7f2434106110 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.333+0000 7f24237fe700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 0x7f241c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:03.167 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.333+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2424011e00 con 0x7f2434106110 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.333+0000 7f2431d9b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 0x7f241c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.334+0000 7f2431d9b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 0x7f241c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2428006fd0 tx=0x7f2428006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.339+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f242402cdf0 con 0x7f2434106110 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.547+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2434102cb0 con 0x7f241c039050 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.658+0000 7f24237fe700 1 -- 172.21.15.38:0/1036363961 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2434102cb0 con 0x7f241c039050 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 msgr2=0x7f241c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 0x7f241c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2428006fd0 tx=0x7f2428006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 msgr2=0x7f243419be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2424000c00 tx=0x7f2424011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 shutdown_connections 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f241c039050 0x7f241c03b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 --2- 172.21.15.38:0/1036363961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2434106110 0x7f243419be20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 >> 172.21.15.38:0/1036363961 conn(0x7f2434101a60 msgr2=0x7f24341025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 shutdown_connections 2024-04-21T23:52:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:02.660+0000 7f2438e5b700 1 -- 172.21.15.38:0/1036363961 wait complete. 2024-04-21T23:52:03.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:03 smithi038 ceph-mon[27972]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:03 smithi038 ceph-mon[27972]: Generating ssh key... 2024-04-21T23:52:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:03 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:03 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:52:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.617+0000 7f04dc585700 1 Processor -- start 2024-04-21T23:52:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.617+0000 7f04dc585700 1 -- start start 2024-04-21T23:52:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04dc585700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04dc585700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4108cf0 con 0x7f04d4108300 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04da321700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04da321700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59826/0 (socket says 172.21.15.38:59826) 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04da321700 1 -- 172.21.15.38:0/788657987 learned_addr learned my addr 172.21.15.38:0/788657987 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04da321700 1 -- 172.21.15.38:0/788657987 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d4106f10 con 0x7f04d4108300 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04da321700 1 --2- 172.21.15.38:0/788657987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04c40098a0 tx=0x7f04c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acd0a634b9658d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.618+0000 7f04d931f700 1 -- 172.21.15.38:0/788657987 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04c4004160 con 0x7f04d4108300 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04d931f700 1 -- 172.21.15.38:0/788657987 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f04c4004300 con 0x7f04d4108300 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04d931f700 1 -- 172.21.15.38:0/788657987 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04c4011b50 con 0x7f04d4108300 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 -- 172.21.15.38:0/788657987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 msgr2=0x7f04d4108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 --2- 172.21.15.38:0/788657987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04c40098a0 tx=0x7f04c4009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 -- 172.21.15.38:0/788657987 shutdown_connections 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 --2- 172.21.15.38:0/788657987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d4108720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 -- 172.21.15.38:0/788657987 >> 172.21.15.38:0/788657987 conn(0x7f04d4101a60 msgr2=0x7f04d4103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 -- 172.21.15.38:0/788657987 shutdown_connections 2024-04-21T23:52:04.404 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.619+0000 7f04dc585700 1 -- 172.21.15.38:0/788657987 wait complete. 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04dc585700 1 Processor -- start 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04dc585700 1 -- start start 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04dc585700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04dc585700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04d410b2d0 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04da321700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04da321700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59834/0 (socket says 172.21.15.38:59834) 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04da321700 1 -- 172.21.15.38:0/3416205641 learned_addr learned my addr 172.21.15.38:0/3416205641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04da321700 1 -- 172.21.15.38:0/3416205641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d419c340 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.620+0000 7f04da321700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011670 tx=0x7f04c4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04c40039c0 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f04c4011e50 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d419c5d0 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04c401a5c0 con 0x7f04d4108300 2024-04-21T23:52:04.405 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d419f2c0 con 0x7f04d4108300 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.621+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f04c401a8c0 con 0x7f04d4108300 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.622+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d4063890 con 0x7f04d4108300 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.622+0000 7f04cb7fe700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 0x7f04c003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.622+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f04c404d5d0 con 0x7f04d4108300 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.622+0000 7f04d9b20700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 0x7f04c003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.622+0000 7f04d9b20700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 0x7f04c003b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.627+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f04c4019c70 con 0x7f04d4108300 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.836+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f04d4102cc0 con 0x7f04c0039000 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.836+0000 7f04cb7fe700 1 -- 172.21.15.38:0/3416205641 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f04d4102cc0 con 0x7f04c0039000 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 msgr2=0x7f04c003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 0x7f04c003b4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f04d0006fd0 tx=0x7f04d0006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 msgr2=0x7f04d419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:04.406 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011670 tx=0x7f04c4011b60 comp rx=0 tx=0).stop 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 shutdown_connections 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f04c0039000 0x7f04c003b4c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 --2- 172.21.15.38:0/3416205641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04d4108300 0x7f04d419be00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.838+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 >> 172.21.15.38:0/3416205641 conn(0x7f04d4101a60 msgr2=0x7f04d41025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.839+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 shutdown_connections 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:03.839+0000 7f04dc585700 1 -- 172.21.15.38:0/3416205641 wait complete. 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T23:52:04.407 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-04-21T23:52:04.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:04 smithi038 ceph-mon[27972]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:05.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:05 smithi038 ceph-mon[27972]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:06.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:06 smithi038 ceph-mon[27972]: Deploying cephadm binary to smithi038 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.847+0000 7f252cbe8700 1 Processor -- start 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.847+0000 7f252cbe8700 1 -- start start 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252cbe8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252cbe8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f252807d090 con 0x7f252807e650 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59422/0 (socket says 172.21.15.38:59422) 2024-04-21T23:52:09.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252659c700 1 -- 172.21.15.38:0/1899650377 learned_addr learned my addr 172.21.15.38:0/1899650377 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252659c700 1 -- 172.21.15.38:0/1899650377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252807d210 con 0x7f252807e650 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252659c700 1 --2- 172.21.15.38:0/1899650377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25180098a0 tx=0x7f2518009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b9ec6873b921ee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252559a700 1 -- 172.21.15.38:0/1899650377 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2518004160 con 0x7f252807e650 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252559a700 1 -- 172.21.15.38:0/1899650377 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2518004300 con 0x7f252807e650 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.848+0000 7f252559a700 1 -- 172.21.15.38:0/1899650377 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2518011b50 con 0x7f252807e650 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 -- 172.21.15.38:0/1899650377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 msgr2=0x7f252807cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 --2- 172.21.15.38:0/1899650377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25180098a0 tx=0x7f2518009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 -- 172.21.15.38:0/1899650377 shutdown_connections 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 --2- 172.21.15.38:0/1899650377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252807cb50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 -- 172.21.15.38:0/1899650377 >> 172.21.15.38:0/1899650377 conn(0x7f2528103470 msgr2=0x7f2528105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 -- 172.21.15.38:0/1899650377 shutdown_connections 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.849+0000 7f252cbe8700 1 -- 172.21.15.38:0/1899650377 wait complete. 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252cbe8700 1 Processor -- start 2024-04-21T23:52:09.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252cbe8700 1 -- start start 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252cbe8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252cbe8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f252810e420 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59432/0 (socket says 172.21.15.38:59432) 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.850+0000 7f252659c700 1 -- 172.21.15.38:0/443825480 learned_addr learned my addr 172.21.15.38:0/443825480 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f252659c700 1 -- 172.21.15.38:0/443825480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252819e4d0 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f252659c700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2518011b60 tx=0x7f2518011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2518003b20 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2518003cc0 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252819e760 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2518024840 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.851+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25281a1450 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.852+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2518022020 con 0x7f252807e650 2024-04-21T23:52:09.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.852+0000 7f25177fe700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 0x7f251003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.852+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2518011e00 con 0x7f252807e650 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.852+0000 7f2525d9b700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 0x7f251003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.852+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2528063890 con 0x7f252807e650 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.853+0000 7f2525d9b700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 0x7f251003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f251c006fd0 tx=0x7f251c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:04.857+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2518019c70 con 0x7f252807e650 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:05.069+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) -- 0x7f2528104670 con 0x7f2510039050 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.311+0000 7f25177fe700 1 -- 172.21.15.38:0/443825480 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f2528104670 con 0x7f2510039050 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.313+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 msgr2=0x7f251003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.313+0000 7f252cbe8700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 0x7f251003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f251c006fd0 tx=0x7f251c006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.313+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 msgr2=0x7f252819df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.313+0000 7f252cbe8700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2518011b60 tx=0x7f2518011c40 comp rx=0 tx=0).stop 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 shutdown_connections 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f2510039050 0x7f251003b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:09.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 --2- 172.21.15.38:0/443825480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f252807e650 0x7f252819df90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:09.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 >> 172.21.15.38:0/443825480 conn(0x7f2528103470 msgr2=0x7f2528103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:09.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 shutdown_connections 2024-04-21T23:52:09.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:09.314+0000 7f252cbe8700 1 -- 172.21.15.38:0/443825480 wait complete. 2024-04-21T23:52:09.915 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2024-04-21T23:52:10.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:10 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:10.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:10 smithi038 ceph-mon[27972]: Added host smithi038 2024-04-21T23:52:10.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:10 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:11.868 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:11 smithi038 ceph-mon[27972]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:11.868 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:11 smithi038 ceph-mon[27972]: Saving service mon spec with placement count:5 2024-04-21T23:52:11.868 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:11 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:11.868 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:11 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.401+0000 7f65f6dd2700 1 Processor -- start 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.401+0000 7f65f6dd2700 1 -- start start 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.401+0000 7f65f6dd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.401+0000 7f65f6dd2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f65f0108790 con 0x7f65f0107e30 2024-04-21T23:52:12.176 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65f4b6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65f4b6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59444/0 (socket says 172.21.15.38:59444) 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65f4b6e700 1 -- 172.21.15.38:0/777506694 learned_addr learned my addr 172.21.15.38:0/777506694 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65f4b6e700 1 -- 172.21.15.38:0/777506694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f0108910 con 0x7f65f0107e30 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65f4b6e700 1 --2- 172.21.15.38:0/777506694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098a0 tx=0x7f65e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc5022eff384d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65ef7fe700 1 -- 172.21.15.38:0/777506694 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e0004160 con 0x7f65f0107e30 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65ef7fe700 1 -- 172.21.15.38:0/777506694 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65e0004300 con 0x7f65f0107e30 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.402+0000 7f65ef7fe700 1 -- 172.21.15.38:0/777506694 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e0011b50 con 0x7f65f0107e30 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- 172.21.15.38:0/777506694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 msgr2=0x7f65f0108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/777506694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098a0 tx=0x7f65e0009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- 172.21.15.38:0/777506694 shutdown_connections 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/777506694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f0108250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- 172.21.15.38:0/777506694 >> 172.21.15.38:0/777506694 conn(0x7f65f0103450 msgr2=0x7f65f01058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- 172.21.15.38:0/777506694 shutdown_connections 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- 172.21.15.38:0/777506694 wait complete. 2024-04-21T23:52:12.177 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 Processor -- start 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.403+0000 7f65f6dd2700 1 -- start start 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f6dd2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f6dd2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f65f010c2a0 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f4b6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f4b6e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59460/0 (socket says 172.21.15.38:59460) 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f4b6e700 1 -- 172.21.15.38:0/490092727 learned_addr learned my addr 172.21.15.38:0/490092727 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f4b6e700 1 -- 172.21.15.38:0/490092727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f019c2d0 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f4b6e700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011670 tx=0x7f65e0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e00039c0 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65e0011e50 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f019c560 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e001a5c0 con 0x7f65f0107e30 2024-04-21T23:52:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.404+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f019f210 con 0x7f65f0107e30 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.405+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f65e001a8c0 con 0x7f65f0107e30 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.405+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f0063890 con 0x7f65f0107e30 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.405+0000 7f65edffb700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 0x7f65d803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.405+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f65e004d5d0 con 0x7f65f0107e30 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.406+0000 7f65effff700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 0x7f65d803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.406+0000 7f65effff700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 0x7f65d803b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65e4006fd0 tx=0x7f65e4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.410+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f65e0019c70 con 0x7f65f0107e30 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.646+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f65f0104670 con 0x7f65d8039000 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.651+0000 7f65edffb700 1 -- 172.21.15.38:0/490092727 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f65f0104670 con 0x7f65d8039000 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 msgr2=0x7f65d803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 0x7f65d803b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f65e4006fd0 tx=0x7f65e4006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:12.179 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 msgr2=0x7f65f019bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011670 tx=0x7f65e0011b60 comp rx=0 tx=0).stop 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 shutdown_connections 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f65d8039000 0x7f65d803b4c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 --2- 172.21.15.38:0/490092727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0107e30 0x7f65f019bd90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 >> 172.21.15.38:0/490092727 conn(0x7f65f0103450 msgr2=0x7f65f0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 shutdown_connections 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:10.653+0000 7f65f6dd2700 1 -- 172.21.15.38:0/490092727 wait complete. 2024-04-21T23:52:12.180 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2024-04-21T23:52:13.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:13 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:13.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:13 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:13.651 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T23:52:13.651 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dedb2700 1 Processor -- start 2024-04-21T23:52:13.651 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dedb2700 1 -- start start 2024-04-21T23:52:13.651 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dedb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:13.651 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dedb2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48d807be30 con 0x7f48d81066e0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dcb4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dcb4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59464/0 (socket says 172.21.15.38:59464) 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.625+0000 7f48dcb4e700 1 -- 172.21.15.38:0/2436107053 learned_addr learned my addr 172.21.15.38:0/2436107053 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.626+0000 7f48dcb4e700 1 -- 172.21.15.38:0/2436107053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d8107040 con 0x7f48d81066e0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.626+0000 7f48dcb4e700 1 --2- 172.21.15.38:0/2436107053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0098a0 tx=0x7f48cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401acc14be2ec3c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.626+0000 7f48d77fe700 1 -- 172.21.15.38:0/2436107053 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48cc004160 con 0x7f48d81066e0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.626+0000 7f48d77fe700 1 -- 172.21.15.38:0/2436107053 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48cc004300 con 0x7f48d81066e0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.626+0000 7f48d77fe700 1 -- 172.21.15.38:0/2436107053 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48cc011b20 con 0x7f48d81066e0 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- 172.21.15.38:0/2436107053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 msgr2=0x7f48d8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2436107053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0098a0 tx=0x7f48cc009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- 172.21.15.38:0/2436107053 shutdown_connections 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2436107053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d8106b00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:13.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- 172.21.15.38:0/2436107053 >> 172.21.15.38:0/2436107053 conn(0x7f48d81022e0 msgr2=0x7f48d8104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- 172.21.15.38:0/2436107053 shutdown_connections 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- 172.21.15.38:0/2436107053 wait complete. 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 Processor -- start 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.627+0000 7f48dedb2700 1 -- start start 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dedb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dedb2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8108110 con 0x7f48d81066e0 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dcb4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dcb4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59468/0 (socket says 172.21.15.38:59468) 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dcb4e700 1 -- 172.21.15.38:0/2421512591 learned_addr learned my addr 172.21.15.38:0/2421512591 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dcb4e700 1 -- 172.21.15.38:0/2421512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d819e4c0 con 0x7f48d81066e0 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dcb4e700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0117e0 tx=0x7f48cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48cc003bf0 con 0x7f48d81066e0 2024-04-21T23:52:13.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d819e750 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48cc01a910 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48cc024970 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.628+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d819ebb0 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.629+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d8063890 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.630+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f48cc022020 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.630+0000 7f48d5ffb700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 0x7f48c803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.630+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f48cc04d630 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.630+0000 7f48d7fff700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 0x7f48c803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.630+0000 7f48d7fff700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 0x7f48c803b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48c4006fd0 tx=0x7f48c4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.639+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f48cc011e20 con 0x7f48d81066e0 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.883+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f48d8062670 con 0x7f48c8039050 2024-04-21T23:52:13.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.889+0000 7f48d5ffb700 1 -- 172.21.15.38:0/2421512591 <== mgr.14116 v2:172.21.15.38:6800/2824973701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f48d8062670 con 0x7f48c8039050 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 msgr2=0x7f48c803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 0x7f48c803b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48c4006fd0 tx=0x7f48c4006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 msgr2=0x7f48d819df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0117e0 tx=0x7f48cc011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 shutdown_connections 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f48c8039050 0x7f48c803b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 --2- 172.21.15.38:0/2421512591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48d81066e0 0x7f48d819df80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 >> 172.21.15.38:0/2421512591 conn(0x7f48d81022e0 msgr2=0x7f48d8194ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 shutdown_connections 2024-04-21T23:52:13.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:12.892+0000 7f48dedb2700 1 -- 172.21.15.38:0/2421512591 wait complete. 2024-04-21T23:52:14.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:14 smithi038 ceph-mon[27972]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:14.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:14 smithi038 ceph-mon[27972]: Saving service mgr spec with placement count:2 2024-04-21T23:52:14.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:14 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f1598536700 1 Processor -- start 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f1598536700 1 -- start start 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f1598536700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f1598536700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f159007be50 con 0x7f15901088d0 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f15962d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.470+0000 7f15962d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47590/0 (socket says 172.21.15.38:47590) 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15962d2700 1 -- 172.21.15.38:0/2407347253 learned_addr learned my addr 172.21.15.38:0/2407347253 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15962d2700 1 -- 172.21.15.38:0/2407347253 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159010b200 con 0x7f15901088d0 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15962d2700 1 --2- 172.21.15.38:0/2407347253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15800098a0 tx=0x7f1580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94cb021ce5021af server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:15.268 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15952d0700 1 -- 172.21.15.38:0/2407347253 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1580004160 con 0x7f15901088d0 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15952d0700 1 -- 172.21.15.38:0/2407347253 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1580004300 con 0x7f15901088d0 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.471+0000 7f15952d0700 1 -- 172.21.15.38:0/2407347253 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1580011b20 con 0x7f15901088d0 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 -- 172.21.15.38:0/2407347253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 msgr2=0x7f159010acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 --2- 172.21.15.38:0/2407347253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15800098a0 tx=0x7f1580009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 -- 172.21.15.38:0/2407347253 shutdown_connections 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 --2- 172.21.15.38:0/2407347253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159010acc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 -- 172.21.15.38:0/2407347253 >> 172.21.15.38:0/2407347253 conn(0x7f15901022e0 msgr2=0x7f1590104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 -- 172.21.15.38:0/2407347253 shutdown_connections 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.472+0000 7f1598536700 1 -- 172.21.15.38:0/2407347253 wait complete. 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f1598536700 1 Processor -- start 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f1598536700 1 -- start start 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f1598536700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f1598536700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f159010c260 con 0x7f15901088d0 2024-04-21T23:52:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15962d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15962d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47602/0 (socket says 172.21.15.38:47602) 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15962d2700 1 -- 172.21.15.38:0/2856399661 learned_addr learned my addr 172.21.15.38:0/2856399661 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15962d2700 1 -- 172.21.15.38:0/2856399661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159019c290 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15962d2700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15800117e0 tx=0x7f1580011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.473+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1580003bf0 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.474+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f158001a910 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.474+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159019c520 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.474+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1580024970 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.474+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159019c980 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1580022020 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1590063890 con 0x7f15901088d0 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f15877fe700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 0x7f157c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f158004d370 con 0x7f15901088d0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f1595ad1700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 0x7f157c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.475+0000 7f1595ad1700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 0x7f157c03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f158c006fd0 tx=0x7f158c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.480+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1580011e20 con 0x7f15901088d0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.692+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f159019f210 con 0x7f15901088d0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.695+0000 7f15877fe700 1 -- 172.21.15.38:0/2856399661 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f158000b3e0 con 0x7f15901088d0 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 msgr2=0x7f157c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 0x7f157c03b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f158c006fd0 tx=0x7f158c006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 msgr2=0x7f159019bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15800117e0 tx=0x7f1580011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 shutdown_connections 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f157c039050 0x7f157c03b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 --2- 172.21.15.38:0/2856399661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15901088d0 0x7f159019bd50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:15.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.697+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 >> 172.21.15.38:0/2856399661 conn(0x7f15901022e0 msgr2=0x7f1590102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:15.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.698+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 shutdown_connections 2024-04-21T23:52:15.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:14.698+0000 7f1598536700 1 -- 172.21.15.38:0/2856399661 wait complete. 2024-04-21T23:52:15.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:15 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2856399661' entity='client.admin' 2024-04-21T23:52:15.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:15 smithi038 ceph-mon[27972]: from='mgr.14116 172.21.15.38:0/4267101605' entity='mgr.x' 2024-04-21T23:52:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:16 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3151825228' entity='client.admin' 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.728+0000 7f0688593700 1 Processor -- start 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.728+0000 7f0688593700 1 -- start start 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f0688593700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f0688593700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f068007be70 con 0x7f06801088b0 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068632f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068632f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47606/0 (socket says 172.21.15.38:47606) 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068632f700 1 -- 172.21.15.38:0/4253727178 learned_addr learned my addr 172.21.15.38:0/4253727178 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:17.220 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068632f700 1 -- 172.21.15.38:0/4253727178 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068010b1e0 con 0x7f06801088b0 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068632f700 1 --2- 172.21.15.38:0/4253727178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f06700098a0 tx=0x7f0670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf60f07fed3dbd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.729+0000 7f068532d700 1 -- 172.21.15.38:0/4253727178 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670004160 con 0x7f06801088b0 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f068532d700 1 -- 172.21.15.38:0/4253727178 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0670004300 con 0x7f06801088b0 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f068532d700 1 -- 172.21.15.38:0/4253727178 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670011b20 con 0x7f06801088b0 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 -- 172.21.15.38:0/4253727178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 msgr2=0x7f068010aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 --2- 172.21.15.38:0/4253727178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f06700098a0 tx=0x7f0670009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 -- 172.21.15.38:0/4253727178 shutdown_connections 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 --2- 172.21.15.38:0/4253727178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f068010aca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 -- 172.21.15.38:0/4253727178 >> 172.21.15.38:0/4253727178 conn(0x7f06801022e0 msgr2=0x7f0680104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 -- 172.21.15.38:0/4253727178 shutdown_connections 2024-04-21T23:52:17.221 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.730+0000 7f0688593700 1 -- 172.21.15.38:0/4253727178 wait complete. 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f0688593700 1 Processor -- start 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f0688593700 1 -- start start 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f0688593700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f0688593700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f068010c1e0 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f068632f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f068632f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47614/0 (socket says 172.21.15.38:47614) 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.731+0000 7f068632f700 1 -- 172.21.15.38:0/3151825228 learned_addr learned my addr 172.21.15.38:0/3151825228 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f068632f700 1 -- 172.21.15.38:0/3151825228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0680108670 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f068632f700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f06700117e0 tx=0x7f0670011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670003ba0 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f067001a910 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0680106860 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06700248f0 con 0x7f06801088b0 2024-04-21T23:52:17.222 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.732+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0680106d20 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.733+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0670022020 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.733+0000 7f06777fe700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 0x7f066c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.733+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f067004d7f0 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.733+0000 7f0685b2e700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 0x7f066c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.733+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0680063890 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.734+0000 7f0685b2e700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 0x7f066c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f067c006fd0 tx=0x7f067c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.738+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0670020c10 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.951+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f06801076b0 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.953+0000 7f06777fe700 1 -- 172.21.15.38:0/3151825228 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f067001aac0 con 0x7f06801088b0 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 msgr2=0x7f066c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 0x7f066c03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f067c006fd0 tx=0x7f067c006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 msgr2=0x7f0680108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:17.223 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f06700117e0 tx=0x7f0670011ba0 comp rx=0 tx=0).stop 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 shutdown_connections 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f066c039050 0x7f066c03b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 --2- 172.21.15.38:0/3151825228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06801088b0 0x7f0680108130 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 >> 172.21.15.38:0/3151825228 conn(0x7f06801022e0 msgr2=0x7f0680102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 shutdown_connections 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:15.956+0000 7f0688593700 1 -- 172.21.15.38:0/3151825228 wait complete. 2024-04-21T23:52:17.224 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-04-21T23:52:18.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-mon[27972]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:18.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1932915281' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T23:52:18.704 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setuser ceph since I am not root 2024-04-21T23:52:18.704 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setgroup ceph since I am not root 2024-04-21T23:52:18.704 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:18.564+0000 7f1b75b09200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:52:18.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:18.758+0000 7f1b75b09200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:52:18.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:18.865+0000 7f1b75b09200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2b87e700 1 Processor -- start 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2b87e700 1 -- start start 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2b87e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2b87e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24108810 con 0x7f1f24107eb0 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2961a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2961a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47616/0 (socket says 172.21.15.38:47616) 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.071+0000 7f1f2961a700 1 -- 172.21.15.38:0/3313923909 learned_addr learned my addr 172.21.15.38:0/3313923909 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2961a700 1 -- 172.21.15.38:0/3313923909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f24108f70 con 0x7f1f24107eb0 2024-04-21T23:52:18.986 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2961a700 1 --2- 172.21.15.38:0/3313923909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f200098a0 tx=0x7f1f20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20c3296556cb8e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f1bfff700 1 -- 172.21.15.38:0/3313923909 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f20004160 con 0x7f1f24107eb0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f1bfff700 1 -- 172.21.15.38:0/3313923909 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f20004300 con 0x7f1f24107eb0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f1bfff700 1 -- 172.21.15.38:0/3313923909 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f20011b20 con 0x7f1f24107eb0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2b87e700 1 -- 172.21.15.38:0/3313923909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 msgr2=0x7f1f241082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/3313923909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1f200098a0 tx=0x7f1f20009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2b87e700 1 -- 172.21.15.38:0/3313923909 shutdown_connections 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/3313923909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f241082d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.072+0000 7f1f2b87e700 1 -- 172.21.15.38:0/3313923909 >> 172.21.15.38:0/3313923909 conn(0x7f1f24103470 msgr2=0x7f1f24105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 -- 172.21.15.38:0/3313923909 shutdown_connections 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 -- 172.21.15.38:0/3313923909 wait complete. 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 Processor -- start 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 -- start start 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.073+0000 7f1f2b87e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2410c3b0 con 0x7f1f24107eb0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2961a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:18.987 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2961a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47630/0 (socket says 172.21.15.38:47630) 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2961a700 1 -- 172.21.15.38:0/1932915281 learned_addr learned my addr 172.21.15.38:0/1932915281 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2961a700 1 -- 172.21.15.38:0/1932915281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2419c270 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2961a700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f200117e0 tx=0x7f1f20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f20003bf0 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f2001a910 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f20024970 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2419c500 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.074+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f2419c960 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.075+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1f20022020 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.075+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f24063890 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.075+0000 7f1f1a7fc700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 0x7f1f1003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.075+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1f2004d3a0 con 0x7f1f24107eb0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.076+0000 7f1f28e19700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 0x7f1f1003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.076+0000 7f1f28e19700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 0x7f1f1003b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f14006fd0 tx=0x7f1f14006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:18.988 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.080+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1f20011e20 con 0x7f1f24107eb0 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.347+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f1f2419f7a0 con 0x7f1f24107eb0 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.384+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f1f2000b3e0 con 0x7f1f24107eb0 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.385+0000 7f1f1a7fc700 1 -- 172.21.15.38:0/1932915281 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f1f2001a430 con 0x7f1f24107eb0 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.386+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 msgr2=0x7f1f1003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.386+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 0x7f1f1003b4c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f14006fd0 tx=0x7f1f14006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.386+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 msgr2=0x7f1f2419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.386+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1f200117e0 tx=0x7f1f20011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 shutdown_connections 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f1f10039000 0x7f1f1003b4c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 --2- 172.21.15.38:0/1932915281 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f24107eb0 0x7f1f2419bd30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 >> 172.21.15.38:0/1932915281 conn(0x7f1f24103470 msgr2=0x7f1f24103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 shutdown_connections 2024-04-21T23:52:18.989 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:18.387+0000 7f1f2b87e700 1 -- 172.21.15.38:0/1932915281 wait complete. 2024-04-21T23:52:19.229 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:19 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:19.192+0000 7f1b75b09200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:52:19.601 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:19 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:19.285+0000 7f1b75b09200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:52:19.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:19 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1932915281' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T23:52:19.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:19 smithi038 ceph-mon[27972]: mgrmap e9: x(active, since 21s) 2024-04-21T23:52:20.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:52:20.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T23:52:20.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T23:52:20.269 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28d67c2700 1 Processor -- start 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28d67c2700 1 -- start start 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28d67c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28d67c2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28d007d0b0 con 0x7f28d007e600 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47662/0 (socket says 172.21.15.38:47662) 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28cffff700 1 -- 172.21.15.38:0/716141926 learned_addr learned my addr 172.21.15.38:0/716141926 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.399+0000 7f28cffff700 1 -- 172.21.15.38:0/716141926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d007d230 con 0x7f28d007e600 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.400+0000 7f28cffff700 1 --2- 172.21.15.38:0/716141926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28c00098a0 tx=0x7f28c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04bb1311191035f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:20.270 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.403+0000 7f28ceffd700 1 -- 172.21.15.38:0/716141926 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0004160 con 0x7f28d007e600 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.403+0000 7f28ceffd700 1 -- 172.21.15.38:0/716141926 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c0004300 con 0x7f28d007e600 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.403+0000 7f28ceffd700 1 -- 172.21.15.38:0/716141926 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0011b20 con 0x7f28d007e600 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.403+0000 7f28d67c2700 1 -- 172.21.15.38:0/716141926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 msgr2=0x7f28d007cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.403+0000 7f28d67c2700 1 --2- 172.21.15.38:0/716141926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28c00098a0 tx=0x7f28c0009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 -- 172.21.15.38:0/716141926 shutdown_connections 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 --2- 172.21.15.38:0/716141926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d007cb70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 -- 172.21.15.38:0/716141926 >> 172.21.15.38:0/716141926 conn(0x7f28d0103450 msgr2=0x7f28d01058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 -- 172.21.15.38:0/716141926 shutdown_connections 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 -- 172.21.15.38:0/716141926 wait complete. 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 Processor -- start 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.404+0000 7f28d67c2700 1 -- start start 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28d67c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28d67c2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28d010e340 con 0x7f28d007e600 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:20.271 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47666/0 (socket says 172.21.15.38:47666) 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cffff700 1 -- 172.21.15.38:0/330985497 learned_addr learned my addr 172.21.15.38:0/330985497 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cffff700 1 -- 172.21.15.38:0/330985497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d019e4a0 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cffff700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28c00117e0 tx=0x7f28c0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.405+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c0003ba0 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.406+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c001a910 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.406+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d019e730 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.406+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c00248f0 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.406+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d019eb90 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f28c0022020 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d0063890 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28cd7fa700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 0x7f28b803b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28c004d500 con 0x7f28d007e600 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28cf7fe700 1 -- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 msgr2=0x7f28b803b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.407+0000 7f28cf7fe700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 0x7f28b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:52:20.272 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.412+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f28c000f2e0 con 0x7f28d007e600 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.608+0000 7f28cf7fe700 1 -- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 msgr2=0x7f28b803b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.608+0000 7f28cf7fe700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 0x7f28b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.702+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f28d01a1540 con 0x7f28d007e600 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.703+0000 7f28cd7fa700 1 -- 172.21.15.38:0/330985497 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f28c002d3f0 con 0x7f28d007e600 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.706+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 msgr2=0x7f28b803b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.707+0000 7f28d67c2700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 0x7f28b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.707+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 msgr2=0x7f28d019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.707+0000 7f28d67c2700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28c00117e0 tx=0x7f28c0011ba0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 shutdown_connections 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7f28b8039100 0x7f28b803b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 --2- 172.21.15.38:0/330985497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28d007e600 0x7f28d019df60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 >> 172.21.15.38:0/330985497 conn(0x7f28d0103450 msgr2=0x7f28d0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:20.273 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 shutdown_connections 2024-04-21T23:52:20.274 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:19.708+0000 7f28d67c2700 1 -- 172.21.15.38:0/330985497 wait complete. 2024-04-21T23:52:20.274 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-04-21T23:52:20.274 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-04-21T23:52:20.510 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/330985497' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T23:52:20.895 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:20 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:20.560+0000 7f1b75b09200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:52:20.896 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:20 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:20.745+0000 7f1b75b09200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:52:21.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:20 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:20.922+0000 7f1b75b09200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:52:21.524 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:21 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:21.224+0000 7f1b75b09200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:52:21.525 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:21 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:21.321+0000 7f1b75b09200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:52:21.787 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:21 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:21.533+0000 7f1b75b09200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:52:21.787 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:21 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:21.740+0000 7f1b75b09200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:52:22.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:22 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:22.383+0000 7f1b75b09200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:52:23.132 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:22 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:22.829+0000 7f1b75b09200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:52:23.133 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:22 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:22.929+0000 7f1b75b09200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:52:23.133 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:23 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:23.139+0000 7f1b75b09200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:52:23.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:23 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:23.685+0000 7f1b75b09200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:52:23.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:23 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:23.781+0000 7f1b75b09200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:52:24.387 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:24.035+0000 7f1b75b09200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:52:24.387 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:24.174+0000 7f1b75b09200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:52:24.387 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:24.273+0000 7f1b75b09200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:52:24.388 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:24.393+0000 7f1b75b09200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:52:24.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:24.489+0000 7f1b75b09200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:52:25.605 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:25.335+0000 7f1b75b09200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:52:25.605 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:52:25.559+0000 7f1b75b09200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:52:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: Active manager daemon x restarted 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: Activating manager daemon x 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: mgrmap e10: x(active, starting, since 0.00507262s) 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: Manager daemon x is now available 2024-04-21T23:52:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:26.806 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:52:26.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:52:26.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:26 smithi038 ceph-mon[27972]: mgrmap e11: x(active, since 1.00922s) 2024-04-21T23:52:27.570 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.696+0000 7fd6d3776700 1 Processor -- start 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.696+0000 7fd6d3776700 1 -- start start 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.696+0000 7fd6d3776700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.696+0000 7fd6d3776700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc108820 con 0x7fd6cc107ec0 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6d1512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6d1512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47678/0 (socket says 172.21.15.38:47678) 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6d1512700 1 -- 172.21.15.38:0/2775130278 learned_addr learned my addr 172.21.15.38:0/2775130278 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:27.571 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6d1512700 1 -- 172.21.15.38:0/2775130278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc108f80 con 0x7fd6cc107ec0 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6d1512700 1 --2- 172.21.15.38:0/2775130278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0098a0 tx=0x7fd6bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc68f7bc2d5702e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6c3fff700 1 -- 172.21.15.38:0/2775130278 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6bc004160 con 0x7fd6cc107ec0 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6c3fff700 1 -- 172.21.15.38:0/2775130278 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc004300 con 0x7fd6cc107ec0 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.697+0000 7fd6c3fff700 1 -- 172.21.15.38:0/2775130278 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6bc011b20 con 0x7fd6cc107ec0 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 -- 172.21.15.38:0/2775130278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 msgr2=0x7fd6cc1082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 --2- 172.21.15.38:0/2775130278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0098a0 tx=0x7fd6bc009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 -- 172.21.15.38:0/2775130278 shutdown_connections 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 --2- 172.21.15.38:0/2775130278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc1082e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 -- 172.21.15.38:0/2775130278 >> 172.21.15.38:0/2775130278 conn(0x7fd6cc1034a0 msgr2=0x7fd6cc1058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 -- 172.21.15.38:0/2775130278 shutdown_connections 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.698+0000 7fd6d3776700 1 -- 172.21.15.38:0/2775130278 wait complete. 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.699+0000 7fd6d3776700 1 Processor -- start 2024-04-21T23:52:27.572 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.699+0000 7fd6d3776700 1 -- start start 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d3776700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d3776700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc10c3c0 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d1512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d1512700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47690/0 (socket says 172.21.15.38:47690) 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d1512700 1 -- 172.21.15.38:0/1736209529 learned_addr learned my addr 172.21.15.38:0/1736209529 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d1512700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc19c2f0 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d1512700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0116a0 tx=0x7fd6bc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6bc003a40 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc003be0 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc19c580 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6bc01a750 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.700+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc19f230 con 0x7fd6cc107ec0 2024-04-21T23:52:27.573 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fd6bc01aad0 con 0x7fd6cc107ec0 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6c27fc700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd6cc19c9e0 con 0x7fd6b8039050 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd6bc04d760 con 0x7fd6cc107ec0 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6d0d11700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.701+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.902+0000 7fd6d0d11700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:20.902+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:21.303+0000 7fd6d0d11700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:21.303+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:22.104+0000 7fd6d0d11700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:22.104+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:23.705+0000 7fd6d0d11700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2824973701 2024-04-21T23:52:27.574 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:23.705+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:25.564+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7fd6bc02d910 con 0x7fd6cc107ec0 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:25.564+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 msgr2=0x7fd6b803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:25.564+0000 7fd6c27fc700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.567+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fd6bc029cf0 con 0x7fd6cc107ec0 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.567+0000 7fd6c27fc700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 0x7fd6b803ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.567+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd6bc021e70 con 0x7fd6b803c920 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.568+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 0x7fd6b803ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.568+0000 7fd6d0d11700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 0x7fd6b803ed10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8003a10 tx=0x7fd6c80092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.569+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd6bc021e70 con 0x7fd6b803c920 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd6cc069390 con 0x7fd6b803c920 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6c27fc700 1 -- 172.21.15.38:0/1736209529 <== mgr.14148 v2:172.21.15.38:6800/2340980806 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fd6cc069390 con 0x7fd6b803c920 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 msgr2=0x7fd6b803ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:27.575 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 0x7fd6b803ed10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8003a10 tx=0x7fd6c80092b0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 msgr2=0x7fd6cc19bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0116a0 tx=0x7fd6bc011ba0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.582+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 shutdown_connections 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6b803c920 0x7fd6b803ed10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6cc107ec0 0x7fd6cc19bdb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 --2- 172.21.15.38:0/1736209529 >> [v2:172.21.15.38:6800/2824973701,v1:172.21.15.38:6801/2824973701] conn(0x7fd6b8039050 0x7fd6b803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 >> 172.21.15.38:0/1736209529 conn(0x7fd6cc1034a0 msgr2=0x7fd6cc103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 shutdown_connections 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:26.583+0000 7fd6d3776700 1 -- 172.21.15.38:0/1736209529 wait complete. 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-04-21T23:52:27.576 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-04-21T23:52:27.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:27 smithi038 ceph-mon[27972]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T23:52:27.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:27 smithi038 ceph-mon[27972]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T23:52:28.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: [21/Apr/2024:23:52:27] ENGINE Bus STARTING 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: [21/Apr/2024:23:52:27] ENGINE Serving on http://172.21.15.38:8765 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: mgrmap e12: x(active, since 2s) 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: [21/Apr/2024:23:52:27] ENGINE Serving on https://172.21.15.38:7150 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: [21/Apr/2024:23:52:27] ENGINE Bus STARTED 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:28.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f490466d700 1 Processor -- start 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f490466d700 1 -- start start 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f490466d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f490466d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48fc106b00 con 0x7f48fc106110 2024-04-21T23:52:28.975 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f4902409700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f4902409700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38390/0 (socket says 172.21.15.38:38390) 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.040+0000 7f4902409700 1 -- 172.21.15.38:0/2369280593 learned_addr learned my addr 172.21.15.38:0/2369280593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.041+0000 7f4902409700 1 -- 172.21.15.38:0/2369280593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48fc1090a0 con 0x7f48fc106110 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.041+0000 7f4902409700 1 --2- 172.21.15.38:0/2369280593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098a0 tx=0x7f48f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa833ab071343ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.041+0000 7f4901407700 1 -- 172.21.15.38:0/2369280593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f8004160 con 0x7f48fc106110 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.041+0000 7f4901407700 1 -- 172.21.15.38:0/2369280593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f8004300 con 0x7f48fc106110 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.041+0000 7f4901407700 1 -- 172.21.15.38:0/2369280593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f8011b20 con 0x7f48fc106110 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- 172.21.15.38:0/2369280593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 msgr2=0x7f48fc106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 --2- 172.21.15.38:0/2369280593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48f80098a0 tx=0x7f48f8009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- 172.21.15.38:0/2369280593 shutdown_connections 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 --2- 172.21.15.38:0/2369280593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc106530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- 172.21.15.38:0/2369280593 >> 172.21.15.38:0/2369280593 conn(0x7f48fc101a60 msgr2=0x7f48fc103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:28.976 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- 172.21.15.38:0/2369280593 shutdown_connections 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- 172.21.15.38:0/2369280593 wait complete. 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 Processor -- start 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.042+0000 7f490466d700 1 -- start start 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f490466d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f490466d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48fc10c450 con 0x7f48fc106110 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f4902409700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f4902409700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38406/0 (socket says 172.21.15.38:38406) 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f4902409700 1 -- 172.21.15.38:0/2766283575 learned_addr learned my addr 172.21.15.38:0/2766283575 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f4902409700 1 -- 172.21.15.38:0/2766283575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48fc108e60 con 0x7f48fc106110 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f4902409700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f48f80117e0 tx=0x7f48f8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f8003bf0 con 0x7f48fc106110 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48fc107050 con 0x7f48fc106110 2024-04-21T23:52:28.977 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f801a910 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.043+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f80249a0 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.044+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48fc107510 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.044+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f48f8022020 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.045+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48fc063890 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.045+0000 7f48f37fe700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 0x7f48e803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.045+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f48f804d6e0 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.045+0000 7f4901c08700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 0x7f48e803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.045+0000 7f4901c08700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 0x7f48e803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48ec006fd0 tx=0x7f48ec006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.050+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f48f8011e20 con 0x7f48fc106110 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.268+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f48fc107e20 con 0x7f48e8039050 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.381+0000 7f48f37fe700 1 -- 172.21.15.38:0/2766283575 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f48fc107e20 con 0x7f48e8039050 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 msgr2=0x7f48e803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 0x7f48e803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48ec006fd0 tx=0x7f48ec006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:28.978 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 msgr2=0x7f48fc108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f48f80117e0 tx=0x7f48f8011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 shutdown_connections 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f48e8039050 0x7f48e803b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 --2- 172.21.15.38:0/2766283575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48fc106110 0x7f48fc108920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 >> 172.21.15.38:0/2766283575 conn(0x7f48fc101a60 msgr2=0x7f48fc102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 shutdown_connections 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:28.383+0000 7f490466d700 1 -- 172.21.15.38:0/2766283575 wait complete. 2024-04-21T23:52:28.979 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-04-21T23:52:30.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:30.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:31.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:31 smithi038 ceph-mon[27972]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:31.254 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.JIkEa.E/fYQaDchONU2UefTrLxxikqwDXCaf5P0RDTtY5oexAZom", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713743549, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.425+0000 7fd9a8755700 1 Processor -- start 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.425+0000 7fd9a8755700 1 -- start start 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.425+0000 7fd9a8755700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.425+0000 7fd9a8755700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a007be10 con 0x7fd9a0106700 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a64f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a64f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38420/0 (socket says 172.21.15.38:38420) 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a64f1700 1 -- 172.21.15.38:0/1428586918 learned_addr learned my addr 172.21.15.38:0/1428586918 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a64f1700 1 -- 172.21.15.38:0/1428586918 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a0107060 con 0x7fd9a0106700 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a64f1700 1 --2- 172.21.15.38:0/1428586918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9900098a0 tx=0x7fd990009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693773b4ca3e83d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a54ef700 1 -- 172.21.15.38:0/1428586918 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd990004160 con 0x7fd9a0106700 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a54ef700 1 -- 172.21.15.38:0/1428586918 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd990004300 con 0x7fd9a0106700 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.426+0000 7fd9a54ef700 1 -- 172.21.15.38:0/1428586918 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd990011b20 con 0x7fd9a0106700 2024-04-21T23:52:31.255 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 -- 172.21.15.38:0/1428586918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 msgr2=0x7fd9a0106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1428586918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9900098a0 tx=0x7fd990009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 -- 172.21.15.38:0/1428586918 shutdown_connections 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1428586918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a0106b20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 -- 172.21.15.38:0/1428586918 >> 172.21.15.38:0/1428586918 conn(0x7fd9a01022e0 msgr2=0x7fd9a0104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 -- 172.21.15.38:0/1428586918 shutdown_connections 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.427+0000 7fd9a8755700 1 -- 172.21.15.38:0/1428586918 wait complete. 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a8755700 1 Processor -- start 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a8755700 1 -- start start 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a8755700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a8755700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0108130 con 0x7fd9a0106700 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a64f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a64f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38436/0 (socket says 172.21.15.38:38436) 2024-04-21T23:52:31.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.428+0000 7fd9a64f1700 1 -- 172.21.15.38:0/1246992904 learned_addr learned my addr 172.21.15.38:0/1246992904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9a64f1700 1 -- 172.21.15.38:0/1246992904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a019e490 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9a64f1700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd9900117e0 tx=0x7fd990011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd990003bf0 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99001a910 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a019e720 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd990024970 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.429+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a019eb80 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.430+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fd990022020 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.430+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a0063890 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.430+0000 7fd9977fe700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 0x7fd98c03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.430+0000 7fd9a5cf0700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 0x7fd98c03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.430+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd99004d4f0 con 0x7fd9a0106700 2024-04-21T23:52:31.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.431+0000 7fd9a5cf0700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 0x7fd98c03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.435+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd990011e20 con 0x7fd9a0106700 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.681+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd9a0197a30 con 0x7fd98c0390b0 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.929+0000 7fd9977fe700 1 -- 172.21.15.38:0/1246992904 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd9a0197a30 con 0x7fd98c0390b0 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 msgr2=0x7fd98c03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 0x7fd98c03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 msgr2=0x7fd9a019df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd9900117e0 tx=0x7fd990011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 shutdown_connections 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd98c0390b0 0x7fd98c03b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 --2- 172.21.15.38:0/1246992904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9a0106700 0x7fd9a019df50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 >> 172.21.15.38:0/1246992904 conn(0x7fd9a01022e0 msgr2=0x7fd9a0194b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:31.258 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 shutdown_connections 2024-04-21T23:52:31.259 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:29.932+0000 7fd9a8755700 1 -- 172.21.15.38:0/1246992904 wait complete. 2024-04-21T23:52:31.259 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-04-21T23:52:32.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:32 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/401807147' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T23:52:32.437 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.694+0000 7feb33481700 1 Processor -- start 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.694+0000 7feb33481700 1 -- start start 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.694+0000 7feb33481700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.694+0000 7feb33481700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c07d090 con 0x7feb2c07e650 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb3121d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb3121d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38444/0 (socket says 172.21.15.38:38444) 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb3121d700 1 -- 172.21.15.38:0/23670217 learned_addr learned my addr 172.21.15.38:0/23670217 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb3121d700 1 -- 172.21.15.38:0/23670217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c07d210 con 0x7feb2c07e650 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb3121d700 1 --2- 172.21.15.38:0/23670217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb24009e80 tx=0x7feb2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6083bd9b50450de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb2bfff700 1 -- 172.21.15.38:0/23670217 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb24017070 con 0x7feb2c07e650 2024-04-21T23:52:32.438 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb2bfff700 1 -- 172.21.15.38:0/23670217 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2400b8c0 con 0x7feb2c07e650 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.695+0000 7feb2bfff700 1 -- 172.21.15.38:0/23670217 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb240045c0 con 0x7feb2c07e650 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 -- 172.21.15.38:0/23670217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 msgr2=0x7feb2c07cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 --2- 172.21.15.38:0/23670217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb24009e80 tx=0x7feb2400b330 comp rx=0 tx=0).stop 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 -- 172.21.15.38:0/23670217 shutdown_connections 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 --2- 172.21.15.38:0/23670217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c07cb50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 -- 172.21.15.38:0/23670217 >> 172.21.15.38:0/23670217 conn(0x7feb2c103470 msgr2=0x7feb2c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 -- 172.21.15.38:0/23670217 shutdown_connections 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 -- 172.21.15.38:0/23670217 wait complete. 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.696+0000 7feb33481700 1 Processor -- start 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb33481700 1 -- start start 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb33481700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb33481700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c10e410 con 0x7feb2c07e650 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb3121d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:32.439 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb3121d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38460/0 (socket says 172.21.15.38:38460) 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb3121d700 1 -- 172.21.15.38:0/401807147 learned_addr learned my addr 172.21.15.38:0/401807147 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb3121d700 1 -- 172.21.15.38:0/401807147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c19e4d0 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb3121d700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb24009fd0 tx=0x7feb24003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb24020020 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2400bba0 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2c19e760 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb24017040 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.697+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb2c1a1450 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.698+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7feb24023020 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.698+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb2c063890 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.698+0000 7feb2a7fc700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.699+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7feb2400f050 con 0x7feb2c07e650 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.699+0000 7feb30a1c700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:32.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.699+0000 7feb30a1c700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 0x7feb1403b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.704+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feb2402a020 con 0x7feb2c07e650 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.911+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7feb2c1a1620 con 0x7feb2c07e650 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.911+0000 7feb2a7fc700 1 -- 172.21.15.38:0/401807147 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7feb2c1a1620 con 0x7feb2c07e650 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 msgr2=0x7feb1403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 0x7feb1403b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feb20006fd0 tx=0x7feb20006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 msgr2=0x7feb2c19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb24009fd0 tx=0x7feb24003d00 comp rx=0 tx=0).stop 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 shutdown_connections 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 --2- 172.21.15.38:0/401807147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb2c07e650 0x7feb2c19df90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 >> 172.21.15.38:0/401807147 conn(0x7feb2c103470 msgr2=0x7feb2c103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.914+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 shutdown_connections 2024-04-21T23:52:32.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:31.915+0000 7feb33481700 1 -- 172.21.15.38:0/401807147 wait complete. 2024-04-21T23:52:32.445 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T23:52:32.446 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-04-21T23:52:32.453 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T23:52:32.454 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-04-21T23:52:32.454 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-04-21T23:52:32.454 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T23:52:32.455 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-04-21T23:52:32.455 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:32.455 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038:8443/ 2024-04-21T23:52:32.455 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-04-21T23:52:32.455 INFO:teuthology.orchestra.run.smithi038.stdout: Password: q49qp08r6p 2024-04-21T23:52:32.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:32.456 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config directory 2024-04-21T23:52:32.456 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-04-21T23:52:33.114 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:33 smithi038 ceph-mon[27972]: mgrmap e13: x(active, since 6s) 2024-04-21T23:52:33.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.904+0000 7f7e246d2700 1 Processor -- start 2024-04-21T23:52:33.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.904+0000 7f7e246d2700 1 -- start start 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.904+0000 7f7e246d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.904+0000 7f7e246d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c108790 con 0x7f7e1c107e30 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2246e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2246e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38474/0 (socket says 172.21.15.38:38474) 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2246e700 1 -- 172.21.15.38:0/2729710660 learned_addr learned my addr 172.21.15.38:0/2729710660 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2246e700 1 -- 172.21.15.38:0/2729710660 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c108910 con 0x7f7e1c107e30 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2246e700 1 --2- 172.21.15.38:0/2729710660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009e80 tx=0x7f7e1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7daad8ad404533cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2146c700 1 -- 172.21.15.38:0/2729710660 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e14017070 con 0x7f7e1c107e30 2024-04-21T23:52:33.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2146c700 1 -- 172.21.15.38:0/2729710660 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1400b8c0 con 0x7f7e1c107e30 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.905+0000 7f7e2146c700 1 -- 172.21.15.38:0/2729710660 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e140045c0 con 0x7f7e1c107e30 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 -- 172.21.15.38:0/2729710660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 msgr2=0x7f7e1c108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 --2- 172.21.15.38:0/2729710660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009e80 tx=0x7f7e1400b330 comp rx=0 tx=0).stop 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 -- 172.21.15.38:0/2729710660 shutdown_connections 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 --2- 172.21.15.38:0/2729710660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c108250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 -- 172.21.15.38:0/2729710660 >> 172.21.15.38:0/2729710660 conn(0x7f7e1c103450 msgr2=0x7f7e1c1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 -- 172.21.15.38:0/2729710660 shutdown_connections 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.906+0000 7f7e246d2700 1 -- 172.21.15.38:0/2729710660 wait complete. 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e246d2700 1 Processor -- start 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e246d2700 1 -- start start 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e246d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e246d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c10c2a0 con 0x7f7e1c107e30 2024-04-21T23:52:33.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e2246e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e2246e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38490/0 (socket says 172.21.15.38:38490) 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e2246e700 1 -- 172.21.15.38:0/3658612534 learned_addr learned my addr 172.21.15.38:0/3658612534 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e2246e700 1 -- 172.21.15.38:0/3658612534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c07e920 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e2246e700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b7b0 tx=0x7f7e1400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e14020020 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14003d90 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.907+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c07cb10 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.908+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e14017040 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.908+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c07d050 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.908+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7e14023020 con 0x7f7e1c107e30 2024-04-21T23:52:33.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.908+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1c063890 con 0x7f7e1c107e30 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.909+0000 7f7e1b7fe700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 0x7f7e0403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.909+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7e1400f050 con 0x7f7e1c107e30 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.909+0000 7f7e21c6d700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 0x7f7e0403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.909+0000 7f7e21c6d700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 0x7f7e0403b5b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e10006fd0 tx=0x7f7e10006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:32.914+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7e1402a020 con 0x7f7e1c107e30 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.121+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f7e1c07d220 con 0x7f7e1c107e30 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.121+0000 7f7e1b7fe700 1 -- 172.21.15.38:0/3658612534 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f7e1402b260 con 0x7f7e1c107e30 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 msgr2=0x7f7e0403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 0x7f7e0403b5b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7e10006fd0 tx=0x7f7e10006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 msgr2=0x7f7e1c07e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b7b0 tx=0x7f7e1400bd50 comp rx=0 tx=0).stop 2024-04-21T23:52:33.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 shutdown_connections 2024-04-21T23:52:33.691 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7e040390f0 0x7f7e0403b5b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:33.691 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 --2- 172.21.15.38:0/3658612534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7e1c107e30 0x7f7e1c07e3e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:33.691 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 >> 172.21.15.38:0/3658612534 conn(0x7f7e1c103450 msgr2=0x7f7e1c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:33.691 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 shutdown_connections 2024-04-21T23:52:33.691 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:33.124+0000 7f7e246d2700 1 -- 172.21.15.38:0/3658612534 wait complete. 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.210+0000 7f5fc748a700 1 Processor -- start 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc748a700 1 -- start start 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc748a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc748a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc007d0b0 con 0x7f5fc007e600 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc5226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc5226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33228/0 (socket says 172.21.15.38:33228) 2024-04-21T23:52:35.110 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc5226700 1 -- 172.21.15.38:0/3844578774 learned_addr learned my addr 172.21.15.38:0/3844578774 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc5226700 1 -- 172.21.15.38:0/3844578774 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc007d230 con 0x7f5fc007e600 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.211+0000 7f5fc5226700 1 --2- 172.21.15.38:0/3844578774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00098a0 tx=0x7f5fb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9152fda579cb62f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fb7fff700 1 -- 172.21.15.38:0/3844578774 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0004160 con 0x7f5fc007e600 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fb7fff700 1 -- 172.21.15.38:0/3844578774 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb0004300 con 0x7f5fc007e600 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fb7fff700 1 -- 172.21.15.38:0/3844578774 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0011b20 con 0x7f5fc007e600 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 -- 172.21.15.38:0/3844578774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 msgr2=0x7f5fc007cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 --2- 172.21.15.38:0/3844578774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00098a0 tx=0x7f5fb0009bb0 comp rx=0 tx=0).stop 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 -- 172.21.15.38:0/3844578774 shutdown_connections 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 --2- 172.21.15.38:0/3844578774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc007cb70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 -- 172.21.15.38:0/3844578774 >> 172.21.15.38:0/3844578774 conn(0x7f5fc01034b0 msgr2=0x7f5fc01058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 -- 172.21.15.38:0/3844578774 shutdown_connections 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.212+0000 7f5fc748a700 1 -- 172.21.15.38:0/3844578774 wait complete. 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc748a700 1 Processor -- start 2024-04-21T23:52:35.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc748a700 1 -- start start 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc748a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc748a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc010e360 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc5226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc5226700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33234/0 (socket says 172.21.15.38:33234) 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.213+0000 7f5fc5226700 1 -- 172.21.15.38:0/4221179473 learned_addr learned my addr 172.21.15.38:0/4221179473 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fc5226700 1 -- 172.21.15.38:0/4221179473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc019e450 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fc5226700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00117e0 tx=0x7f5fb0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0003bf0 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb001a910 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc019e6e0 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0024970 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.214+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc019eb40 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.215+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f5fb0022020 con 0x7f5fc007e600 2024-04-21T23:52:35.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.215+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc0063890 con 0x7f5fc007e600 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.215+0000 7f5fb67fc700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 0x7f5fac03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.215+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5fb004d720 con 0x7f5fc007e600 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.215+0000 7f5fc4a25700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 0x7f5fac03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.216+0000 7f5fc4a25700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 0x7f5fac03b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc006fd0 tx=0x7f5fbc006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.220+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5fb0011e20 con 0x7f5fc007e600 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.544+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f5fc01a15d0 con 0x7f5fc007e600 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.547+0000 7f5fb67fc700 1 -- 172.21.15.38:0/4221179473 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f5fb000bc70 con 0x7f5fc007e600 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 msgr2=0x7f5fac03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 0x7f5fac03b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc006fd0 tx=0x7f5fbc006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 msgr2=0x7f5fc019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00117e0 tx=0x7f5fb0011c10 comp rx=0 tx=0).stop 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 shutdown_connections 2024-04-21T23:52:35.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5fac039150 0x7f5fac03b610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 --2- 172.21.15.38:0/4221179473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fc007e600 0x7f5fc019df10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 >> 172.21.15.38:0/4221179473 conn(0x7f5fc01034b0 msgr2=0x7f5fc0103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 shutdown_connections 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-04-21T23:52:34.549+0000 7f5fc748a700 1 -- 172.21.15.38:0/4221179473 wait complete. 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.114 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:35.115 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-04-21T23:52:35.150 INFO:tasks.cephadm:Fetching config... 2024-04-21T23:52:35.151 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:52:35.151 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T23:52:35.168 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T23:52:35.168 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:52:35.169 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T23:52:35.224 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T23:52:35.224 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:52:35.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T23:52:35.290 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T23:52:35.290 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:52:35.290 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T23:52:35.345 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T23:52:35.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:52:35.420 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:52:35.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:52:35.470 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:52:35.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T23:52:35.521 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/8nZAjy1b4ICtAPGav0Phq2mjb1qiJKe7IMgj0tR2fW/DiXz/H1qgY7WzWfx9Q2QH7mzaDtun3WdOdn+4WE+LJ1qBgFMy9ozGqlFBKp/0Pj0/G83nnhCHWdA1+KyFh7ZTnIIew/wncVpNEB2VhhYg5n14XX8woNo+TbQCVPX8zQWfq6Q+3ta7OzeZ+YxRYtNBvYVALtKBXzASCBs2alaYwAUyO1TYD11bbsrHrWa1ZZt2V0lrMD8sdWKPqExNg0VdF4N/Nwa2lGMSzzVhDaqgHuab76Y82ESeO18Lq3mgT6PKFeBsD1DKszk2kXlbkFCgs9VRFYD/4siDvE6I+2DRACsjpN843x2VZ7ya8dwYrZMb9qK5YHrrcJ5mdU7gDc/xNapdXV8t9qv22TQz+ZxW90ovX+ebCdf0+KStD2iRPVPvK/29M0OT0vRI9XsCA70KAZBXzpnvAEO8YhbrMGHkE1JBv3eczB78DThHDFO+xVM2mW6sJuqjXtSmCavzmD0= ceph-eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:52:35.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T23:52:35.638 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:35 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4221179473' entity='client.admin' 2024-04-21T23:52:35.781 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.383+0000 7fee23fa7700 1 -- 172.21.15.38:0/1116432669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 msgr2=0x7fee1c100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.383+0000 7fee23fa7700 1 --2- 172.21.15.38:0/1116432669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c100fb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fee18009910 tx=0x7fee18009c20 comp rx=0 tx=0).stop 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.383+0000 7fee23fa7700 1 -- 172.21.15.38:0/1116432669 shutdown_connections 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.383+0000 7fee23fa7700 1 --2- 172.21.15.38:0/1116432669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c100fb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.383+0000 7fee23fa7700 1 -- 172.21.15.38:0/1116432669 >> 172.21.15.38:0/1116432669 conn(0x7fee1c0fc130 msgr2=0x7fee1c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 -- 172.21.15.38:0/1116432669 shutdown_connections 2024-04-21T23:52:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 -- 172.21.15.38:0/1116432669 wait complete. 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 Processor -- start 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 -- start start 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.384+0000 7fee23fa7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee1c106c20 con 0x7fee1c100b90 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee21d43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee21d43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33252/0 (socket says 172.21.15.38:33252) 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee21d43700 1 -- 172.21.15.38:0/504070370 learned_addr learned my addr 172.21.15.38:0/504070370 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee21d43700 1 -- 172.21.15.38:0/504070370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1c1978d0 con 0x7fee1c100b90 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee21d43700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee18006010 tx=0x7fee18005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee18027070 con 0x7fee1c100b90 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee1c197b60 con 0x7fee1c100b90 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1800be40 con 0x7fee1c100b90 2024-04-21T23:52:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee1800f460 con 0x7fee1c100b90 2024-04-21T23:52:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.385+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1c19a850 con 0x7fee1c100b90 2024-04-21T23:52:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.386+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee1c1073a0 con 0x7fee1c100b90 2024-04-21T23:52:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.386+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fee1802a070 con 0x7fee1c100b90 2024-04-21T23:52:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.387+0000 7fee12ffd700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 0x7fee0803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.387+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fee1801e030 con 0x7fee1c100b90 2024-04-21T23:52:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.387+0000 7fee21542700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 0x7fee0803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.387+0000 7fee21542700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 0x7fee0803b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:38.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.393+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fee18013070 con 0x7fee1c100b90 2024-04-21T23:52:38.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.600+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fee1c02dc40 con 0x7fee1c100b90 2024-04-21T23:52:38.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.604+0000 7fee12ffd700 1 -- 172.21.15.38:0/504070370 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fee18031020 con 0x7fee1c100b90 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.606+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 msgr2=0x7fee0803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.606+0000 7fee23fa7700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 0x7fee0803b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee0c006fd0 tx=0x7fee0c006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.606+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 msgr2=0x7fee1c197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.606+0000 7fee23fa7700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee18006010 tx=0x7fee18005050 comp rx=0 tx=0).stop 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 shutdown_connections 2024-04-21T23:52:38.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fee08039150 0x7fee0803b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:38.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 --2- 172.21.15.38:0/504070370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee1c100b90 0x7fee1c197390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:38.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 >> 172.21.15.38:0/504070370 conn(0x7fee1c0fc130 msgr2=0x7fee1c069a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:38.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 shutdown_connections 2024-04-21T23:52:38.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:38.607+0000 7fee23fa7700 1 -- 172.21.15.38:0/504070370 wait complete. 2024-04-21T23:52:39.418 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T23:52:39.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T23:52:39.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:39 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/504070370' entity='client.admin' 2024-04-21T23:52:39.847 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:41.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 -- 172.21.15.38:0/1175112449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 msgr2=0x7f30a8101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1175112449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a8101000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f309c009910 tx=0x7f309c009c20 comp rx=0 tx=0).stop 2024-04-21T23:52:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 -- 172.21.15.38:0/1175112449 shutdown_connections 2024-04-21T23:52:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1175112449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a8101000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 -- 172.21.15.38:0/1175112449 >> 172.21.15.38:0/1175112449 conn(0x7f30a80fc140 msgr2=0x7f30a80fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 -- 172.21.15.38:0/1175112449 shutdown_connections 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.389+0000 7f30aebd0700 1 -- 172.21.15.38:0/1175112449 wait complete. 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30aebd0700 1 Processor -- start 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30aebd0700 1 -- start start 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30aebd0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30aebd0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f30a8106c70 con 0x7f30a8100be0 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30ac96c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30ac96c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33284/0 (socket says 172.21.15.38:33284) 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.390+0000 7f30ac96c700 1 -- 172.21.15.38:0/1759154969 learned_addr learned my addr 172.21.15.38:0/1759154969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30ac96c700 1 -- 172.21.15.38:0/1759154969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a80fe580 con 0x7f30a8100be0 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30ac96c700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f309c005850 tx=0x7f309c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f309c026070 con 0x7f30a8100be0 2024-04-21T23:52:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a80fc770 con 0x7f30a8100be0 2024-04-21T23:52:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f309c0044a0 con 0x7f30a8100be0 2024-04-21T23:52:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f309c003ab0 con 0x7f30a8100be0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.391+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a80fccb0 con 0x7f30a8100be0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.392+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f309c029070 con 0x7f30a8100be0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.392+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a81073f0 con 0x7f30a8100be0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.392+0000 7f30a5ffb700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 0x7f309803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.392+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f309c01e070 con 0x7f30a8100be0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.392+0000 7f30a7fff700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 0x7f309803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:41.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.393+0000 7f30a7fff700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 0x7f309803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.397+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f309c013070 con 0x7f30a8100be0 2024-04-21T23:52:41.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.608+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f30a8190910 con 0x7f30980390f0 2024-04-21T23:52:41.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.614+0000 7f30a5ffb700 1 -- 172.21.15.38:0/1759154969 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30a8190910 con 0x7f30980390f0 2024-04-21T23:52:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 msgr2=0x7f309803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 0x7f309803b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3094006fd0 tx=0x7f3094006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 msgr2=0x7f30a80fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f309c005850 tx=0x7f309c005030 comp rx=0 tx=0).stop 2024-04-21T23:52:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 shutdown_connections 2024-04-21T23:52:41.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30980390f0 0x7f309803b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:41.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 --2- 172.21.15.38:0/1759154969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a8100be0 0x7f30a80fe040 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:41.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 >> 172.21.15.38:0/1759154969 conn(0x7f30a80fc140 msgr2=0x7f30a806a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:41.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 shutdown_connections 2024-04-21T23:52:41.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:41.616+0000 7f30aebd0700 1 -- 172.21.15.38:0/1759154969 wait complete. 2024-04-21T23:52:42.384 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-04-21T23:52:42.384 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:52:42.384 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-04-21T23:52:42.402 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:52:42.403 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:52:42.463 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-04-21T23:52:42.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch host add smithi111 2024-04-21T23:52:42.710 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:42.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:44.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 -- 172.21.15.38:0/3691789452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 msgr2=0x7facf41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:44.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 --2- 172.21.15.38:0/3691789452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41011c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7face4009910 tx=0x7face4009c20 comp rx=0 tx=0).stop 2024-04-21T23:52:44.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 -- 172.21.15.38:0/3691789452 shutdown_connections 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 --2- 172.21.15.38:0/3691789452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41011c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 -- 172.21.15.38:0/3691789452 >> 172.21.15.38:0/3691789452 conn(0x7facf40fc690 msgr2=0x7facf40feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 -- 172.21.15.38:0/3691789452 shutdown_connections 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.219+0000 7facfa769700 1 -- 172.21.15.38:0/3691789452 wait complete. 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facfa769700 1 Processor -- start 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facfa769700 1 -- start start 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facfa769700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facfa769700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7facf410b1b0 con 0x7facf4100de0 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facf3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facf3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36296/0 (socket says 172.21.15.38:36296) 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facf3fff700 1 -- 172.21.15.38:0/2191733241 learned_addr learned my addr 172.21.15.38:0/2191733241 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facf3fff700 1 -- 172.21.15.38:0/2191733241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf4198b00 con 0x7facf4100de0 2024-04-21T23:52:44.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.220+0000 7facf3fff700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7face4005850 tx=0x7face4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:44.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.221+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7face4026070 con 0x7facf4100de0 2024-04-21T23:52:44.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.221+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face400bbd0 con 0x7facf4100de0 2024-04-21T23:52:44.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.221+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf4198df0 con 0x7facf4100de0 2024-04-21T23:52:44.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.221+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7face4003970 con 0x7facf4100de0 2024-04-21T23:52:44.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.221+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf41992b0 con 0x7facf4100de0 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7face4029070 con 0x7facf4100de0 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf41022c0 con 0x7facf4100de0 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facf17fa700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 0x7face003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7face401e070 con 0x7facf4100de0 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facf37fe700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 0x7face003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:44.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.222+0000 7facf37fe700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 0x7face003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facdc006fd0 tx=0x7facdc006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:44.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.227+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7face4013070 con 0x7facf4100de0 2024-04-21T23:52:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:44.438+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}) -- 0x7facf406a6f0 con 0x7face00390f0 2024-04-21T23:52:44.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:44 smithi038 ceph-mon[27972]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:45.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:45 smithi038 ceph-mon[27972]: Deploying cephadm binary to smithi111 2024-04-21T23:52:46.839 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:46 smithi038 ceph-mon[27972]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:48.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.660+0000 7facf17fa700 1 -- 172.21.15.38:0/2191733241 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7facf406a6f0 con 0x7face00390f0 2024-04-21T23:52:48.654 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-04-21T23:52:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 msgr2=0x7face003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 0x7face003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facdc006fd0 tx=0x7facdc006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 msgr2=0x7facf41985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:48.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7face4005850 tx=0x7face4005030 comp rx=0 tx=0).stop 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 shutdown_connections 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7face00390f0 0x7face003b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 --2- 172.21.15.38:0/2191733241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4100de0 0x7facf41985c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 >> 172.21.15.38:0/2191733241 conn(0x7facf40fc690 msgr2=0x7facf4103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 shutdown_connections 2024-04-21T23:52:48.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:48.663+0000 7facfa769700 1 -- 172.21.15.38:0/2191733241 wait complete. 2024-04-21T23:52:48.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:48 smithi038 ceph-mon[27972]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:49.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T23:52:49.524 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:49.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:49.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:49 smithi038 ceph-mon[27972]: Added host smithi111 2024-04-21T23:52:49.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:49.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:50 smithi038 ceph-mon[27972]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 -- 172.21.15.38:0/944427399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 msgr2=0x7f35d01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/944427399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d01031d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35c4009910 tx=0x7f35c4009c20 comp rx=0 tx=0).stop 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 -- 172.21.15.38:0/944427399 shutdown_connections 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/944427399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d01031d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 -- 172.21.15.38:0/944427399 >> 172.21.15.38:0/944427399 conn(0x7f35d00fc660 msgr2=0x7f35d00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 -- 172.21.15.38:0/944427399 shutdown_connections 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.163+0000 7f35d4ef4700 1 -- 172.21.15.38:0/944427399 wait complete. 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35d4ef4700 1 Processor -- start 2024-04-21T23:52:51.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35d4ef4700 1 -- start start 2024-04-21T23:52:51.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35d4ef4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35d4ef4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010c2e0 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35ce59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36312/0 (socket says 172.21.15.38:36312) 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35ce59c700 1 -- 172.21.15.38:0/1192194831 learned_addr learned my addr 172.21.15.38:0/1192194831 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35ce59c700 1 -- 172.21.15.38:0/1192194831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d019dab0 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.164+0000 7f35ce59c700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006010 tx=0x7f35c4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35c4026070 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d019dd40 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c400bc30 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35c40039a0 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d0197640 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.165+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f35c4029070 con 0x7f35d0102df0 2024-04-21T23:52:51.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.166+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d006aa20 con 0x7f35d0102df0 2024-04-21T23:52:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.166+0000 7f35bf7fe700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 0x7f35b803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.166+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f35c401e070 con 0x7f35d0102df0 2024-04-21T23:52:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.166+0000 7f35cdd9b700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 0x7f35b803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.166+0000 7f35cdd9b700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 0x7f35b803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f35c0006fd0 tx=0x7f35c0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.171+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f35c4013070 con 0x7f35d0102df0 2024-04-21T23:52:51.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.384+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35d0063890 con 0x7f35b80390f0 2024-04-21T23:52:51.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.385+0000 7f35bf7fe700 1 -- 172.21.15.38:0/1192194831 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f35d0063890 con 0x7f35b80390f0 2024-04-21T23:52:51.378 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:52:51.379 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 msgr2=0x7f35b803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 0x7f35b803b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f35c0006fd0 tx=0x7f35c0006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 msgr2=0x7f35d019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c4006010 tx=0x7f35c4005030 comp rx=0 tx=0).stop 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 shutdown_connections 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f35b80390f0 0x7f35b803b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 --2- 172.21.15.38:0/1192194831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35d0102df0 0x7f35d019d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 >> 172.21.15.38:0/1192194831 conn(0x7f35d00fc660 msgr2=0x7f35d0103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.387+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 shutdown_connections 2024-04-21T23:52:51.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:51.388+0000 7f35d4ef4700 1 -- 172.21.15.38:0/1192194831 wait complete. 2024-04-21T23:52:52.075 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-04-21T23:52:52.076 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:52:52.076 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-04-21T23:52:52.094 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:52:52.094 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:52:52.152 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-04-21T23:52:52.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch host add smithi203 2024-04-21T23:52:52.397 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:52.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:52.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:52 smithi038 ceph-mon[27972]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:52:53.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:53 smithi038 ceph-mon[27972]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:54.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/3970566774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 msgr2=0x7f5b14106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/3970566774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14106fe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5b04009910 tx=0x7f5b04009c20 comp rx=0 tx=0).stop 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/3970566774 shutdown_connections 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/3970566774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14106fe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/3970566774 >> 172.21.15.38:0/3970566774 conn(0x7f5b14076690 msgr2=0x7f5b14076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/3970566774 shutdown_connections 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.070+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/3970566774 wait complete. 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b1bdb0700 1 Processor -- start 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b1bdb0700 1 -- start start 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b1bdb0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b1bdb0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1410b1c0 con 0x7f5b14106c00 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b19b4c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:54.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b19b4c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39836/0 (socket says 172.21.15.38:39836) 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.071+0000 7f5b19b4c700 1 -- 172.21.15.38:0/2740031141 learned_addr learned my addr 172.21.15.38:0/2740031141 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b19b4c700 1 -- 172.21.15.38:0/2740031141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b14198ab0 con 0x7f5b14106c00 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b19b4c700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5b04006010 tx=0x7f5b04005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b04026070 con 0x7f5b14106c00 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b14198d40 con 0x7f5b14106c00 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0400bc30 con 0x7f5b14106c00 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b040039d0 con 0x7f5b14106c00 2024-04-21T23:52:54.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.072+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b141991a0 con 0x7f5b14106c00 2024-04-21T23:52:54.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.073+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f5b04029070 con 0x7f5b14106c00 2024-04-21T23:52:54.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.073+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b14102200 con 0x7f5b14106c00 2024-04-21T23:52:54.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.073+0000 7f5b0affd700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 0x7f5b0003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:52:54.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.073+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5b0401e070 con 0x7f5b14106c00 2024-04-21T23:52:54.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.073+0000 7f5b1934b700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 0x7f5b0003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:52:54.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.074+0000 7f5b1934b700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 0x7f5b0003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5b10006fd0 tx=0x7f5b10006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:52:54.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.078+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5b04013070 con 0x7f5b14106c00 2024-04-21T23:52:54.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:54.288+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f5b14069c40 con 0x7f5b000390f0 2024-04-21T23:52:55.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:52:55.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:55 smithi038 ceph-mon[27972]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:52:56.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:52:56.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:52:56.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: Deploying cephadm binary to smithi203 2024-04-21T23:52:56.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:52:56.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:56.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:56.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:57.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:57 smithi038 ceph-mon[27972]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:58.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.582+0000 7f5b0affd700 1 -- 172.21.15.38:0/2740031141 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5b14069c40 con 0x7f5b000390f0 2024-04-21T23:52:58.577 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 msgr2=0x7f5b0003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 0x7f5b0003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5b10006fd0 tx=0x7f5b10006e40 comp rx=0 tx=0).stop 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 msgr2=0x7f5b14198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5b04006010 tx=0x7f5b04005030 comp rx=0 tx=0).stop 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 shutdown_connections 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5b000390f0 0x7f5b0003b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 --2- 172.21.15.38:0/2740031141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b14106c00 0x7f5b14198570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 >> 172.21.15.38:0/2740031141 conn(0x7f5b14076690 msgr2=0x7f5b140ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 shutdown_connections 2024-04-21T23:52:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:52:58.585+0000 7f5b1bdb0700 1 -- 172.21.15.38:0/2740031141 wait complete. 2024-04-21T23:52:59.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T23:52:59.433 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:52:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:59 smithi038 ceph-mon[27972]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:52:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:52:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:52:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:52:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:00.660 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:00 smithi038 ceph-mon[27972]: Added host smithi203 2024-04-21T23:53:00.660 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:00 smithi038 ceph-mon[27972]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:00.660 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 -- 172.21.15.38:0/4039576515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 msgr2=0x7f80d81022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 --2- 172.21.15.38:0/4039576515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d81022b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009910 tx=0x7f80c8009c20 comp rx=0 tx=0).stop 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 -- 172.21.15.38:0/4039576515 shutdown_connections 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 --2- 172.21.15.38:0/4039576515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d81022b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 -- 172.21.15.38:0/4039576515 >> 172.21.15.38:0/4039576515 conn(0x7f80d80fd760 msgr2=0x7f80d80ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 -- 172.21.15.38:0/4039576515 shutdown_connections 2024-04-21T23:53:00.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.905+0000 7f80dcad9700 1 -- 172.21.15.38:0/4039576515 wait complete. 2024-04-21T23:53:00.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.906+0000 7f80dcad9700 1 Processor -- start 2024-04-21T23:53:00.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.906+0000 7f80dcad9700 1 -- start start 2024-04-21T23:53:00.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.906+0000 7f80dcad9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:00.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.906+0000 7f80dcad9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80d810c2a0 con 0x7f80d8101ed0 2024-04-21T23:53:00.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.906+0000 7f80d659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80d659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39860/0 (socket says 172.21.15.38:39860) 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80d659c700 1 -- 172.21.15.38:0/221667345 learned_addr learned my addr 172.21.15.38:0/221667345 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80d659c700 1 -- 172.21.15.38:0/221667345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d8076420 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80d659c700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80c8006010 tx=0x7f80c8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80c8026070 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c800bc30 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d8074ad0 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80c80039d0 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.907+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d8074f90 con 0x7f80d8101ed0 2024-04-21T23:53:00.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.908+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f80c8029070 con 0x7f80d8101ed0 2024-04-21T23:53:00.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.908+0000 7f80cf7fe700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 0x7f80c403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:00.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.908+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d81033b0 con 0x7f80d8101ed0 2024-04-21T23:53:00.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.908+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f80c801e070 con 0x7f80d8101ed0 2024-04-21T23:53:00.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.909+0000 7f80d5d9b700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 0x7f80c403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:00.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.909+0000 7f80d5d9b700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 0x7f80c403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f80c0006fd0 tx=0x7f80c0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:00.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:00.914+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f80c8013070 con 0x7f80d8101ed0 2024-04-21T23:53:01.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.127+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80d8075220 con 0x7f80c4039150 2024-04-21T23:53:01.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.128+0000 7f80cf7fe700 1 -- 172.21.15.38:0/221667345 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f80d8075220 con 0x7f80c4039150 2024-04-21T23:53:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:53:01.122 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-04-21T23:53:01.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 msgr2=0x7f80c403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 0x7f80c403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f80c0006fd0 tx=0x7f80c0006e40 comp rx=0 tx=0).stop 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 msgr2=0x7f80d8074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80c8006010 tx=0x7f80c8005030 comp rx=0 tx=0).stop 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 shutdown_connections 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80c4039150 0x7f80c403b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 --2- 172.21.15.38:0/221667345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80d8101ed0 0x7f80d8074530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 >> 172.21.15.38:0/221667345 conn(0x7f80d80fd760 msgr2=0x7f80d8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 shutdown_connections 2024-04-21T23:53:01.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:01.130+0000 7f80dcad9700 1 -- 172.21.15.38:0/221667345 wait complete. 2024-04-21T23:53:01.791 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T23:53:01.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T23:53:02.032 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:53:02.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:02 smithi038 ceph-mon[27972]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:53:02.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:03.355 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:03 smithi038 ceph-mon[27972]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1928844861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 msgr2=0x7fd6e81022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1928844861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e81022b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009910 tx=0x7fd6d0009c20 comp rx=0 tx=0).stop 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1928844861 shutdown_connections 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1928844861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e81022b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1928844861 >> 172.21.15.38:0/1928844861 conn(0x7fd6e80fd760 msgr2=0x7fd6e80ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1928844861 shutdown_connections 2024-04-21T23:53:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.568+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1928844861 wait complete. 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6ed60c700 1 Processor -- start 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6ed60c700 1 -- start start 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6ed60c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6ed60c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e810c2a0 con 0x7fd6e8101ed0 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39874/0 (socket says 172.21.15.38:39874) 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.569+0000 7fd6e6ffd700 1 -- 172.21.15.38:0/1798023414 learned_addr learned my addr 172.21.15.38:0/1798023414 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6e6ffd700 1 -- 172.21.15.38:0/1798023414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e8076420 con 0x7fd6e8101ed0 2024-04-21T23:53:03.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6e6ffd700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005850 tx=0x7fd6d0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0026070 con 0x7fd6e8101ed0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d000bc30 con 0x7fd6e8101ed0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e8074ad0 con 0x7fd6e8101ed0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0003a20 con 0x7fd6e8101ed0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.570+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e8074f90 con 0x7fd6e8101ed0 2024-04-21T23:53:03.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.571+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd6d0029070 con 0x7fd6e8101ed0 2024-04-21T23:53:03.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.571+0000 7fd6dffff700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 0x7fd6d403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:03.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.571+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd6d001e070 con 0x7fd6e8101ed0 2024-04-21T23:53:03.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.571+0000 7fd6e67fc700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 0x7fd6d403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:03.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.571+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e81033b0 con 0x7fd6e8101ed0 2024-04-21T23:53:03.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.572+0000 7fd6e67fc700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 0x7fd6d403b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:03.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.576+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd6d0030070 con 0x7fd6e8101ed0 2024-04-21T23:53:03.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:03.789+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fd6e8075220 con 0x7fd6e8101ed0 2024-04-21T23:53:04.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.214+0000 7fd6dffff700 1 -- 172.21.15.38:0/1798023414 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fd6d0013070 con 0x7fd6e8101ed0 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 msgr2=0x7fd6d403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 0x7fd6d403b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).stop 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 msgr2=0x7fd6e8074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005850 tx=0x7fd6d0005030 comp rx=0 tx=0).stop 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 shutdown_connections 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6d4039150 0x7fd6d403b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:04.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 --2- 172.21.15.38:0/1798023414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6e8101ed0 0x7fd6e8074530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:04.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.216+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 >> 172.21.15.38:0/1798023414 conn(0x7fd6e80fd760 msgr2=0x7fd6e8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:04.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.217+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 shutdown_connections 2024-04-21T23:53:04.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:04.217+0000 7fd6ed60c700 1 -- 172.21.15.38:0/1798023414 wait complete. 2024-04-21T23:53:04.211 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-04-21T23:53:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:04 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1798023414' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T23:53:04.900 INFO:tasks.cephadm:Adding mon.a on smithi038 2024-04-21T23:53:04.900 INFO:tasks.cephadm:Adding mon.b on smithi111 2024-04-21T23:53:04.900 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-04-21T23:53:04.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c' 2024-04-21T23:53:05.136 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1798023414' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:06.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 -- 172.21.15.203:0/87149532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 msgr2=0x7f91ac102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:06.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 --2- 172.21.15.203:0/87149532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac102280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009910 tx=0x7f91a0009c20 comp rx=0 tx=0).stop 2024-04-21T23:53:06.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 -- 172.21.15.203:0/87149532 shutdown_connections 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 --2- 172.21.15.203:0/87149532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac102280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 -- 172.21.15.203:0/87149532 >> 172.21.15.203:0/87149532 conn(0x7f91ac0fd750 msgr2=0x7f91ac0ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 -- 172.21.15.203:0/87149532 shutdown_connections 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 -- 172.21.15.203:0/87149532 wait complete. 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.693+0000 7f91b1148700 1 Processor -- start 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91b1148700 1 -- start start 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91b1148700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91b1148700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac10c270 con 0x7f91ac101ea0 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91aad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.203:44576/0 (socket says 172.21.15.203:44576) 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91aad9d700 1 -- 172.21.15.203:0/919079523 learned_addr learned my addr 172.21.15.203:0/919079523 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.694+0000 7f91aad9d700 1 -- 172.21.15.203:0/919079523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac19d920 con 0x7f91ac101ea0 2024-04-21T23:53:06.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f91aad9d700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f91a0005850 tx=0x7f91a0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91a0026070 con 0x7f91ac101ea0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ac19dc10 con 0x7f91ac101ea0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a000bbd0 con 0x7f91ac101ea0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91a0003970 con 0x7f91ac101ea0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.695+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ac1975a0 con 0x7f91ac101ea0 2024-04-21T23:53:06.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.696+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac103380 con 0x7f91ac101ea0 2024-04-21T23:53:06.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.696+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f91a0029070 con 0x7f91ac101ea0 2024-04-21T23:53:06.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.696+0000 7f9193fff700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 0x7f919403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:06.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.696+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f91a001e070 con 0x7f91ac101ea0 2024-04-21T23:53:06.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.696+0000 7f91aa59c700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 0x7f919403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:06.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.697+0000 7f91aa59c700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 0x7f919403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f919c006fd0 tx=0x7f919c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:06.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.702+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f91a0013070 con 0x7f91ac101ea0 2024-04-21T23:53:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-21T23:53:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:53:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:53:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:06.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:06.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:06 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:06.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.912+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7f91ac198ad0 con 0x7f91940390f0 2024-04-21T23:53:06.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.923+0000 7f9193fff700 1 -- 172.21.15.203:0/919079523 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f91ac198ad0 con 0x7f91940390f0 2024-04-21T23:53:06.916 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 msgr2=0x7f919403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 0x7f919403b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f919c006fd0 tx=0x7f919c006e40 comp rx=0 tx=0).stop 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 msgr2=0x7f91ac19d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f91a0005850 tx=0x7f91a0005030 comp rx=0 tx=0).stop 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 shutdown_connections 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f91940390f0 0x7f919403b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 --2- 172.21.15.203:0/919079523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91ac101ea0 0x7f91ac19d3e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 >> 172.21.15.203:0/919079523 conn(0x7f91ac0fd750 msgr2=0x7f91ac103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 shutdown_connections 2024-04-21T23:53:06.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:06.925+0000 7f91b1148700 1 -- 172.21.15.203:0/919079523 wait complete. 2024-04-21T23:53:07.519 DEBUG:teuthology.orchestra.run.smithi111:mon.b> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.b.service 2024-04-21T23:53:07.522 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.c.service 2024-04-21T23:53:07.524 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:53:07.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:53:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:08.527 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:08.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:08 smithi038 ceph-mon[27972]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:08.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:08 smithi038 ceph-mon[27972]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c;count:3 2024-04-21T23:53:08.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:08 smithi038 ceph-mon[27972]: Deploying daemon mon.c on smithi203 2024-04-21T23:53:09.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:09 smithi038 ceph-mon[27972]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:10.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 -- 172.21.15.203:0/483762275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 msgr2=0x7f76fc101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:10.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 --2- 172.21.15.203:0/483762275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc101180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76f8009910 tx=0x7f76f8009c20 comp rx=0 tx=0).stop 2024-04-21T23:53:10.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 -- 172.21.15.203:0/483762275 shutdown_connections 2024-04-21T23:53:10.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 --2- 172.21.15.203:0/483762275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc101180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 -- 172.21.15.203:0/483762275 >> 172.21.15.203:0/483762275 conn(0x7f76fc0fc630 msgr2=0x7f76fc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 -- 172.21.15.203:0/483762275 shutdown_connections 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.962+0000 7f770446d700 1 -- 172.21.15.203:0/483762275 wait complete. 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f770446d700 1 Processor -- start 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f770446d700 1 -- start start 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f770446d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f770446d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc10b170 con 0x7f76fc100da0 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f7702209700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f7702209700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.203:46304/0 (socket says 172.21.15.203:46304) 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.963+0000 7f7702209700 1 -- 172.21.15.203:0/190487721 learned_addr learned my addr 172.21.15.203:0/190487721 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:53:10.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f7702209700 1 -- 172.21.15.203:0/190487721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc195a70 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f7702209700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76f8005850 tx=0x7f76f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76f8026070 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76f800bbd0 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc195d00 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76f8003970 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.964+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc1ae230 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.965+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f76f8029070 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.965+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc102280 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.965+0000 7f76f37fe700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 0x7f76e803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.965+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f76f801e070 con 0x7f76fc100da0 2024-04-21T23:53:10.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.966+0000 7f7701a08700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 0x7f76e803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:10.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.966+0000 7f7701a08700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 0x7f76e803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:10.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:10.971+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f76f8013070 con 0x7f76fc100da0 2024-04-21T23:53:11.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.280+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f76fc063890 con 0x7f76fc100da0 2024-04-21T23:53:11.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.281+0000 7f76f37fe700 1 -- 172.21.15.203:0/190487721 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f76f8030070 con 0x7f76fc100da0 2024-04-21T23:53:11.275 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:53:11.275 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","modified":"2024-04-21T23:51:28.138627Z","created":"2024-04-21T23:51:28.138627Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-21T23:53:11.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 msgr2=0x7f76e803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:11.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 0x7f76e803b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006fd0 tx=0x7f76ec006e40 comp rx=0 tx=0).stop 2024-04-21T23:53:11.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 msgr2=0x7f76fc195530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:11.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76f8005850 tx=0x7f76f8005030 comp rx=0 tx=0).stop 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 shutdown_connections 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f76e80390f0 0x7f76e803b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 --2- 172.21.15.203:0/190487721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76fc100da0 0x7f76fc195530 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 >> 172.21.15.203:0/190487721 conn(0x7f76fc0fc630 msgr2=0x7f76fc06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 shutdown_connections 2024-04-21T23:53:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:11.283+0000 7f770446d700 1 -- 172.21.15.203:0/190487721 wait complete. 2024-04-21T23:53:11.279 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-21T23:53:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:11 smithi038 ceph-mon[27972]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:11.987 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:11 smithi203 systemd[1]: Started Ceph mon.c for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:53:13.016 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:53:13.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:53:13.263 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: mon.a calling monitor election 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: mon.c calling monitor election 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: monmap epoch 2 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: last_changed 2024-04-21T23:53:11.626385+0000 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: min_mon_release 19 (squid) 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: election_strategy: 1 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: fsmap 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: mgrmap e13: x(active, since 51s) 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: overall HEALTH_OK 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:16.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:17.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:17.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:53:17.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: mon.a calling monitor election 2024-04-21T23:53:17.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:17.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: mon.c calling monitor election 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: monmap epoch 2 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: last_changed 2024-04-21T23:53:11.626385+0000 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: min_mon_release 19 (squid) 2024-04-21T23:53:17.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: election_strategy: 1 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: fsmap 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: mgrmap e13: x(active, since 51s) 2024-04-21T23:53:17.035 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: overall HEALTH_OK 2024-04-21T23:53:17.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:17.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:17.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:17.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:17.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:17.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2499779541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3d00035c0 msgr2=0x7fb3d0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2499779541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3d00035c0 0x7fb3d0005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4100ba0 tx=0x7fb3dc009e80 comp rx=0 tx=0).stop 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2499779541 shutdown_connections 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2499779541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3d00035c0 0x7fb3d0005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2499779541 >> 172.21.15.203:0/2499779541 conn(0x7fb3e40fc780 msgr2=0x7fb3e40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2499779541 shutdown_connections 2024-04-21T23:53:17.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.649+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2499779541 wait complete. 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.650+0000 7fb3e96ad700 1 Processor -- start 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e96ad700 1 -- start start 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e96ad700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e96ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 0x7fb3e4103330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e96ad700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4108d00 con 0x7fb3e4102eb0 2024-04-21T23:53:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e96ad700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4108b80 con 0x7fb3d00035c0 2024-04-21T23:53:17.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:17.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:53048/0 (socket says 172.21.15.203:53048) 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 -- 172.21.15.203:0/2150548064 learned_addr learned my addr 172.21.15.203:0/2150548064 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 -- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 msgr2=0x7fb3e4102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 -- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 msgr2=0x7fb3e4102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e27fc700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 0x7fb3e4103330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.651+0000 7fb3e2ffd700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e27fc700 1 -- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 msgr2=0x7fb3e4102970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e27fc700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e27fc700 1 -- 172.21.15.203:0/2150548064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e4104890 con 0x7fb3e4102eb0 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e27fc700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 0x7fb3e4103330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00b4d0 tx=0x7fb3cc00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb3cc011630 con 0x7fb3e4102eb0 2024-04-21T23:53:17.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc011cb0 con 0x7fb3e4102eb0 2024-04-21T23:53:17.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e41a3b90 con 0x7fb3e4102eb0 2024-04-21T23:53:17.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb3cc010e60 con 0x7fb3e4102eb0 2024-04-21T23:53:17.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.652+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e41a40d0 con 0x7fb3e4102eb0 2024-04-21T23:53:17.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.653+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb3cc018020 con 0x7fb3e4102eb0 2024-04-21T23:53:17.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.653+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e41023b0 con 0x7fb3e4102eb0 2024-04-21T23:53:17.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.654+0000 7fb3dbfff700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 0x7fb3c403b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:17.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.654+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb3cc04bc40 con 0x7fb3e4102eb0 2024-04-21T23:53:17.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.654+0000 7fb3e2ffd700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 0x7fb3c403b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:17.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.654+0000 7fb3e2ffd700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 0x7fb3c403b2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc019000 tx=0x7fb3dc005040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:17.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.659+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb3cc01f020 con 0x7fb3e4102eb0 2024-04-21T23:53:17.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.940+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb3e402db30 con 0x7fb3e4102eb0 2024-04-21T23:53:17.933 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:17 smithi203 ceph-mon[31431]: Deploying daemon mon.b on smithi111 2024-04-21T23:53:17.933 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:17.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.941+0000 7fb3dbfff700 1 -- 172.21.15.203:0/2150548064 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fb3cc018310 con 0x7fb3e4102eb0 2024-04-21T23:53:17.934 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:53:17.934 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","modified":"2024-04-21T23:53:11.626385Z","created":"2024-04-21T23:51:28.138627Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T23:53:17.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 msgr2=0x7fb3c403b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:17.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 0x7fb3c403b2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc019000 tx=0x7fb3dc005040 comp rx=0 tx=0).stop 2024-04-21T23:53:17.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 msgr2=0x7fb3e4103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:17.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 0x7fb3e4103330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00b4d0 tx=0x7fb3cc00b890 comp rx=0 tx=0).stop 2024-04-21T23:53:17.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 shutdown_connections 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb3c4038e30 0x7fb3c403b2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d00035c0 0x7fb3e4102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 --2- 172.21.15.203:0/2150548064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3e4102eb0 0x7fb3e4103330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 >> 172.21.15.203:0/2150548064 conn(0x7fb3e40fc780 msgr2=0x7fb3e418ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 shutdown_connections 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:17.944+0000 7fb3e96ad700 1 -- 172.21.15.203:0/2150548064 wait complete. 2024-04-21T23:53:17.938 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-04-21T23:53:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:17 smithi038 ceph-mon[27972]: Deploying daemon mon.b on smithi111 2024-04-21T23:53:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:18 smithi203 ceph-mon[31431]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:18 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/2150548064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:53:18.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:18 smithi038 ceph-mon[27972]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:18.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/2150548064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:53:18.959 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:53:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:53:18.626+0000 7f1b32faf700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-21T23:53:19.507 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:53:19.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:53:19.767 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:53:20.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:20 smithi111 systemd[1]: Started Ceph mon.b for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: mon.a calling monitor election 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: mon.c calling monitor election 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: mon.b calling monitor election 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:25.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: monmap epoch 3 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: last_changed 2024-04-21T23:53:20.630338+0000 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: min_mon_release 19 (squid) 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: election_strategy: 1 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: fsmap 2024-04-21T23:53:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: mgrmap e13: x(active, since 60s) 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: overall HEALTH_OK 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:53:26.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: mon.a calling monitor election 2024-04-21T23:53:26.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: mon.c calling monitor election 2024-04-21T23:53:26.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: mon.b calling monitor election 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: monmap epoch 3 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: last_changed 2024-04-21T23:53:20.630338+0000 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:53:26.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: min_mon_release 19 (squid) 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: election_strategy: 1 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: fsmap 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: mgrmap e13: x(active, since 60s) 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: overall HEALTH_OK 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: mon.a calling monitor election 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: mon.c calling monitor election 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: mon.b calling monitor election 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: monmap epoch 3 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: last_changed 2024-04-21T23:53:20.630338+0000 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: created 2024-04-21T23:51:28.138627+0000 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: min_mon_release 19 (squid) 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: election_strategy: 1 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: 0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: 2: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: fsmap 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: mgrmap e13: x(active, since 60s) 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: overall HEALTH_OK 2024-04-21T23:53:26.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:26.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:26 smithi038 ceph-mon[27972]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:27.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:26 smithi203 ceph-mon[31431]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:27.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:27.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:26 smithi111 ceph-mon[31543]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.230+0000 7f322bfd6700 1 -- 172.21.15.203:0/2224933094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32100035c0 msgr2=0x7f3210005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.230+0000 7f322bfd6700 1 --2- 172.21.15.203:0/2224933094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32100035c0 0x7f3210005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3224079470 tx=0x7f3220010f60 comp rx=0 tx=0).stop 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 -- 172.21.15.203:0/2224933094 shutdown_connections 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 --2- 172.21.15.203:0/2224933094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32100035c0 0x7f3210005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 -- 172.21.15.203:0/2224933094 >> 172.21.15.203:0/2224933094 conn(0x7f32240772e0 msgr2=0x7f32240776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 -- 172.21.15.203:0/2224933094 shutdown_connections 2024-04-21T23:53:27.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 -- 172.21.15.203:0/2224933094 wait complete. 2024-04-21T23:53:27.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 Processor -- start 2024-04-21T23:53:27.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.231+0000 7f322bfd6700 1 -- start start 2024-04-21T23:53:27.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 0x7f3224102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:27.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 0x7f32241a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3224107be0 con 0x7f32241a3ba0 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3224107a60 con 0x7f32100035c0 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322bfd6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3224107d60 con 0x7f3224102c60 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:34310/0 (socket says 172.21.15.203:34310) 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 -- 172.21.15.203:0/139639906 learned_addr learned my addr 172.21.15.203:0/139639906 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322a573700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 0x7f32241a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229d72700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 0x7f3224102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 -- 172.21.15.203:0/139639906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 msgr2=0x7f3224102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 0x7f3224102720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 -- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 msgr2=0x7f32241a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:27.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 0x7f32241a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 -- 172.21.15.203:0/139639906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32241a66c0 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229d72700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 0x7f3224102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f322a573700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 0x7f32241a5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f3229571700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f321400b780 tx=0x7f321400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.232+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214004230 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.233+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32241a69b0 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.233+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32140043d0 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.233+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214011c90 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.233+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32241a6f40 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.233+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f321401a400 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.234+0000 7f321affd700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 0x7f320803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.234+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f321404cd60 con 0x7f3224102c60 2024-04-21T23:53:27.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.234+0000 7f3229d72700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 0x7f320803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:27.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.234+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3224063890 con 0x7f3224102c60 2024-04-21T23:53:27.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.234+0000 7f3229d72700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 0x7f320803ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f3220010fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:27.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.239+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3214020500 con 0x7f3224102c60 2024-04-21T23:53:27.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.543+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f322402db30 con 0x7f3224102c60 2024-04-21T23:53:27.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.544+0000 7f321affd700 1 -- 172.21.15.203:0/139639906 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f32140202a0 con 0x7f3224102c60 2024-04-21T23:53:27.538 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-21T23:53:27.538 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","modified":"2024-04-21T23:53:20.630338Z","created":"2024-04-21T23:51:28.138627Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-21T23:53:27.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 msgr2=0x7f320803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:27.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 0x7f320803ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3220000c00 tx=0x7f3220010fd0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 msgr2=0x7f32241030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f321400b780 tx=0x7f321400ba90 comp rx=0 tx=0).stop 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 shutdown_connections 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3208039580 0x7f320803ba40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32241a3ba0 0x7f32241a5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32100035c0 0x7f3224102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 --2- 172.21.15.203:0/139639906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3224102c60 0x7f32241030e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 >> 172.21.15.203:0/139639906 conn(0x7f32240772e0 msgr2=0x7f3224105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 shutdown_connections 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:27.546+0000 7f322bfd6700 1 -- 172.21.15.203:0/139639906 wait complete. 2024-04-21T23:53:27.541 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-04-21T23:53:28.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/139639906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.120 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:28.120 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:28.120 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:28.144 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T23:53:28.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T23:53:28.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/139639906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:53:28.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:28.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:28.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/139639906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:28.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:29.025 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:53:29.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:29 smithi038 ceph-mon[27972]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:29.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:29 smithi038 ceph-mon[27972]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:53:29.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:29 smithi038 ceph-mon[27972]: Reconfiguring daemon mon.a on smithi038 2024-04-21T23:53:29.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:29 smithi203 ceph-mon[31431]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:29.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:29 smithi203 ceph-mon[31431]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:53:29.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:29 smithi203 ceph-mon[31431]: Reconfiguring daemon mon.a on smithi038 2024-04-21T23:53:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:29 smithi111 ceph-mon[31543]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:29 smithi111 ceph-mon[31543]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:53:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:29 smithi111 ceph-mon[31543]: Reconfiguring daemon mon.a on smithi038 2024-04-21T23:53:30.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:30.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:30.548 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:30.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:30.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:30.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 -- 172.21.15.38:0/2738304616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164107590 msgr2=0x7fa16410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 --2- 172.21.15.38:0/2738304616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164107590 0x7fa16410f160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00b0a0 tx=0x7fa15c00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 -- 172.21.15.38:0/2738304616 shutdown_connections 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 --2- 172.21.15.38:0/2738304616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164107590 0x7fa16410f160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 --2- 172.21.15.38:0/2738304616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164102510 0x7fa1641028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 --2- 172.21.15.38:0/2738304616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102ec0 0x7fa164106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 -- 172.21.15.38:0/2738304616 >> 172.21.15.38:0/2738304616 conn(0x7fa1640793a0 msgr2=0x7fa1640797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:31.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 -- 172.21.15.38:0/2738304616 shutdown_connections 2024-04-21T23:53:31.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 -- 172.21.15.38:0/2738304616 wait complete. 2024-04-21T23:53:31.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.006+0000 7fa16954d700 1 Processor -- start 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 -- start start 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 0x7fa16419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 0x7fa164198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa164112570 con 0x7fa164102ec0 2024-04-21T23:53:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa1641123f0 con 0x7fa164107590 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa16954d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa1641126f0 con 0x7fa164102510 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38564/0 (socket says 172.21.15.38:38564) 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 -- 172.21.15.38:0/2902516320 learned_addr learned my addr 172.21.15.38:0/2902516320 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa162ffd700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 0x7fa16419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1637fe700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 0x7fa164198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 -- 172.21.15.38:0/2902516320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 msgr2=0x7fa16419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 0x7fa16419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 -- 172.21.15.38:0/2902516320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 msgr2=0x7fa164198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 0x7fa164198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1627fc700 1 -- 172.21.15.38:0/2902516320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa164199050 con 0x7fa164102ec0 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa162ffd700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 0x7fa16419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.007+0000 7fa1637fe700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 0x7fa164198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa1627fc700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa15800d950 tx=0x7fa15800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158017070 con 0x7fa164102ec0 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa164199340 con 0x7fa164102ec0 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15800de30 con 0x7fa164102ec0 2024-04-21T23:53:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580054c0 con 0x7fa164102ec0 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.008+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1641a5620 con 0x7fa164102ec0 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa158010460 con 0x7fa164102ec0 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa14bfff700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 0x7fa14c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa164103990 con 0x7fa164102ec0 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa15804bdd0 con 0x7fa164102ec0 2024-04-21T23:53:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa162ffd700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 0x7fa14c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.009+0000 7fa162ffd700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 0x7fa14c03b7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa154005fd0 tx=0x7fa154005e30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.014+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa158005660 con 0x7fa164102ec0 2024-04-21T23:53:31.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.221+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa164199ca0 con 0x7fa164102ec0 2024-04-21T23:53:31.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.222+0000 7fa14bfff700 1 -- 172.21.15.38:0/2902516320 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fa158010e00 con 0x7fa164102ec0 2024-04-21T23:53:31.215 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:31.216 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-04-21T23:53:31.216 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = eab6f840-0039-11ef-bc93-c7b262605968 2024-04-21T23:53:31.216 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 msgr2=0x7fa14c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 0x7fa14c03b7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa154005fd0 tx=0x7fa154005e30 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 msgr2=0x7fa16419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa15800d950 tx=0x7fa15800dd10 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 shutdown_connections 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa14c039330 0x7fa14c03b7f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa164107590 0x7fa164198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa164102510 0x7fa16419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 --2- 172.21.15.38:0/2902516320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa164102ec0 0x7fa16419ebe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 >> 172.21.15.38:0/2902516320 conn(0x7fa1640793a0 msgr2=0x7fa164100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 shutdown_connections 2024-04-21T23:53:31.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:31.224+0000 7fa16954d700 1 -- 172.21.15.38:0/2902516320 wait complete. 2024-04-21T23:53:31.528 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2902516320' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.529 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:31 smithi038 ceph-mon[27972]: Reconfiguring daemon mon.c on smithi203 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2902516320' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:31.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:31.769 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.769 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:31 smithi203 ceph-mon[31431]: Reconfiguring daemon mon.c on smithi203 2024-04-21T23:53:31.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:53:31.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: Reconfiguring daemon mon.b on smithi111 2024-04-21T23:53:31.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:31.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2902516320' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:31.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:31 smithi111 ceph-mon[31543]: Reconfiguring daemon mon.c on smithi203 2024-04-21T23:53:31.851 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T23:53:31.852 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:53:31.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:53:31.881 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:53:31.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:53:31.948 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:53:31.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:53:31.980 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:53:31.980 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:53:32.048 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:53:32.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:53:32.074 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:53:32.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:53:32.138 INFO:tasks.cephadm:Adding mgr.x on smithi038 2024-04-21T23:53:32.138 INFO:tasks.cephadm:Adding mgr.y on smithi111 2024-04-21T23:53:32.138 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-04-21T23:53:32.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi038=x;smithi111=y;smithi203=z' 2024-04-21T23:53:32.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:32 smithi038 ceph-mon[27972]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:32.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:32 smithi203 ceph-mon[31431]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:32.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:32 smithi111 ceph-mon[31543]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:32.835 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:34.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 -- 172.21.15.203:0/669993619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103540 msgr2=0x7f99581039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:34.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 --2- 172.21.15.203:0/669993619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103540 0x7f99581039a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9944009990 tx=0x7f9944009ca0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 -- 172.21.15.203:0/669993619 shutdown_connections 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 --2- 172.21.15.203:0/669993619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103ee0 0x7f995810a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 --2- 172.21.15.203:0/669993619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 0x7f9958100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 --2- 172.21.15.203:0/669993619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103540 0x7f99581039a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 -- 172.21.15.203:0/669993619 >> 172.21.15.203:0/669993619 conn(0x7f99580fbea0 msgr2=0x7f99580fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 -- 172.21.15.203:0/669993619 shutdown_connections 2024-04-21T23:53:34.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.325+0000 7f995eb81700 1 -- 172.21.15.203:0/669993619 wait complete. 2024-04-21T23:53:34.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 Processor -- start 2024-04-21T23:53:34.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 -- start start 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 0x7f9958198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103540 0x7f99581989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f995810de70 con 0x7f9958103540 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f995810dcf0 con 0x7f9958100560 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995eb81700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f995810dff0 con 0x7f9958103ee0 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51166/0 (socket says 172.21.15.203:51166) 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 -- 172.21.15.203:0/3430618054 learned_addr learned my addr 172.21.15.203:0/3430618054 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f9957fff700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103540 0x7f99581989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995c91d700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 0x7f9958198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 -- 172.21.15.203:0/3430618054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 msgr2=0x7f9958198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 0x7f9958198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 -- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103540 msgr2=0x7f99581989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103540 0x7f99581989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.326+0000 7f995d11e700 1 -- 172.21.15.203:0/3430618054 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99581a00f0 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f995d11e700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9950007f10 tx=0x7f995000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995000dd20 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9950004020 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950015660 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99581a03e0 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.327+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99581a0970 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.328+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9950015800 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.328+0000 7f9955ffb700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 0x7f994803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.328+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f995004be10 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.328+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9958068f20 con 0x7f9958103ee0 2024-04-21T23:53:34.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.328+0000 7f995c91d700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 0x7f994803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:34.325 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.329+0000 7f995c91d700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 0x7f994803b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f995806a7e0 tx=0x7f994c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:34.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.333+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9950010070 con 0x7f9958103ee0 2024-04-21T23:53:34.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.543+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7f9958062670 con 0x7f9948039330 2024-04-21T23:53:34.547 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.553+0000 7f9955ffb700 1 -- 172.21.15.203:0/3430618054 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9958062670 con 0x7f9948039330 2024-04-21T23:53:34.547 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-04-21T23:53:34.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 msgr2=0x7f994803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:34.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 0x7f994803b7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f995806a7e0 tx=0x7f994c009260 comp rx=0 tx=0).stop 2024-04-21T23:53:34.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 msgr2=0x7f995819fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9950007f10 tx=0x7f995000d3b0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 shutdown_connections 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9948039330 0x7f994803b7f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9958103ee0 0x7f995819fa50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9958100560 0x7f9958198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 --2- 172.21.15.203:0/3430618054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9958103540 0x7f99581989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.555+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 >> 172.21.15.203:0/3430618054 conn(0x7f99580fbea0 msgr2=0x7f9958106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.556+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 shutdown_connections 2024-04-21T23:53:34.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:53:34.556+0000 7f995eb81700 1 -- 172.21.15.203:0/3430618054 wait complete. 2024-04-21T23:53:35.142 DEBUG:teuthology.orchestra.run.smithi111:mgr.y> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.y.service 2024-04-21T23:53:35.146 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.z.service 2024-04-21T23:53:35.148 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T23:53:35.148 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:53:35.148 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:53:35.168 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:53:35.168 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 659 Links: 1 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-21 23:52:38.575936118 +0000 2024-04-21T23:53:35.228 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-21 23:47:13.075281680 +0000 2024-04-21T23:53:35.229 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-21 23:47:13.075281680 +0000 2024-04-21T23:53:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-21 23:47:13.075281680 +0000 2024-04-21T23:53:35.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:53:35.296 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-21T23:53:35.296 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-21T23:53:35.297 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000327075 s, 1.6 MB/s 2024-04-21T23:53:35.298 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:53:35.356 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:35.415 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-21 23:52:38.617935388 +0000 2024-04-21T23:53:35.416 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-21 23:47:13.420275433 +0000 2024-04-21T23:53:35.416 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-21 23:47:13.420275433 +0000 2024-04-21T23:53:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-21 23:47:13.420275433 +0000 2024-04-21T23:53:35.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:53:35.481 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-21T23:53:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-21T23:53:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000315155 s, 1.6 MB/s 2024-04-21T23:53:35.483 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:53:35.542 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-21T23:53:35.599 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-21 23:52:38.654934745 +0000 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-21 23:47:13.768269131 +0000 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-21 23:47:13.768269131 +0000 2024-04-21T23:53:35.600 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-21 23:47:13.768269131 +0000 2024-04-21T23:53:35.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:35.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.667 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-21T23:53:35.667 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-21T23:53:35.667 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000296076 s, 1.7 MB/s 2024-04-21T23:53:35.668 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:53:35.726 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:35.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:35.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:35.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-21 23:52:38.690934119 +0000 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-21 23:47:14.108262974 +0000 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-21 23:47:14.108262974 +0000 2024-04-21T23:53:35.784 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-21 23:47:14.108262974 +0000 2024-04-21T23:53:35.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:35.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:35.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:35.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:35.849 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-21T23:53:35.849 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-21T23:53:35.849 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00033335 s, 1.5 MB/s 2024-04-21T23:53:35.851 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:53:35.909 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-21T23:53:35.909 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:53:35.928 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:53:35.928 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-21T23:53:35.986 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:53:35.986 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:35.986 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T23:53:35.986 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:35.986 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:35.987 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:53:20.037896411 +0000 2024-04-21T23:53:35.987 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:47:40.479726008 +0000 2024-04-21T23:53:35.987 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:47:40.479726008 +0000 2024-04-21T23:53:35.987 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:47:40.479726008 +0000 2024-04-21T23:53:35.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:53:36.052 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:53:36.052 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:53:36.052 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000263521 s, 1.9 MB/s 2024-04-21T23:53:36.053 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:53:36.112 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-21T23:53:36.171 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:53:36.171 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:36.171 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-21T23:53:36.171 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:36.172 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:36.172 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:53:20.038896393 +0000 2024-04-21T23:53:36.172 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:47:40.802719928 +0000 2024-04-21T23:53:36.172 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:47:40.802719928 +0000 2024-04-21T23:53:36.172 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:47:40.802719928 +0000 2024-04-21T23:53:36.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:53:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:53:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:53:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000270194 s, 1.9 MB/s 2024-04-21T23:53:36.239 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:53:36.297 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:53:20.038896393 +0000 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:47:41.158713226 +0000 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:47:41.158713226 +0000 2024-04-21T23:53:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:47:41.158713226 +0000 2024-04-21T23:53:36.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:53:36.422 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:53:36.422 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:53:36.422 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000223024 s, 2.3 MB/s 2024-04-21T23:53:36.423 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:53:36.481 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-21T23:53:36.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:36 smithi203 ceph-mon[31431]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:36.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:36 smithi203 ceph-mon[31431]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-21T23:53:36.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:36 smithi203 ceph-mon[31431]: Deploying daemon mgr.z on smithi203 2024-04-21T23:53:36.537 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:53:36.537 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:36.537 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T23:53:36.537 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:36.538 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:36.538 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-21 23:53:20.039896375 +0000 2024-04-21T23:53:36.538 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-21 23:47:41.493706919 +0000 2024-04-21T23:53:36.538 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-21 23:47:41.493706919 +0000 2024-04-21T23:53:36.538 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-21 23:47:41.493706919 +0000 2024-04-21T23:53:36.538 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:53:36.603 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-21T23:53:36.603 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-21T23:53:36.603 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000306096 s, 1.7 MB/s 2024-04-21T23:53:36.604 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:53:36.664 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-21T23:53:36.664 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:53:36.682 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:53:36.683 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-21T23:53:36.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:36 smithi038 ceph-mon[27972]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:36.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:36 smithi038 ceph-mon[27972]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-21T23:53:36.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:36 smithi038 ceph-mon[27972]: Deploying daemon mgr.z on smithi203 2024-04-21T23:53:36.739 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:53:36.739 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:36.739 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T23:53:36.739 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:36.739 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:36.740 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-21 23:53:36.578691405 +0000 2024-04-21T23:53:36.740 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-21 23:47:29.178528416 +0000 2024-04-21T23:53:36.740 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-21 23:47:29.178528416 +0000 2024-04-21T23:53:36.740 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-21 23:47:29.178528416 +0000 2024-04-21T23:53:36.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:53:36.805 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-21T23:53:36.805 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-21T23:53:36.805 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000282611 s, 1.8 MB/s 2024-04-21T23:53:36.806 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:53:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:36 smithi111 ceph-mon[31543]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi111=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:36 smithi111 ceph-mon[31543]: Saving service mgr spec with placement smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-21T23:53:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:36 smithi111 ceph-mon[31543]: Deploying daemon mgr.z on smithi203 2024-04-21T23:53:36.863 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-21T23:53:36.918 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:53:36.918 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-21 23:53:36.578691405 +0000 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-21 23:47:29.530520062 +0000 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-21 23:47:29.530520062 +0000 2024-04-21T23:53:36.919 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-21 23:47:29.530520062 +0000 2024-04-21T23:53:36.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:53:36.979 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:36 smithi203 podman[33101]: 2024-04-21 23:53:36.896518694 +0000 UTC m=+0.138448498 container create a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T23:53:36.982 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-21T23:53:36.982 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-21T23:53:36.982 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000189835 s, 2.7 MB/s 2024-04-21T23:53:36.983 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:53:37.040 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-21T23:53:37.095 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:53:37.095 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-21 23:53:36.579691382 +0000 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-21 23:47:29.887511590 +0000 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-21 23:47:29.887511590 +0000 2024-04-21T23:53:37.096 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-21 23:47:29.887511590 +0000 2024-04-21T23:53:37.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:53:37.158 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-21T23:53:37.158 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-21T23:53:37.158 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000182029 s, 2.8 MB/s 2024-04-21T23:53:37.159 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:53:37.214 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-21T23:53:37.234 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 podman[33101]: 2024-04-21 23:53:37.174968191 +0000 UTC m=+0.416897996 container init a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-21T23:53:37.234 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 podman[33101]: 2024-04-21 23:53:37.177726922 +0000 UTC m=+0.419656733 container start a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:53:37.234 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mgr[33253]: -- 172.21.15.203:0/3851680821 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591298665a0 con 0x55912983b000 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-21 23:53:36.579691382 +0000 2024-04-21T23:53:37.251 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-21 23:47:30.225503568 +0000 2024-04-21T23:53:37.252 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-21 23:47:30.225503568 +0000 2024-04-21T23:53:37.252 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-21 23:47:30.225503568 +0000 2024-04-21T23:53:37.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:53:37.315 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-21T23:53:37.315 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-21T23:53:37.315 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000236027 s, 2.2 MB/s 2024-04-21T23:53:37.316 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:53:37.372 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-21T23:53:37.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:53:37.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 bash[33101]: a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 2024-04-21T23:53:37.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 systemd[1]: Started Ceph mgr.z for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:53:37.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:37.352+0000 7fd4128ee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:37.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:37.614 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:53:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:37.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:37 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:37.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:37 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:37.947 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:37.542+0000 7fd4128ee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:53:37.948 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:37.647+0000 7fd4128ee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:53:37.948 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:37 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:37.953+0000 7fd4128ee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:53:38.280 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:38 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:38.038+0000 7fd4128ee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:53:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:38 smithi038 ceph-mon[27972]: Deploying daemon mgr.y on smithi111 2024-04-21T23:53:38.776 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:38 smithi111 ceph-mon[31543]: Deploying daemon mgr.y on smithi111 2024-04-21T23:53:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:38 smithi203 ceph-mon[31431]: Deploying daemon mgr.y on smithi111 2024-04-21T23:53:39.446 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 podman[32951]: 2024-04-21 23:53:39.36891101 +0000 UTC m=+0.130341466 container create 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-21T23:53:39.506 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:53:39.507 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:53:39.507 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-21T23:53:39.507 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-21T23:53:39.507 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108661 s, 965 MB/s 2024-04-21T23:53:39.507 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-21T23:53:39.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-mon[31431]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:39.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:39.224+0000 7fd4128ee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:53:39.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:39.392+0000 7fd4128ee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:53:39.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:39 smithi038 ceph-mon[27972]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:39.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:39 smithi111 ceph-mon[31543]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:39.718 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 podman[32951]: 2024-04-21 23:53:39.571254907 +0000 UTC m=+0.332685364 container init 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2) 2024-04-21T23:53:39.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 podman[32951]: 2024-04-21 23:53:39.57387693 +0000 UTC m=+0.335307392 container start 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T23:53:39.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 ceph-mgr[32970]: -- 172.21.15.111:0/992252090 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556dbca805a0 con 0x556dbca54c00 2024-04-21T23:53:39.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 bash[32951]: 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 2024-04-21T23:53:39.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 systemd[1]: Started Ceph mgr.y for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:53:39.841 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:39.562+0000 7fd4128ee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:53:39.841 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:39.847+0000 7fd4128ee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:53:40.012 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:39.725+0000 7fc2abc1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:53:40.012 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:39 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:39.912+0000 7fc2abc1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:53:40.012 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:40.018+0000 7fc2abc1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:53:40.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-21T23:53:40.148 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:39 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:39.940+0000 7fd4128ee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:53:40.148 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:40.154+0000 7fd4128ee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:53:40.368 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:53:40.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:40.352+0000 7fd4128ee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:53:40.707 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:40.341+0000 7fc2abc1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:53:40.707 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:40.433+0000 7fc2abc1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:40.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:40.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:41.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:41.280 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:41.015+0000 7fd4128ee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:53:41.758 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:41.467+0000 7fd4128ee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:53:41.758 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:41.563+0000 7fd4128ee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:53:41.758 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:41.762+0000 7fd4128ee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:53:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 -- 172.21.15.38:0/231177431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 msgr2=0x7f25e8103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 --2- 172.21.15.38:0/231177431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e8103910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0099c0 tx=0x7f25dc009cd0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 -- 172.21.15.38:0/231177431 shutdown_connections 2024-04-21T23:53:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 --2- 172.21.15.38:0/231177431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 0x7f25e810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 --2- 172.21.15.38:0/231177431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 0x7f25e81026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 --2- 172.21.15.38:0/231177431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e8103910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 -- 172.21.15.38:0/231177431 >> 172.21.15.38:0/231177431 conn(0x7f25e80fd850 msgr2=0x7f25e80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 -- 172.21.15.38:0/231177431 shutdown_connections 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.892+0000 7f25f05b8700 1 -- 172.21.15.38:0/231177431 wait complete. 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 Processor -- start 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 -- start start 2024-04-21T23:53:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 0x7f25e8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 0x7f25e819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25e810de80 con 0x7f25e81034b0 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25e810dd00 con 0x7f25e81022b0 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25f05b8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25e810e000 con 0x7f25e8103e50 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25edb53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25edb53700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39470/0 (socket says 172.21.15.38:39470) 2024-04-21T23:53:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25edb53700 1 -- 172.21.15.38:0/3605457209 learned_addr learned my addr 172.21.15.38:0/3605457209 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25ee354700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 0x7f25e8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.893+0000 7f25eeb55700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 0x7f25e819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 -- 172.21.15.38:0/3605457209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 msgr2=0x7f25e819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 0x7f25e819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 -- 172.21.15.38:0/3605457209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 msgr2=0x7f25e8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 0x7f25e8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 -- 172.21.15.38:0/3605457209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e81a00c0 con 0x7f25e81034b0 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25edb53700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25dc000c00 tx=0x7f25dc00bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc009e60 con 0x7f25e81034b0 2024-04-21T23:53:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25dc00f460 con 0x7f25e81034b0 2024-04-21T23:53:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e81a0350 con 0x7f25e81034b0 2024-04-21T23:53:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc017650 con 0x7f25e81034b0 2024-04-21T23:53:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.894+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e81a0800 con 0x7f25e81034b0 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.895+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f25dc0178b0 con 0x7f25e81034b0 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.895+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e8068f20 con 0x7f25e81034b0 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.895+0000 7f25db7fe700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 0x7f25d403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.895+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f25dc04d340 con 0x7f25e81034b0 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.896+0000 7f25ee354700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 0x7f25d403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:53:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.896+0000 7f25ee354700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 0x7f25d403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25e8103310 tx=0x7f25e4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:53:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:41.901+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f25dc013070 con 0x7f25e81034b0 2024-04-21T23:53:41.990 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:41 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:41.659+0000 7fc2abc1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:53:41.990 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:41 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:41.827+0000 7fc2abc1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:53:41.990 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:41 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:41.996+0000 7fc2abc1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:53:42.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:42.134+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f25e8062670 con 0x7f25d4039330 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.274 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:53:42.275 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:53:42.275 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:53:42.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.281+0000 7fd4128ee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:53:42.530 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.371+0000 7fd4128ee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:53:42.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:53:42.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:42.565 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:42.280+0000 7fc2abc1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:53:42.565 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:42.371+0000 7fc2abc1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:53:42.565 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:42.572+0000 7fc2abc1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:53:42.827 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:42 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:42.762+0000 7fc2abc1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:53:42.941 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.608+0000 7fd4128ee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:53:42.941 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.741+0000 7fd4128ee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:53:42.941 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.834+0000 7fd4128ee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:53:42.941 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:42 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:42.947+0000 7fd4128ee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:53:43.261 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:43.037+0000 7fd4128ee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: Reconfiguring daemon mgr.x on smithi038 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:43.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:43.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:43.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:53:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: Reconfiguring daemon mgr.x on smithi038 2024-04-21T23:53:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:43.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:43.361+0000 7fc2abc1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: Reconfiguring daemon mgr.x on smithi038 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:43.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:43.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:53:43.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:44.061 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:43.776+0000 7fc2abc1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:53:44.061 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:43 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:43.871+0000 7fc2abc1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:53:44.267 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:43 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:43.841+0000 7fd4128ee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:53:44.268 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-21T23:53:44.049+0000 7fd4128ee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:53:44.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:44.278+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f25dc02b960 con 0x7f25e81034b0 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:44.068+0000 7fc2abc1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-mon[31543]: Standby manager daemon z started 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:53:44.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:44.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:44 smithi038 ceph-mon[27972]: Standby manager daemon z started 2024-04-21T23:53:44.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:44 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:53:44.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:44 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:44.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:44 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:53:44.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:44 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-mon[31431]: Standby manager daemon z started 2024-04-21T23:53:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:53:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:53:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:44 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/1453569918' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:44.907 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:44.584+0000 7fc2abc1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:53:44.907 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:44.675+0000 7fc2abc1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:53:45.244 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:44 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:44.913+0000 7fc2abc1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:53:45.244 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:45.045+0000 7fc2abc1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:53:45.244 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:45.138+0000 7fc2abc1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:53:45.244 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:45.250+0000 7fc2abc1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:53:45.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:45 smithi203 ceph-mon[31431]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:45.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:45 smithi203 ceph-mon[31431]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T23:53:45.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:53:45.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-mon[31543]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:45.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-mon[31543]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T23:53:45.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:53:45.577 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:45 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:45.341+0000 7fc2abc1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:53:45.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:45 smithi038 ceph-mon[27972]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:45 smithi038 ceph-mon[27972]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-21T23:53:45.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:53:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]: dispatch 2024-04-21T23:53:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]': finished 2024-04-21T23:53:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:46 smithi203 ceph-mon[31431]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:53:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:46 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:53:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1295723194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:46.141+0000 7fc2abc1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-21T23:53:46.352+0000 7fc2abc1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]: dispatch 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]': finished 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-mon[31543]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:53:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1295723194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:53:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]: dispatch 2024-04-21T23:53:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3506309923' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41c96fae-afbf-4345-b838-bfdab30012d1"}]': finished 2024-04-21T23:53:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:46 smithi038 ceph-mon[27972]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:53:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:46 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:53:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1295723194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:53:47.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:53:47.289+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f25dc0163a0 con 0x7f25e81034b0 2024-04-21T23:53:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: Standby manager daemon y started 2024-04-21T23:53:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:53:47.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:47.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:53:47.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:47 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: Standby manager daemon y started 2024-04-21T23:53:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:53:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:47.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:53:47.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:47 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: Standby manager daemon y started 2024-04-21T23:53:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:53:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:53:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:53:47.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:47 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/2926369976' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:53:48.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:48 smithi111 ceph-mon[31543]: mgrmap e15: x(active, since 81s), standbys: y, z 2024-04-21T23:53:48.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:53:48.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:48 smithi038 ceph-mon[27972]: mgrmap e15: x(active, since 81s), standbys: y, z 2024-04-21T23:53:48.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:53:48.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:48 smithi203 ceph-mon[31431]: mgrmap e15: x(active, since 81s), standbys: y, z 2024-04-21T23:53:48.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:53:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:49 smithi111 ceph-mon[31543]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:49.415 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:49 smithi038 ceph-mon[27972]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:49 smithi203 ceph-mon[31431]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:51.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:51 smithi111 ceph-mon[31543]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:51 smithi038 ceph-mon[27972]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:51 smithi203 ceph-mon[31431]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:53.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:53 smithi038 ceph-mon[27972]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:53.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:53 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:53:53.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:53 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:53.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:53 smithi111 ceph-mon[31543]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:53.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:53 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:53:53.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:53 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:53 smithi203 ceph-mon[31431]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:53 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:53:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:53 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:53:54.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:54 smithi038 ceph-mon[27972]: Deploying daemon osd.0 on smithi038 2024-04-21T23:53:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:54 smithi111 ceph-mon[31543]: Deploying daemon osd.0 on smithi038 2024-04-21T23:53:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:54 smithi203 ceph-mon[31431]: Deploying daemon osd.0 on smithi038 2024-04-21T23:53:55.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:55 smithi038 ceph-mon[27972]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:55 smithi111 ceph-mon[31543]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:55 smithi203 ceph-mon[31431]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:57.570 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:57 smithi038 ceph-mon[27972]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:57.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:57 smithi111 ceph-mon[31543]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:57.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:57 smithi203 ceph-mon[31431]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:59.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:59 smithi111 ceph-mon[31543]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:59.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:53:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:59.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:59 smithi038 ceph-mon[27972]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:59.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:53:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:53:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:59 smithi203 ceph-mon[31431]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:53:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:53:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:53:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:01.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:01.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:01.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:01.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:01.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:01.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.941+0000 7f25db7fe700 1 -- 172.21.15.38:0/3605457209 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f25e8062670 con 0x7f25d4039330 2024-04-21T23:54:01.936 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 msgr2=0x7f25d403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 0x7f25d403b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25e8103310 tx=0x7f25e4009260 comp rx=0 tx=0).stop 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 msgr2=0x7f25e81989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25dc000c00 tx=0x7f25dc00bee0 comp rx=0 tx=0).stop 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 shutdown_connections 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25d4039330 0x7f25d403b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25e8103e50 0x7f25e819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:01.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25e81022b0 0x7f25e8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:01.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 --2- 172.21.15.38:0/3605457209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e81034b0 0x7f25e81989a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:01.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 >> 172.21.15.38:0/3605457209 conn(0x7f25e80fd850 msgr2=0x7f25e81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:01.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 shutdown_connections 2024-04-21T23:54:01.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:01.944+0000 7f25f05b8700 1 -- 172.21.15.38:0/3605457209 wait complete. 2024-04-21T23:54:02.520 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.0.service 2024-04-21T23:54:02.523 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_4... 2024-04-21T23:54:02.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:54:02.764 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:54:03.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:02 smithi111 ceph-mon[31543]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:03.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:02 smithi111 ceph-mon[31543]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:54:03.176 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:02 smithi038 ceph-mon[27972]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:02 smithi038 ceph-mon[27972]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:54:03.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:03.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:02 smithi203 ceph-mon[31431]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:03.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:03.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:02 smithi203 ceph-mon[31431]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:54:03.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:03 smithi038 ceph-mon[27972]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:54:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:03 smithi038 ceph-mon[27972]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:54:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:03 smithi038 ceph-mon[27972]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:54:03.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:04.051 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:03 smithi111 ceph-mon[31543]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:54:04.052 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:03 smithi111 ceph-mon[31543]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:54:04.052 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:03 smithi111 ceph-mon[31543]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:54:04.052 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:04.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:03 smithi203 ceph-mon[31431]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:54:04.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:03 smithi203 ceph-mon[31431]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:54:04.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:03 smithi203 ceph-mon[31431]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:54:04.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:04.611 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:54:04.612 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:54:04.612 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:54:04.612 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:54:04.612 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111826 s, 938 MB/s 2024-04-21T23:54:04.612 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:54:04.931 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0[42821]: 2024-04-21T23:54:04.667+0000 7f4007565700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T23:54:04.932 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0[42821]: 2024-04-21T23:54:04.672+0000 7f3fff350700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:54:05.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:05.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:54:05.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:54:05.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.206 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.206 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:04 smithi111 ceph-mon[31543]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' 2024-04-21T23:54:05.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:05.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:54:05.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:54:05.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:04 smithi038 ceph-mon[27972]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' 2024-04-21T23:54:05.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:05.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:04 smithi203 ceph-mon[31431]: from='osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841]' entity='osd.0' 2024-04-21T23:54:05.479 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:54:06.203 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] boot 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:54:06.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:06.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:54:06.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:54:06.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] boot 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:54:06.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:06.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:54:06.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:54:06.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:06.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: osd.0 [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] boot 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:54:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 -- 172.21.15.111:0/3052287150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e81011b0 msgr2=0x7f94e81015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 --2- 172.21.15.111:0/3052287150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e81011b0 0x7f94e81015d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f94e40099c0 tx=0x7f94e4009cd0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 -- 172.21.15.111:0/3052287150 shutdown_connections 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 --2- 172.21.15.111:0/3052287150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e8102d50 0x7f94e8109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 --2- 172.21.15.111:0/3052287150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e81011b0 0x7f94e81015d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 --2- 172.21.15.111:0/3052287150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81023b0 0x7f94e8102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 -- 172.21.15.111:0/3052287150 >> 172.21.15.111:0/3052287150 conn(0x7f94e80fc790 msgr2=0x7f94e80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 -- 172.21.15.111:0/3052287150 shutdown_connections 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.009+0000 7f94efd77700 1 -- 172.21.15.111:0/3052287150 wait complete. 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 Processor -- start 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 -- start start 2024-04-21T23:54:07.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 0x7f94e806b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e81023b0 0x7f94e819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810ce00 con 0x7f94e81023b0 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810cc80 con 0x7f94e8102d50 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94efd77700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810cf80 con 0x7f94e81011b0 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94ee314700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94ee314700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42430/0 (socket says 172.21.15.111:42430) 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94ee314700 1 -- 172.21.15.111:0/2920533067 learned_addr learned my addr 172.21.15.111:0/2920533067 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94ed312700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e81023b0 0x7f94e819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.010+0000 7f94edb13700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 0x7f94e806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 -- 172.21.15.111:0/2920533067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 msgr2=0x7f94e806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 0x7f94e806b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 -- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e81023b0 msgr2=0x7f94e819d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e81023b0 0x7f94e819d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 -- 172.21.15.111:0/2920533067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e81a01a0 con 0x7f94e8102d50 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94edb13700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 0x7f94e806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94ee314700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94e000ada0 tx=0x7f94e000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0017020 con 0x7f94e8102d50 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e000a0d0 con 0x7f94e8102d50 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e81a0490 con 0x7f94e8102d50 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0005480 con 0x7f94e8102d50 2024-04-21T23:54:07.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.011+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e81a0a50 con 0x7f94e8102d50 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.012+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e8068f20 con 0x7f94e8102d50 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.013+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f94e0005620 con 0x7f94e8102d50 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.014+0000 7f94deffd700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 0x7f94d40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.014+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f94e00ccfd0 con 0x7f94e8102d50 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.014+0000 7f94edb13700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 0x7f94d40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:07.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.015+0000 7f94edb13700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 0x7f94d40a50d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f94e400b580 tx=0x7f94e4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:07.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.018+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f94e0021060 con 0x7f94e8102d50 2024-04-21T23:54:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:06 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi038 2024-04-21T23:54:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:06 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-21T23:54:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:06 smithi111 ceph-mon[31543]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:06 smithi111 ceph-mon[31543]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:54:07.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:06 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi038 2024-04-21T23:54:07.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:06 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-21T23:54:07.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:06 smithi038 ceph-mon[27972]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:07.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:06 smithi038 ceph-mon[27972]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:54:07.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:07.228+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f94e8062730 con 0x7f94d40a2c10 2024-04-21T23:54:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:06 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi038 2024-04-21T23:54:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:06 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-21T23:54:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:06 smithi203 ceph-mon[31431]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:54:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:06 smithi203 ceph-mon[31431]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:54:08.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:07 smithi111 ceph-mon[31543]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:08.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:08.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:08.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:08.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:07 smithi038 ceph-mon[27972]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:08.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:08.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:08.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:08.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:07 smithi203 ceph-mon[31431]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:08.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:08.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:08.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:09.036 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:08 smithi111 ceph-mon[31543]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:09.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:08 smithi038 ceph-mon[27972]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:09.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:08 smithi203 ceph-mon[31431]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:10.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/618452166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:09 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:09 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]': finished 2024-04-21T23:54:10.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:09 smithi038 ceph-mon[27972]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:54:10.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/618452166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:09 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:09 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]': finished 2024-04-21T23:54:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:09 smithi203 ceph-mon[31431]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:54:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:10.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/618452166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:09 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]: dispatch 2024-04-21T23:54:10.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:09 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87fe4b77-db92-4f21-b74f-76c1aec76f38"}]': finished 2024-04-21T23:54:10.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:09 smithi111 ceph-mon[31543]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:54:10.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:11.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:10 smithi038 ceph-mon[27972]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:11.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:10 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/4075929237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:11.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:10 smithi203 ceph-mon[31431]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:11.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:10 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/4075929237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:10 smithi111 ceph-mon[31543]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:10 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/4075929237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:13.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:12 smithi038 ceph-mon[27972]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:13.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:12 smithi203 ceph-mon[31431]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:13.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:12 smithi111 ceph-mon[31543]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:15.009 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:14 smithi111 ceph-mon[31543]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:15.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:14 smithi203 ceph-mon[31431]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:14 smithi038 ceph-mon[27972]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:16 smithi111 ceph-mon[31543]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:16 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:54:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:16 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:17.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:16 smithi203 ceph-mon[31431]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:17.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:54:17.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:16 smithi038 ceph-mon[27972]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:54:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:18.206 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:17 smithi111 ceph-mon[31543]: Deploying daemon osd.1 on smithi111 2024-04-21T23:54:18.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:17 smithi203 ceph-mon[31431]: Deploying daemon osd.1 on smithi111 2024-04-21T23:54:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:17 smithi038 ceph-mon[27972]: Deploying daemon osd.1 on smithi111 2024-04-21T23:54:19.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:18 smithi111 ceph-mon[31543]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:19.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:18 smithi203 ceph-mon[31431]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:19.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:18 smithi038 ceph-mon[27972]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:21.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:20 smithi203 ceph-mon[31431]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:21.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:20 smithi111 ceph-mon[31543]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:21.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:20 smithi038 ceph-mon[27972]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:23 smithi111 ceph-mon[31543]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:23 smithi038 ceph-mon[27972]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:23 smithi203 ceph-mon[31431]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:23.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:25.555 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:25 smithi111 ceph-mon[31543]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.646+0000 7f94deffd700 1 -- 172.21.15.111:0/2920533067 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f94e8062730 con 0x7f94d40a2c10 2024-04-21T23:54:25.639 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 msgr2=0x7f94d40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 0x7f94d40a50d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f94e400b580 tx=0x7f94e4005fb0 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 msgr2=0x7f94e819f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94e000ada0 tx=0x7f94e000c330 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 shutdown_connections 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f94d40a2c10 0x7f94d40a50d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94e8102d50 0x7f94e819f8d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94e81011b0 0x7f94e806b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 --2- 172.21.15.111:0/2920533067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e81023b0 0x7f94e819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 >> 172.21.15.111:0/2920533067 conn(0x7f94e80fc790 msgr2=0x7f94e8105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 shutdown_connections 2024-04-21T23:54:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:54:25.648+0000 7f94efd77700 1 -- 172.21.15.111:0/2920533067 wait complete. 2024-04-21T23:54:25.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:25 smithi038 ceph-mon[27972]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:25 smithi203 ceph-mon[31431]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:26.244 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.1.service 2024-04-21T23:54:26.247 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-04-21T23:54:26.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:54:26.490 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:54:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:26.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:26 smithi111 ceph-mon[31543]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:26.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:26 smithi203 ceph-mon[31431]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:26.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:26 smithi038 ceph-mon[27972]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:54:27.819 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:27 smithi203 ceph-mon[31431]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:54:27.819 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:27 smithi203 ceph-mon[31431]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:54:27.819 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:27.819 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:27 smithi203 ceph-mon[31431]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:27.819 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:27 smithi203 ceph-mon[31431]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-mon[31543]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:54:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-mon[31543]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:54:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-mon[31543]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-mon[31543]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:28.078 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1[36145]: 2024-04-21T23:54:27.733+0000 7fdd07e1c700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T23:54:28.078 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1[36145]: 2024-04-21T23:54:27.736+0000 7fdd03c0f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:54:28.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:27 smithi038 ceph-mon[27972]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:54:28.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:27 smithi038 ceph-mon[27972]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:54:28.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:28.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:27 smithi038 ceph-mon[27972]: from='osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:28.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:27 smithi038 ceph-mon[27972]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:54:28.400 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:54:28.400 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:54:28.400 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-21T23:54:28.400 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-21T23:54:28.401 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131596 s, 797 MB/s 2024-04-21T23:54:28.401 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.016 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-04-21T23:54:29.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:29.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:54:29.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:54:29.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.281 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:54:29.997 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] boot 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi111 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:29.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:29.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:29.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:29.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:29.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:29.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:54:30.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:54:30.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] boot 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi111 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:30.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:30.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:30.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:54:30.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:54:30.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: osd.1 [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] boot 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi111 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 17088M 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:30.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:30.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:30.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 -- 172.21.15.203:0/340558818 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c41022c0 msgr2=0x7f92c4102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 --2- 172.21.15.203:0/340558818 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c41022c0 0x7f92c4102720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009990 tx=0x7f92b4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 -- 172.21.15.203:0/340558818 shutdown_connections 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 --2- 172.21.15.203:0/340558818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c4102c60 0x7f92c4109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 --2- 172.21.15.203:0/340558818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 0x7f92c41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 --2- 172.21.15.203:0/340558818 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c41022c0 0x7f92c4102720 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 -- 172.21.15.203:0/340558818 >> 172.21.15.203:0/340558818 conn(0x7f92c40fc640 msgr2=0x7f92c40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 -- 172.21.15.203:0/340558818 shutdown_connections 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.797+0000 7f92c359e700 1 -- 172.21.15.203:0/340558818 wait complete. 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 Processor -- start 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 -- start start 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 0x7f92c406aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:30.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 0x7f92c419d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410cdf0 con 0x7f92c41010c0 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410cc70 con 0x7f92c41022c0 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c359e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410cf70 con 0x7f92c4102c60 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:38620/0 (socket says 172.21.15.203:38620) 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c2d9d700 1 -- 172.21.15.203:0/1538112930 learned_addr learned my addr 172.21.15.203:0/1538112930 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c259c700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 0x7f92c406aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c1d9b700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 0x7f92c419d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.798+0000 7f92c2d9d700 1 -- 172.21.15.203:0/1538112930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 msgr2=0x7f92c419d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c2d9d700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 0x7f92c419d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c2d9d700 1 -- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 msgr2=0x7f92c406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:30.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c2d9d700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 0x7f92c406aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c2d9d700 1 -- 172.21.15.203:0/1538112930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c41a01a0 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c1d9b700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 0x7f92c419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c2d9d700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00b970 tx=0x7f92bc00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc019070 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92bc00ce10 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c41a0490 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc007370 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.799+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c41a0aa0 con 0x7f92c4102c60 2024-04-21T23:54:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.800+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c4068f20 con 0x7f92c4102c60 2024-04-21T23:54:30.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.801+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f92bc01c070 con 0x7f92c4102c60 2024-04-21T23:54:30.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.802+0000 7f92b37fe700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 0x7f92ac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:30.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.802+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f92bc0ccf10 con 0x7f92c4102c60 2024-04-21T23:54:30.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.802+0000 7f92c259c700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 0x7f92ac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:30.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.803+0000 7f92c259c700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 0x7f92ac0a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f92c4102120 tx=0x7f92b8009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:30.796 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:30 smithi203 ceph-mon[31431]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:30.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:30 smithi203 ceph-mon[31431]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:54:30.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:30.806+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f92bc09b6c0 con 0x7f92c4102c60 2024-04-21T23:54:30.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:31.007+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f92c4062670 con 0x7f92ac0a2c10 2024-04-21T23:54:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:30 smithi111 ceph-mon[31543]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:30 smithi111 ceph-mon[31543]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:54:31.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:30 smithi038 ceph-mon[27972]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:54:31.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:30 smithi038 ceph-mon[27972]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:54:31.871 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:31 smithi203 ceph-mon[31431]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:31.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:31.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:31.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:31 smithi111 ceph-mon[31543]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:31 smithi038 ceph-mon[27972]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:33.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:32 smithi203 ceph-mon[31431]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:33.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:32 smithi111 ceph-mon[31543]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:32 smithi038 ceph-mon[27972]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:33 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/170181851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:33 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:33 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]': finished 2024-04-21T23:54:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:33 smithi203 ceph-mon[31431]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:54:34.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:33 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/170181851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:33 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:33 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]': finished 2024-04-21T23:54:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:33 smithi111 ceph-mon[31543]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:54:34.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:34.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:33 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/170181851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:33 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]: dispatch 2024-04-21T23:54:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:33 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2e0ed53-ee4a-432b-9147-1e4df2da6dd5"}]': finished 2024-04-21T23:54:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:33 smithi038 ceph-mon[27972]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:54:34.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:35.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:34 smithi203 ceph-mon[31431]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:35.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/1446393316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:34 smithi111 ceph-mon[31543]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:35.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/1446393316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:34 smithi038 ceph-mon[27972]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/1446393316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:54:37.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:36 smithi203 ceph-mon[31431]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:37.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:36 smithi111 ceph-mon[31543]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:37.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:36 smithi038 ceph-mon[27972]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:39.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:38 smithi203 ceph-mon[31431]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:38 smithi111 ceph-mon[31543]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:38 smithi038 ceph-mon[27972]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:41.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:40 smithi111 ceph-mon[31543]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:41.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:54:41.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:41.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:40 smithi038 ceph-mon[27972]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:41.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:54:41.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:40 smithi203 ceph-mon[31431]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:54:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:41.905 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:41 smithi203 ceph-mon[31431]: Deploying daemon osd.2 on smithi203 2024-04-21T23:54:42.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:41 smithi111 ceph-mon[31543]: Deploying daemon osd.2 on smithi203 2024-04-21T23:54:42.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:41 smithi038 ceph-mon[27972]: Deploying daemon osd.2 on smithi203 2024-04-21T23:54:43.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:42 smithi111 ceph-mon[31543]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:43.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:42 smithi038 ceph-mon[27972]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:43.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:42 smithi203 ceph-mon[31431]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:45.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:44 smithi203 ceph-mon[31431]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:44 smithi111 ceph-mon[31543]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:45.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:44 smithi038 ceph-mon[27972]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:47.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:46 smithi203 ceph-mon[31431]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:47.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:46 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:46 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:46 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:47.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:46 smithi111 ceph-mon[31543]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:47.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:46 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:46 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:46 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:46 smithi038 ceph-mon[27972]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:46 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:46 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:47.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:46 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:49.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:49.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:49.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:49.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.722+0000 7f92b37fe700 1 -- 172.21.15.203:0/1538112930 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f92c4062670 con 0x7f92ac0a2c10 2024-04-21T23:54:49.716 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 msgr2=0x7f92ac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 0x7f92ac0a50d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f92c4102120 tx=0x7f92b8009190 comp rx=0 tx=0).stop 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 msgr2=0x7f92c419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00b970 tx=0x7f92bc00bc80 comp rx=0 tx=0).stop 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 shutdown_connections 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92ac0a2c10 0x7f92ac0a50d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92c4102c60 0x7f92c419f8d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:49.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92c41010c0 0x7f92c406aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:49.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 --2- 172.21.15.203:0/1538112930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92c41022c0 0x7f92c419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:49.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 >> 172.21.15.203:0/1538112930 conn(0x7f92c40fc640 msgr2=0x7f92c41055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:49.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 shutdown_connections 2024-04-21T23:54:49.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:54:49.724+0000 7f92c359e700 1 -- 172.21.15.203:0/1538112930 wait complete. 2024-04-21T23:54:50.374 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.2.service 2024-04-21T23:54:50.376 INFO:tasks.cephadm:Deploying osd.3 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-21T23:54:50.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:54:50.632 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:54:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:50.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:50 smithi038 ceph-mon[27972]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:50 smithi203 ceph-mon[31431]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:50 smithi111 ceph-mon[31543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:54:51.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:51 smithi203 ceph-mon[31431]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:54:51.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:51 smithi203 ceph-mon[31431]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:51.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:51 smithi203 ceph-mon[31431]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:54:51.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:51 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:51.915 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:51 smithi203 ceph-mon[31431]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:51.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:51 smithi038 ceph-mon[27972]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:54:51.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:51 smithi038 ceph-mon[27972]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:51.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:51 smithi038 ceph-mon[27972]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:54:51.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:51 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:51.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:51 smithi038 ceph-mon[27972]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:51 smithi111 ceph-mon[31543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:54:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:51 smithi111 ceph-mon[31543]: from='osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:51 smithi111 ceph-mon[31543]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:54:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:51 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:51 smithi111 ceph-mon[31543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:54:52.702 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2[36583]: 2024-04-21T23:54:52.426+0000 7f8685291700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T23:54:52.703 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2[36583]: 2024-04-21T23:54:52.432+0000 7f867d07c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109018 s, 962 MB/s 2024-04-21T23:54:52.870 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-21T23:54:52.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:52.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:54:52.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:54:52.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:52.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:52.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:52 smithi038 ceph-mon[27972]: from='osd.2 ' entity='osd.2' 2024-04-21T23:54:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:54:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:54:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:53.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:52 smithi203 ceph-mon[31431]: from='osd.2 ' entity='osd.2' 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:52 smithi111 ceph-mon[31543]: from='osd.2 ' entity='osd.2' 2024-04-21T23:54:53.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-21T23:54:53.856 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:54:54.241 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:54:54.241 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:54:54.241 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.241 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] boot 2024-04-21T23:54:54.241 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:54.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] boot 2024-04-21T23:54:54.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:54.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: osd.2 [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] boot 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:54:54.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:54.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:54.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi203 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:54:55.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi203 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:54:55.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:55 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi203 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:54:55.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:55 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:54:55.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1455917830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac103e30 msgr2=0x7fd3ac10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1455917830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac103e30 0x7fd3ac10a6e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400b0d0 tx=0x7fd3a400b3e0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1455917830 shutdown_connections 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1455917830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac103e30 0x7fd3ac10a6e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1455917830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac102290 0x7fd3ac1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1455917830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1455917830 >> 172.21.15.38:0/1455917830 conn(0x7fd3ac0fd830 msgr2=0x7fd3ac0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1455917830 shutdown_connections 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1455917830 wait complete. 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.644+0000 7fd3b1c8b700 1 Processor -- start 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 -- start start 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 0x7fd3ac19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac10de50 con 0x7fd3ac102290 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac10dcd0 con 0x7fd3ac103e30 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3b1c8b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac10dfd0 con 0x7fd3ac103490 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39246/0 (socket says 172.21.15.38:39246) 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 -- 172.21.15.38:0/1583568108 learned_addr learned my addr 172.21.15.38:0/1583568108 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3aaffd700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3abfff700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 0x7fd3ac19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 -- 172.21.15.38:0/1583568108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 msgr2=0x7fd3ac198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 -- 172.21.15.38:0/1583568108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 msgr2=0x7fd3ac19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:54:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 0x7fd3ac19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3ab7fe700 1 -- 172.21.15.38:0/1583568108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac1a0070 con 0x7fd3ac102290 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3aaffd700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.645+0000 7fd3abfff700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 0x7fd3ac19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3ab7fe700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00d9b0 tx=0x7fd39c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c017030 con 0x7fd3ac102290 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac1a0360 con 0x7fd3ac102290 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd39c004500 con 0x7fd3ac102290 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c003aa0 con 0x7fd3ac102290 2024-04-21T23:54:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.646+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac1a08f0 con 0x7fd3ac102290 2024-04-21T23:54:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.647+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac068f20 con 0x7fd3ac102290 2024-04-21T23:54:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.648+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fd39c01a070 con 0x7fd3ac102290 2024-04-21T23:54:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.649+0000 7fd3a8ff9700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 0x7fd3940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:54:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.649+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7fd39c0cd8d0 con 0x7fd3ac102290 2024-04-21T23:54:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.649+0000 7fd3aaffd700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 0x7fd3940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:54:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.649+0000 7fd3aaffd700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 0x7fd3940a5050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000bc10 tx=0x7fd3a000b5f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:54:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.653+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd39c09bcf0 con 0x7fd3ac102290 2024-04-21T23:54:55.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:55.883+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd3ac062670 con 0x7fd3940a2b90 2024-04-21T23:54:56.534 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47157]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:56.534 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47157]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:56.534 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47157]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:56.534 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47157]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:56.860 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:54:56.860 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:56.860 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:56.860 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 21 23:54:56 smithi038 sudo[47219]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:56.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:56 smithi038 ceph-mon[27972]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:54:56.972 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:56.973 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:56 smithi111 ceph-mon[31543]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:54:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:54:57.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:54:57.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:56 smithi203 ceph-mon[31431]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:54:57.227 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:56 smithi111 sudo[39994]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:57.227 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:56 smithi111 sudo[39994]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:57.228 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:56 smithi111 sudo[39994]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:57.228 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:56 smithi111 sudo[39994]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:57.577 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:57 smithi111 sudo[39997]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:54:57.577 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:57 smithi111 sudo[39997]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:57.577 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:57 smithi111 sudo[39997]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:57.577 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 21 23:54:57 smithi111 sudo[39997]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:57.746 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40507]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:57.746 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40507]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:57.746 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40507]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:57.746 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40507]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:57.746 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:57 smithi203 ceph-mon[31431]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:54:57.746 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:57.746 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:57.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:57 smithi038 ceph-mon[27972]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:54:57.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:57.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:58.030 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40510]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:54:58.030 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:58.030 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40510]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:58.030 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 21 23:54:57 smithi203 sudo[40510]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:58.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:57 smithi111 ceph-mon[31543]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:54:58.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:58.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:58.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47273]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:58.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:58.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47273]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:58.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47273]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:58.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:54:58.705+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd39c097c60 con 0x7fd3ac102290 2024-04-21T23:54:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:54:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47304]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:58.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:58 smithi038 sudo[47304]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:59.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:58 smithi203 sudo[40513]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:58 smithi203 sudo[40513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:58 smithi203 sudo[40513]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:58 smithi203 sudo[40513]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 sudo[40516]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 sudo[40516]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 sudo[40516]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:59.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 sudo[40516]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:54:59.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:54:59.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:59.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: Cluster is now healthy 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:54:59.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-21T23:54:59.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:54:59.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:54:59.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:54:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:54:59.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: Cluster is now healthy 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:54:59.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:54:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:54:59.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40000]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:54:59.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40000]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:54:59.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40000]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:54:59.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40000]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40003]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40003]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40003]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 sudo[40003]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:00.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: Cluster is now healthy 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:55:00.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:54:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:55:00.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]: dispatch 2024-04-21T23:55:00.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]': finished 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:00 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4237132839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]: dispatch 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]': finished 2024-04-21T23:55:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:55:00.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:55:00.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:55:00.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:00.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:00 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4237132839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2998622203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "28f56df4-c518-4ecc-bbfa-51b25a0104d5"}]': finished 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:55:00.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:00 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4237132839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:01.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:01 smithi038 ceph-mon[27972]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:01.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:01 smithi203 ceph-mon[31431]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:01.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:01 smithi111 ceph-mon[31543]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:03.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:03 smithi038 ceph-mon[27972]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:03 smithi203 ceph-mon[31431]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:03 smithi111 ceph-mon[31543]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:05 smithi203 ceph-mon[31431]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:05.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:05 smithi038 ceph-mon[27972]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:05 smithi111 ceph-mon[31543]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:07.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:07 smithi203 ceph-mon[31431]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:07.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:55:07.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:07.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:07 smithi038 ceph-mon[27972]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:07.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:55:07.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:07 smithi111 ceph-mon[31543]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:55:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:08.478 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:08 smithi111 ceph-mon[31543]: Deploying daemon osd.3 on smithi038 2024-04-21T23:55:08.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:08 smithi203 ceph-mon[31431]: Deploying daemon osd.3 on smithi038 2024-04-21T23:55:08.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:08 smithi038 ceph-mon[27972]: Deploying daemon osd.3 on smithi038 2024-04-21T23:55:09.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:09 smithi203 ceph-mon[31431]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:09.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:09 smithi111 ceph-mon[31543]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:09.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:09 smithi038 ceph-mon[27972]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:11 smithi038 ceph-mon[27972]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:11 smithi203 ceph-mon[31431]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:11.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:11 smithi111 ceph-mon[31543]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:13.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:13 smithi038 ceph-mon[27972]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:13.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:13 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:13 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:13 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:13 smithi203 ceph-mon[31431]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:13 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:13 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:13 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:13 smithi111 ceph-mon[31543]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:13 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:13 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:13 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:15.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:15 smithi038 ceph-mon[27972]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:15.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:15 smithi203 ceph-mon[31431]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:15 smithi111 ceph-mon[31543]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:16.456 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 3 on host 'smithi038' 2024-04-21T23:55:16.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.462+0000 7fd3a8ff9700 1 -- 172.21.15.38:0/1583568108 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd3ac062670 con 0x7fd3940a2b90 2024-04-21T23:55:16.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 msgr2=0x7fd3940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 0x7fd3940a5050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000bc10 tx=0x7fd3a000b5f0 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 msgr2=0x7fd3ac198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd39c00d9b0 tx=0x7fd39c00dd70 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 shutdown_connections 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd3940a2b90 0x7fd3940a5050 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3ac103e30 0x7fd3ac19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3ac102290 0x7fd3ac198410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 --2- 172.21.15.38:0/1583568108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3ac103490 0x7fd3ac198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 >> 172.21.15.38:0/1583568108 conn(0x7fd3ac0fd830 msgr2=0x7fd3ac106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 shutdown_connections 2024-04-21T23:55:16.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:55:16.465+0000 7fd3b1c8b700 1 -- 172.21.15.38:0/1583568108 wait complete. 2024-04-21T23:55:17.181 DEBUG:teuthology.orchestra.run.smithi038:osd.3> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.3.service 2024-04-21T23:55:17.183 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_3... 2024-04-21T23:55:17.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:55:17.426 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.549 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:17.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:18 smithi203 ceph-mon[31431]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:55:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:18 smithi203 ceph-mon[31431]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:55:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:18 smithi203 ceph-mon[31431]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:55:18.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:18.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:18 smithi111 ceph-mon[31543]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:55:18.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:18 smithi111 ceph-mon[31543]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:55:18.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:18 smithi111 ceph-mon[31543]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:55:18.825 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:18.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-mon[27972]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:55:18.914 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-mon[27972]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:55:18.914 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-mon[27972]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:55:18.914 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:18.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3[49358]: 2024-04-21T23:55:18.477+0000 7f418d21e700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T23:55:18.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 21 23:55:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3[49358]: 2024-04-21T23:55:18.484+0000 7f4189011700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112702 s, 930 MB/s 2024-04-21T23:55:19.514 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:55:19.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:19.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] boot 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:55:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:19 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] boot 2024-04-21T23:55:19.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:55:19.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:19 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:55:19.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: from='osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:55:19.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:55:19.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:19.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: osd.3 [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] boot 2024-04-21T23:55:19.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:55:19.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:19 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:55:20.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-04-21T23:55:20.420 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:55:20.659 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:20 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:55:20.659 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:20 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:55:20.659 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:20 smithi038 ceph-mon[27972]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:20.660 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:20 smithi038 ceph-mon[27972]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:55:20.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:20 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:55:20.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:20 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:55:20.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:20 smithi203 ceph-mon[31431]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:20.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:20 smithi203 ceph-mon[31431]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:55:20.784 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:20 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:55:20.785 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:20 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:55:20.785 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:20 smithi111 ceph-mon[31543]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:20.785 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:20 smithi111 ceph-mon[31543]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi038 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-21T23:55:21.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:21 smithi111 ceph-mon[31543]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:55:22.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi038 2024-04-21T23:55:22.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:22.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:21 smithi203 ceph-mon[31431]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:55:22.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 -- 172.21.15.111:0/2263794155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1023a0 msgr2=0x7f79dc102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 --2- 172.21.15.111:0/2263794155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1023a0 0x7f79dc102800 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f79d00099c0 tx=0x7f79d0009cd0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 -- 172.21.15.111:0/2263794155 shutdown_connections 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 --2- 172.21.15.111:0/2263794155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc102d40 0x7f79dc1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 --2- 172.21.15.111:0/2263794155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc1011a0 0x7f79dc1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 --2- 172.21.15.111:0/2263794155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1023a0 0x7f79dc102800 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 -- 172.21.15.111:0/2263794155 >> 172.21.15.111:0/2263794155 conn(0x7f79dc0fc760 msgr2=0x7f79dc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 -- 172.21.15.111:0/2263794155 shutdown_connections 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 -- 172.21.15.111:0/2263794155 wait complete. 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.100+0000 7f79e0973700 1 Processor -- start 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 -- start start 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:22.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 0x7f79dc19d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc102d40 0x7f79dc19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc10cdd0 con 0x7f79dc1023a0 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc10cc50 con 0x7f79dc1011a0 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79e0973700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc10cf50 con 0x7f79dc102d40 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56686/0 (socket says 172.21.15.111:56686) 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 -- 172.21.15.111:0/2874140710 learned_addr learned my addr 172.21.15.111:0/2874140710 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79d9d9b700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 0x7f79dc19d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79dad9d700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc102d40 0x7f79dc19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 -- 172.21.15.111:0/2874140710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc102d40 msgr2=0x7f79dc19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc102d40 0x7f79dc19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 -- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 msgr2=0x7f79dc19d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 0x7f79dc19d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 -- 172.21.15.111:0/2874140710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc1a01f0 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79d9d9b700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 0x7f79dc19d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79da59c700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00d980 tx=0x7f79cc00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.101+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc017070 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.102+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc1a04e0 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.102+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79cc00fd70 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.102+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00e3b0 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.102+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc1a0a20 con 0x7f79dc1011a0 2024-04-21T23:55:22.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.103+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc068f20 con 0x7f79dc1011a0 2024-04-21T23:55:22.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.104+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f79cc00f890 con 0x7f79dc1011a0 2024-04-21T23:55:22.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.104+0000 7f79cb7fe700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 0x7f79c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:22.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.104+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f79cc0cd940 con 0x7f79dc1011a0 2024-04-21T23:55:22.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.104+0000 7f79d9d9b700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 0x7f79c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:22.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.105+0000 7f79d9d9b700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 0x7f79c40a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f79d00038c0 tx=0x7f79d0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:55:22.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.109+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f79cc010460 con 0x7f79dc1011a0 2024-04-21T23:55:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi038 2024-04-21T23:55:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:22.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:21 smithi038 ceph-mon[27972]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:55:22.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:22.313+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f79dc02dfa0 con 0x7f79c40a2c20 2024-04-21T23:55:23.027 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:23.027 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:23.027 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:23.027 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:23.028 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:23.028 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:22 smithi111 ceph-mon[31543]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:55:23.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:22 smithi203 ceph-mon[31431]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:55:23.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:23.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:23.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:23.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:23.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:23.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:22 smithi038 ceph-mon[27972]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:55:24.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:24 smithi203 ceph-mon[31431]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:55:24.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:24 smithi203 ceph-mon[31431]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:24.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:24 smithi111 ceph-mon[31543]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:55:24.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:24 smithi111 ceph-mon[31543]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:24.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:24 smithi038 ceph-mon[27972]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:55:24.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:24 smithi038 ceph-mon[27972]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:25 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/2021905517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:25 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:25 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]': finished 2024-04-21T23:55:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:25 smithi111 ceph-mon[31543]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:55:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:25 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/2021905517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:25 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:25 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]': finished 2024-04-21T23:55:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:25 smithi203 ceph-mon[31431]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:55:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:25.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:25 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/2021905517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:25 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]: dispatch 2024-04-21T23:55:25.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:25 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cab4f9b6-2a83-4faf-87a1-1f5b6baaae91"}]': finished 2024-04-21T23:55:25.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:25 smithi038 ceph-mon[27972]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:55:25.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:26 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/1839170470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:26 smithi203 ceph-mon[31431]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-21T23:55:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:26 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/1839170470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:26 smithi111 ceph-mon[31543]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-21T23:55:26.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:26 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/1839170470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:26.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:26 smithi038 ceph-mon[27972]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-21T23:55:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:28 smithi111 ceph-mon[31543]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:55:28.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:28 smithi038 ceph-mon[27972]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:55:29.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:28 smithi203 ceph-mon[31431]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:55:30.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:30 smithi038 ceph-mon[27972]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-21T23:55:30.961 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:30 smithi111 ceph-mon[31543]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-21T23:55:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:30 smithi203 ceph-mon[31431]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-21T23:55:32.951 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:32 smithi111 ceph-mon[31543]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-21T23:55:32.951 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:32 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:55:32.951 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:32 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:32.951 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:32 smithi111 ceph-mon[31543]: Deploying daemon osd.4 on smithi111 2024-04-21T23:55:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:32 smithi038 ceph-mon[27972]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-21T23:55:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:32 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:55:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:32 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:32 smithi038 ceph-mon[27972]: Deploying daemon osd.4 on smithi111 2024-04-21T23:55:33.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:32 smithi203 ceph-mon[31431]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-21T23:55:33.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:32 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:55:33.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:32 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:33.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:32 smithi203 ceph-mon[31431]: Deploying daemon osd.4 on smithi111 2024-04-21T23:55:34.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:34 smithi038 ceph-mon[27972]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:55:35.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:34 smithi203 ceph-mon[31431]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:55:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:34 smithi111 ceph-mon[31543]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:55:36.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:36 smithi038 ceph-mon[27972]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:37.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:36 smithi203 ceph-mon[31431]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:37.062 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:36 smithi111 ceph-mon[31543]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:38 smithi038 ceph-mon[27972]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:39.255 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:38 smithi111 ceph-mon[31543]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:39.256 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.256 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.256 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:39.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:38 smithi203 ceph-mon[31431]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:39.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:39.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:41.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:41.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:41.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:41.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.298 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:41.298 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.298 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.298 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:41.298 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:41.299 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.732+0000 7f79cb7fe700 1 -- 172.21.15.111:0/2874140710 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f79dc02dfa0 con 0x7f79c40a2c20 2024-04-21T23:55:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 msgr2=0x7f79c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 0x7f79c40a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f79d00038c0 tx=0x7f79d0005fb0 comp rx=0 tx=0).stop 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 msgr2=0x7f79dc06b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00d980 tx=0x7f79cc00dc90 comp rx=0 tx=0).stop 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 shutdown_connections 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f79c40a2c20 0x7f79c40a50e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79dc102d40 0x7f79dc19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79dc1011a0 0x7f79dc06b540 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 --2- 172.21.15.111:0/2874140710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1023a0 0x7f79dc19d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 >> 172.21.15.111:0/2874140710 conn(0x7f79dc0fc760 msgr2=0x7f79dc105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:55:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 shutdown_connections 2024-04-21T23:55:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:55:41.734+0000 7f79e0973700 1 -- 172.21.15.111:0/2874140710 wait complete. 2024-04-21T23:55:42.389 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.4.service 2024-04-21T23:55:42.391 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-04-21T23:55:42.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:55:42.636 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:55:42.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:42.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.923 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:42.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:42.924 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:42 smithi111 ceph-mon[31543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:42.951 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:42.952 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.952 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.952 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:42.952 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:42.952 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:42 smithi203 ceph-mon[31431]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:42 smithi038 ceph-mon[27972]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:55:43.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-mon[31543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:55:43.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-mon[31543]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:43.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-mon[31543]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:55:43.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:43.910 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-mon[31543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:43.910 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4[42805]: 2024-04-21T23:55:43.693+0000 7faf1fcd8700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T23:55:43.910 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 21 23:55:43 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4[42805]: 2024-04-21T23:55:43.699+0000 7faf1bacb700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:55:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:43 smithi038 ceph-mon[27972]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:55:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:43 smithi038 ceph-mon[27972]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:43 smithi038 ceph-mon[27972]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:55:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:43 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:43 smithi038 ceph-mon[27972]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:43.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:43 smithi203 ceph-mon[31431]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:55:43.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:43 smithi203 ceph-mon[31431]: from='osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:43.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:43 smithi203 ceph-mon[31431]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:55:43.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:43 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:43.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:43 smithi203 ceph-mon[31431]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:55:44.558 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:55:44.558 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:55:44.558 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-21T23:55:44.558 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-21T23:55:44.558 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144595 s, 725 MB/s 2024-04-21T23:55:44.559 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-21T23:55:44.879 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:44 smithi203 ceph-mon[31431]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:44.879 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:44 smithi203 ceph-mon[31431]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:55:44.879 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:44 smithi203 ceph-mon[31431]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:55:44.879 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:44 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:44.879 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:44 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:44.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:44 smithi038 ceph-mon[27972]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:44.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:44 smithi038 ceph-mon[27972]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:55:44.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:44 smithi038 ceph-mon[27972]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:55:44.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:44 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:44.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:44 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:44 smithi111 ceph-mon[31543]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:44 smithi111 ceph-mon[31543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:55:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:44 smithi111 ceph-mon[31543]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:55:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:44 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:45.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:44 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:45.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-04-21T23:55:45.496 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:55:45.769 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] boot 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:45.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:55:45.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] boot 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:45.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: osd.4 [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] boot 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:46.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:46.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:55:46.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:46.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:46.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:55:46.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:55:46.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:46 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi111 2024-04-21T23:55:46.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:46 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:55:46.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:46 smithi203 ceph-mon[31431]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:46.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:46 smithi203 ceph-mon[31431]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:55:47.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:46 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi111 2024-04-21T23:55:47.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:46 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:55:47.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:46 smithi111 ceph-mon[31543]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:47.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:46 smithi111 ceph-mon[31543]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:55:47.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- 172.21.15.203:0/156016954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 msgr2=0x7f1ee8101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:47.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 --2- 172.21.15.203:0/156016954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8101f80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009990 tx=0x7f1ed8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- 172.21.15.203:0/156016954 shutdown_connections 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 --2- 172.21.15.203:0/156016954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee81024c0 0x7f1ee8108d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 --2- 172.21.15.203:0/156016954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee8100920 0x7f1ee8100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 --2- 172.21.15.203:0/156016954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8101f80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- 172.21.15.203:0/156016954 >> 172.21.15.203:0/156016954 conn(0x7f1ee80fbea0 msgr2=0x7f1ee80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- 172.21.15.203:0/156016954 shutdown_connections 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- 172.21.15.203:0/156016954 wait complete. 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 Processor -- start 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.184+0000 7f1ef0107700 1 -- start start 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee8100920 0x7f1ee8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:47.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee81024c0 0x7f1ee819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee810bc80 con 0x7f1ee8100920 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee810bb00 con 0x7f1ee81024c0 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee810be00 con 0x7f1ee8101b20 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:59554/0 (socket says 172.21.15.203:59554) 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 -- 172.21.15.203:0/2087758898 learned_addr learned my addr 172.21.15.203:0/2087758898 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eee6a4700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee81024c0 0x7f1ee819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eedea3700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee8100920 0x7f1ee8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 -- 172.21.15.203:0/2087758898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee81024c0 msgr2=0x7f1ee819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:47.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee81024c0 0x7f1ee819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 -- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee8100920 msgr2=0x7f1ee8198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee8100920 0x7f1ee8198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 -- 172.21.15.203:0/2087758898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee81a00a0 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1eed6a2700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8005910 tx=0x7f1ed800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8026070 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ed800fe50 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee81a0330 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8017650 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.185+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee81a0780 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.186+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee8068f20 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.187+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1ed8029070 con 0x7f1ee8101b20 2024-04-21T23:55:47.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.188+0000 7f1edeffd700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:55:47.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.188+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f1ed80cf4b0 con 0x7f1ee8101b20 2024-04-21T23:55:47.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.188+0000 7f1eedea3700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:55:47.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.189+0000 7f1eedea3700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8101980 tx=0x7f1ee4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:55:47.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.192+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1ed8013070 con 0x7f1ee8101b20 2024-04-21T23:55:47.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:46 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi111 2024-04-21T23:55:47.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:46 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 8544M 2024-04-21T23:55:47.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:46 smithi038 ceph-mon[27972]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:55:47.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:46 smithi038 ceph-mon[27972]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:55:47.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:55:47.395+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1ee81086f0 con 0x7f1ed40a2c20 2024-04-21T23:55:48.001 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:47 smithi203 ceph-mon[31431]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:48.002 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:47 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:48.002 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:47 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:48.002 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:47 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:48.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:47 smithi111 ceph-mon[31543]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:48.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:47 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:48.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:47 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:48.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:47 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:48.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:47 smithi038 ceph-mon[27972]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:55:48.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:47 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:55:48.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:47 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:55:48.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:47 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:49.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:48 smithi203 ceph-mon[31431]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:49.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:48 smithi111 ceph-mon[31543]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:49.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:48 smithi038 ceph-mon[27972]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/2913631558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]': finished 2024-04-21T23:55:50.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:55:50.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/2913631558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]': finished 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:55:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/2913631558' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]: dispatch 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89520131-b2cc-4385-a99f-e77fbde2670b"}]': finished 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:55:51.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:55:52.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/1175010648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:52.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/1175010648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:52.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/1175010648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:55:53.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:52 smithi203 ceph-mon[31431]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:53.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:52 smithi111 ceph-mon[31543]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:53.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:52 smithi038 ceph-mon[27972]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:54.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:54 smithi203 ceph-mon[31431]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:55.011 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:54 smithi038 ceph-mon[27972]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:55.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:54 smithi111 ceph-mon[31543]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:56 smithi203 ceph-mon[31431]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:57.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:56 smithi111 ceph-mon[31543]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:57.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:56 smithi038 ceph-mon[27972]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:57.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:55:57.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:58.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:55:58.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:58.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:55:58.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:55:59.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:58 smithi203 ceph-mon[31431]: Deploying daemon osd.5 on smithi203 2024-04-21T23:55:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:55:58 smithi203 ceph-mon[31431]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:59.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:58 smithi111 ceph-mon[31543]: Deploying daemon osd.5 on smithi203 2024-04-21T23:55:59.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:55:58 smithi111 ceph-mon[31543]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:55:59.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:58 smithi038 ceph-mon[27972]: Deploying daemon osd.5 on smithi203 2024-04-21T23:55:59.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:55:58 smithi038 ceph-mon[27972]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:01.023 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:00 smithi203 ceph-mon[31431]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:01.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:00 smithi111 ceph-mon[31543]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:01.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:00 smithi038 ceph-mon[27972]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:03.036 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:02 smithi203 ceph-mon[31431]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:03.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:02 smithi111 ceph-mon[31543]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:02 smithi038 ceph-mon[27972]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:04 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:04 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:04 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:04 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:04 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:04 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:04 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:04 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:04 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:05.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:05 smithi038 ceph-mon[27972]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:05.487 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:05 smithi203 ceph-mon[31431]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:05 smithi111 ceph-mon[31543]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:06.708 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-04-21T23:56:06.708 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.714+0000 7f1edeffd700 1 -- 172.21.15.203:0/2087758898 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1ee81086f0 con 0x7f1ed40a2c20 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 msgr2=0x7f1ed40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8101980 tx=0x7f1ee4009260 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 msgr2=0x7f1ee8198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8005910 tx=0x7f1ed800f690 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 shutdown_connections 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ee81024c0 0x7f1ee819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee8100920 0x7f1ee8198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 --2- 172.21.15.203:0/2087758898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ee8101b20 0x7f1ee8198980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 >> 172.21.15.203:0/2087758898 conn(0x7f1ee80fbea0 msgr2=0x7f1ee8106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 shutdown_connections 2024-04-21T23:56:06.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:56:06.717+0000 7f1ef0107700 1 -- 172.21.15.203:0/2087758898 wait complete. 2024-04-21T23:56:07.305 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:07.305 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:07.306 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:07 smithi203 ceph-mon[31431]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:07.331 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.5.service 2024-04-21T23:56:07.334 INFO:tasks.cephadm:Deploying osd.6 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-21T23:56:07.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:07.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:07 smithi111 ceph-mon[31543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:07.591 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:07.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:07 smithi038 ceph-mon[27972]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:56:08.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:08.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:56:08.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:56:08.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:08.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:08.792 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:08 smithi038 ceph-mon[27972]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:08.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:08 smithi203 ceph-mon[31431]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: from='osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:09.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:08 smithi111 ceph-mon[31543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:56:09.675 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:56:09.675 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:56:09.675 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-21T23:56:09.675 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-21T23:56:09.676 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100826 s, 1.0 GB/s 2024-04-21T23:56:09.676 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-21T23:56:09.689 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5[43412]: 2024-04-21T23:56:09.398+0000 7fa144ebe700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T23:56:09.689 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5[43412]: 2024-04-21T23:56:09.405+0000 7fa13cca9700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:56:09.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:56:09.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:56:09.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:09.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: from='osd.5 ' entity='osd.5' 2024-04-21T23:56:09.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] boot 2024-04-21T23:56:09.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:56:09.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: from='osd.5 ' entity='osd.5' 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] boot 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:56:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: from='osd.5 ' entity='osd.5' 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: osd.5 [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] boot 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:56:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:56:10.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-21T23:56:10.610 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:10.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:10 smithi203 ceph-mon[31431]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:10.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:10.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:10 smithi038 ceph-mon[27972]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:11.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:11.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:11.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:11.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:11.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:10 smithi111 ceph-mon[31543]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:56:11.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:11 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi203 2024-04-21T23:56:11.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:11 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-21T23:56:12.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:11 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi203 2024-04-21T23:56:12.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:11 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-21T23:56:12.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:11 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi203 2024-04-21T23:56:12.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:11 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 -- 172.21.15.38:0/1098609172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 msgr2=0x7fe938103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1098609172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938103910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe9240099c0 tx=0x7fe924009cd0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 -- 172.21.15.38:0/1098609172 shutdown_connections 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1098609172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 0x7fe93810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1098609172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 0x7fe9381026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1098609172 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938103910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.339+0000 7fe93ec68700 1 -- 172.21.15.38:0/1098609172 >> 172.21.15.38:0/1098609172 conn(0x7fe9380fd850 msgr2=0x7fe9380ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:56:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- 172.21.15.38:0/1098609172 shutdown_connections 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- 172.21.15.38:0/1098609172 wait complete. 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 Processor -- start 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- start start 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 0x7fe938075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 0x7fe938074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe93810de80 con 0x7fe9381034b0 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe93810dd00 con 0x7fe9381022b0 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.340+0000 7fe93ec68700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe93810e000 con 0x7fe938103e50 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46066/0 (socket says 172.21.15.38:46066) 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 -- 172.21.15.38:0/1647561297 learned_addr learned my addr 172.21.15.38:0/1647561297 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe93ca04700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 0x7fe938075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe93d205700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 0x7fe938074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 -- 172.21.15.38:0/1647561297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 msgr2=0x7fe938074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 0x7fe938074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 -- 172.21.15.38:0/1647561297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 msgr2=0x7fe938075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 0x7fe938075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 -- 172.21.15.38:0/1647561297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe938075410 con 0x7fe9381034b0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe937fff700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe924000c00 tx=0x7fe924005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924027070 con 0x7fe9381034b0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9381a24a0 con 0x7fe9381034b0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9240041d0 con 0x7fe9381034b0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92400f460 con 0x7fe9381034b0 2024-04-21T23:56:12.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.341+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9381a28a0 con 0x7fe9381034b0 2024-04-21T23:56:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.342+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe938068f20 con 0x7fe9381034b0 2024-04-21T23:56:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.344+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe92402a070 con 0x7fe9381034b0 2024-04-21T23:56:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.344+0000 7fe935ffb700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 0x7fe9280a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.344+0000 7fe93ca04700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 0x7fe9280a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:12.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.344+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7fe9240cf6f0 con 0x7fe9381034b0 2024-04-21T23:56:12.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.345+0000 7fe93ca04700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 0x7fe9280a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005fd0 tx=0x7fe92c005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:56:12.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.348+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe924013070 con 0x7fe9381034b0 2024-04-21T23:56:12.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:12.571+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe938062670 con 0x7fe9280a2c20 2024-04-21T23:56:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:12 smithi038 ceph-mon[27972]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:12 smithi038 ceph-mon[27972]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:56:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:12 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:12 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:12 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:13.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:12 smithi203 ceph-mon[31431]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:13.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:12 smithi203 ceph-mon[31431]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:56:13.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:12 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:13.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:12 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:13.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:12 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:12 smithi111 ceph-mon[31543]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:12 smithi111 ceph-mon[31543]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:56:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:12 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:12 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:12 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:13.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:13 smithi038 ceph-mon[27972]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:13.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:13 smithi038 ceph-mon[27972]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:56:14.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:13 smithi203 ceph-mon[31431]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:14.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:13 smithi203 ceph-mon[31431]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:56:14.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:13 smithi111 ceph-mon[31543]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:14.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:13 smithi111 ceph-mon[31543]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:56:15.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:14 smithi203 ceph-mon[31431]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:15.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:14 smithi203 ceph-mon[31431]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:56:15.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:14 smithi111 ceph-mon[31543]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:15.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:14 smithi111 ceph-mon[31543]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:56:15.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:14 smithi038 ceph-mon[27972]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:15.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:14 smithi038 ceph-mon[27972]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:56:16.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:15 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]: dispatch 2024-04-21T23:56:16.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:15 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]': finished 2024-04-21T23:56:16.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:15 smithi038 ceph-mon[27972]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:56:16.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:15 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:16.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:15 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]: dispatch 2024-04-21T23:56:16.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:15 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]': finished 2024-04-21T23:56:16.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:15 smithi203 ceph-mon[31431]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:56:16.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:15 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:16.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:15 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]: dispatch 2024-04-21T23:56:16.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:15 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3156310850' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe082d0b-2acc-446e-ba14-845d68b911e2"}]': finished 2024-04-21T23:56:16.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:15 smithi111 ceph-mon[31543]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:56:16.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:15 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:16 smithi203 ceph-mon[31431]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:56:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:16 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2999260667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:16 smithi111 ceph-mon[31543]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:56:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:16 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2999260667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:16 smithi038 ceph-mon[27972]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:56:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:16 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2999260667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:18 smithi203 ceph-mon[31431]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T23:56:19.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:18 smithi111 ceph-mon[31543]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T23:56:19.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:18 smithi038 ceph-mon[27972]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-21T23:56:20.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:20 smithi038 ceph-mon[27972]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:56:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:20 smithi203 ceph-mon[31431]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:56:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:20 smithi111 ceph-mon[31543]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:56:22.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:22 smithi038 ceph-mon[27972]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:56:23.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:22 smithi203 ceph-mon[31431]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:56:23.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:22 smithi111 ceph-mon[31543]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:56:24.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:56:24.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:24.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:23 smithi203 ceph-mon[31431]: Deploying daemon osd.6 on smithi038 2024-04-21T23:56:24.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:56:24.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:24.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:23 smithi111 ceph-mon[31543]: Deploying daemon osd.6 on smithi038 2024-04-21T23:56:24.114 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:56:24.114 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:24.114 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:23 smithi038 ceph-mon[27972]: Deploying daemon osd.6 on smithi038 2024-04-21T23:56:25.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:24 smithi203 ceph-mon[31431]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:56:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:24 smithi111 ceph-mon[31543]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:56:25.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:24 smithi038 ceph-mon[27972]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:56:27.023 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:26 smithi038 ceph-mon[27972]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:26 smithi111 ceph-mon[31543]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:27.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:26 smithi203 ceph-mon[31431]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:28.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:28 smithi038 ceph-mon[27972]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:29.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:28 smithi111 ceph-mon[31543]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:29.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:28 smithi203 ceph-mon[31431]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:30.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:30.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:30.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:30.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:31.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:31 smithi111 ceph-mon[31543]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:31.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:31 smithi038 ceph-mon[27972]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:31.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:31 smithi203 ceph-mon[31431]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:33.328 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 6 on host 'smithi038' 2024-04-21T23:56:33.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.334+0000 7fe935ffb700 1 -- 172.21.15.38:0/1647561297 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe938062670 con 0x7fe9280a2c20 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 msgr2=0x7fe9280a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 0x7fe9280a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005fd0 tx=0x7fe92c005e50 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 msgr2=0x7fe938074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe924000c00 tx=0x7fe924005e80 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 shutdown_connections 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe9280a2c20 0x7fe9280a50e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe938103e50 0x7fe938074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9381022b0 0x7fe938075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 --2- 172.21.15.38:0/1647561297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9381034b0 0x7fe938074620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:33.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.336+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 >> 172.21.15.38:0/1647561297 conn(0x7fe9380fd850 msgr2=0x7fe9381066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:56:33.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.337+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 shutdown_connections 2024-04-21T23:56:33.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:56:33.337+0000 7fe93ec68700 1 -- 172.21.15.38:0/1647561297 wait complete. 2024-04-21T23:56:33.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:33.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:56:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:33 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:33 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:33.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:33 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:34.253 DEBUG:teuthology.orchestra.run.smithi038:osd.6> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.6.service 2024-04-21T23:56:34.255 INFO:tasks.cephadm:Deploying osd.7 on smithi111 with /dev/vg_nvme/lv_2... 2024-04-21T23:56:34.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:56:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-mon[27972]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:56:34.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-mon[27972]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:56:34.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-mon[27972]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:56:34.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:34.494 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6[56355]: 2024-04-21T23:56:34.425+0000 7fea4360d700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T23:56:34.494 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 21 23:56:34 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6[56355]: 2024-04-21T23:56:34.433+0000 7fea3f400700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:56:34.502 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:56:34.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:34 smithi111 ceph-mon[31543]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:56:34.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:34 smithi111 ceph-mon[31543]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:56:34.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:34 smithi111 ceph-mon[31543]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:56:34.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:34.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:34 smithi203 ceph-mon[31431]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:56:34.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:34 smithi203 ceph-mon[31431]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:56:34.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:34 smithi203 ceph-mon[31431]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:56:34.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:35.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:56:35.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:56:35.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:35.811 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:56:35.811 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:56:35.811 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.812 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.812 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: from='osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:36.700 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:36 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:56:36.700 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:36 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:56:36.700 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:36 smithi203 ceph-mon[31431]: osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] boot 2024-04-21T23:56:36.700 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:36 smithi203 ceph-mon[31431]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:56:36.700 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:36 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:56:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:36 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:56:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:36 smithi038 ceph-mon[27972]: osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] boot 2024-04-21T23:56:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:36 smithi038 ceph-mon[27972]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:56:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:36.728 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114051 s, 919 MB/s 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 2024-04-21T23:56:36.729 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:56:36.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:36 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:56:36.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:36 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:56:36.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:36 smithi111 ceph-mon[31543]: osd.6 [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] boot 2024-04-21T23:56:36.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:36 smithi111 ceph-mon[31543]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:56:36.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:56:37.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-04-21T23:56:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:37 smithi038 ceph-mon[27972]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:37 smithi038 ceph-mon[27972]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:56:37.717 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:56:37.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:37 smithi203 ceph-mon[31431]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:37.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:37 smithi203 ceph-mon[31431]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:56:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:37 smithi111 ceph-mon[31543]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:56:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:37 smithi111 ceph-mon[31543]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:56:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi038 2024-04-21T23:56:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:38.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi038 2024-04-21T23:56:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:38.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi038 2024-04-21T23:56:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 -- 172.21.15.111:0/1049110212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103e20 msgr2=0x7f4c7810a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/1049110212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103e20 0x7f4c7810a6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000b0d0 tx=0x7f4c7000b3e0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 -- 172.21.15.111:0/1049110212 shutdown_connections 2024-04-21T23:56:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/1049110212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103e20 0x7f4c7810a6d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/1049110212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 0x7f4c781026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.423+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/1049110212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103480 0x7f4c781038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- 172.21.15.111:0/1049110212 >> 172.21.15.111:0/1049110212 conn(0x7f4c780fd820 msgr2=0x7f4c780ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:56:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- 172.21.15.111:0/1049110212 shutdown_connections 2024-04-21T23:56:39.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- 172.21.15.111:0/1049110212 wait complete. 2024-04-21T23:56:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 Processor -- start 2024-04-21T23:56:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- start start 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 0x7f4c78198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 0x7f4c7819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810de60 con 0x7f4c78103e20 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810dce0 con 0x7f4c78103480 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.424+0000 7f4c7ff88700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810dfe0 con 0x7f4c78102280 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51718/0 (socket says 172.21.15.111:51718) 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 -- 172.21.15.111:0/2625787702 learned_addr learned my addr 172.21.15.111:0/2625787702 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7dd24700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 0x7f4c78198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7e525700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 0x7f4c7819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 -- 172.21.15.111:0/2625787702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 msgr2=0x7f4c78198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 0x7f4c78198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 -- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 msgr2=0x7f4c7819f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 0x7f4c7819f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 -- 172.21.15.111:0/2625787702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c781a0070 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7e525700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 0x7f4c7819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7d523700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800d980 tx=0x7f4c6800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68016070 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c68004d10 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68005680 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c781a0360 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.425+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c781a08f0 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.426+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c78068f20 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.427+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4c68004340 con 0x7f4c78103480 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.428+0000 7f4c6effd700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 0x7f4c640a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:56:39.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.428+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f4c680ce5f0 con 0x7f4c78103480 2024-04-21T23:56:39.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.428+0000 7f4c7dd24700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 0x7f4c640a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:56:39.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.428+0000 7f4c7dd24700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 0x7f4c640a5160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c781032e0 tx=0x7f4c7400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:56:39.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.433+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c6801f020 con 0x7f4c78103480 2024-04-21T23:56:39.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:39.639+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4c78062670 con 0x7f4c640a2ca0 2024-04-21T23:56:39.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:39 smithi203 ceph-mon[31431]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:39.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:39 smithi111 ceph-mon[31543]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:39.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:39 smithi038 ceph-mon[27972]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:40.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:40.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:40.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:40 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:40.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:40.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:40.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:40 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:40.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:56:40.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:56:40.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:40 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:41.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:41 smithi111 ceph-mon[31543]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:41.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:41 smithi111 ceph-mon[31543]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:41.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:41 smithi203 ceph-mon[31431]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:41.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:41 smithi203 ceph-mon[31431]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:41.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:41 smithi038 ceph-mon[27972]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:41.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:41 smithi038 ceph-mon[27972]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:56:42.764 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:42 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/2504918032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.765 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:42 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.765 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:42 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]': finished 2024-04-21T23:56:42.765 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:42 smithi111 ceph-mon[31543]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:56:42.765 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:56:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:42 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/2504918032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:42 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:42 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]': finished 2024-04-21T23:56:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:42 smithi203 ceph-mon[31431]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:56:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:56:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:42 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/2504918032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:42 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]: dispatch 2024-04-21T23:56:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:42 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bff1d33-4cb9-45ba-8cf4-efc6051da49a"}]': finished 2024-04-21T23:56:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:42 smithi038 ceph-mon[27972]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:56:42.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:56:43.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:43 smithi203 ceph-mon[31431]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:43 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/2188108932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:43 smithi111 ceph-mon[31543]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:43 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/2188108932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:43 smithi038 ceph-mon[27972]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:43.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:43 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/2188108932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:56:44.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:44 smithi203 ceph-mon[31431]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:44.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:44 smithi111 ceph-mon[31543]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:44.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:44 smithi038 ceph-mon[27972]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:46.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:46 smithi038 ceph-mon[27972]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:47.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:46 smithi203 ceph-mon[31431]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:47.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:46 smithi111 ceph-mon[31543]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:48.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:48 smithi111 ceph-mon[31543]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:48.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:48 smithi038 ceph-mon[27972]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:49.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:48 smithi203 ceph-mon[31431]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:50 smithi038 ceph-mon[27972]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:56:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:50.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:50 smithi038 ceph-mon[27972]: Deploying daemon osd.7 on smithi111 2024-04-21T23:56:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:50 smithi203 ceph-mon[31431]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:56:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:51.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:50 smithi203 ceph-mon[31431]: Deploying daemon osd.7 on smithi111 2024-04-21T23:56:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:50 smithi111 ceph-mon[31543]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:56:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:51.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:50 smithi111 ceph-mon[31543]: Deploying daemon osd.7 on smithi111 2024-04-21T23:56:52.926 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:52 smithi111 ceph-mon[31543]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:52.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:52 smithi038 ceph-mon[27972]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:53.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:52 smithi203 ceph-mon[31431]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:54.919 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:54 smithi111 ceph-mon[31543]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:54.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:54 smithi038 ceph-mon[27972]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:55.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:54 smithi203 ceph-mon[31431]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:56.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:56.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:56.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:56.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:56 smithi111 ceph-mon[31543]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:56 smithi038 ceph-mon[27972]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:57.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:56:57.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:56 smithi203 ceph-mon[31431]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:58.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:56:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:56:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.056 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:56:59.056 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.057 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.057 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:56:59.057 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:56:59.057 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:56:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:56:59.451 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.458+0000 7f4c6effd700 1 -- 172.21.15.111:0/2625787702 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4c78062670 con 0x7f4c640a2ca0 2024-04-21T23:56:59.452 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 7 on host 'smithi111' 2024-04-21T23:56:59.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 msgr2=0x7f4c640a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:59.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 0x7f4c640a5160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c781032e0 tx=0x7f4c7400b410 comp rx=0 tx=0).stop 2024-04-21T23:56:59.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 msgr2=0x7f4c78198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:56:59.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800d980 tx=0x7f4c6800dc90 comp rx=0 tx=0).stop 2024-04-21T23:56:59.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 shutdown_connections 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c640a2ca0 0x7f4c640a5160 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c78103e20 0x7f4c7819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c78102280 0x7f4c78198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 --2- 172.21.15.111:0/2625787702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c78103480 0x7f4c78198950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 >> 172.21.15.111:0/2625787702 conn(0x7f4c780fd820 msgr2=0x7f4c78106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 shutdown_connections 2024-04-21T23:56:59.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:56:59.461+0000 7f4c7ff88700 1 -- 172.21.15.111:0/2625787702 wait complete. 2024-04-21T23:57:00.180 DEBUG:teuthology.orchestra.run.smithi111:osd.7> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.7.service 2024-04-21T23:57:00.182 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-04-21T23:57:00.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:57:00.403 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.403 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.404 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:00.404 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:00 smithi111 ceph-mon[31543]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:00.404 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:00 smithi111 ceph-mon[31543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:00.429 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:57:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:00 smithi203 ceph-mon[31431]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:00 smithi203 ceph-mon[31431]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:00 smithi038 ceph-mon[27972]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:00 smithi038 ceph-mon[27972]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:57:01.685 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:01.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:57:01.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:01.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:57:01.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:01.686 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-mon[31543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:01.686 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7[49513]: 2024-04-21T23:57:01.431+0000 7f4b47c40700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T23:57:01.686 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 21 23:57:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7[49513]: 2024-04-21T23:57:01.442+0000 7f4b43a33700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:01 smithi038 ceph-mon[27972]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: from='osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:01 smithi203 ceph-mon[31431]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132435 s, 792 MB/s 2024-04-21T23:57:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-21T23:57:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:57:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:57:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] boot 2024-04-21T23:57:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:57:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:02 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] boot 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:57:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:02 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: osd.7 [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] boot 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:57:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:02 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:57:03.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-04-21T23:57:03.471 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:03.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:03.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:03 smithi203 ceph-mon[31431]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:03.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:03.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:03 smithi111 ceph-mon[31543]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:03 smithi038 ceph-mon[27972]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:57:04.614 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:04 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi111 2024-04-21T23:57:04.614 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:04 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:57:04.614 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:04 smithi203 ceph-mon[31431]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:57:04.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:04 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi111 2024-04-21T23:57:04.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:04 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:57:04.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:04 smithi038 ceph-mon[27972]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:57:04.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:04 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi111 2024-04-21T23:57:04.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:04 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 5696M 2024-04-21T23:57:04.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:04 smithi111 ceph-mon[31543]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:57:05.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- 172.21.15.203:0/2447643717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 msgr2=0x7f8ea8103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/2447643717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8103d50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009990 tx=0x7f8e98009ca0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- 172.21.15.203:0/2447643717 shutdown_connections 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/2447643717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea810a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/2447643717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8103d50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/2447643717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- 172.21.15.203:0/2447643717 >> 172.21.15.203:0/2447643717 conn(0x7f8ea80fbf70 msgr2=0x7f8ea80fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- 172.21.15.203:0/2447643717 shutdown_connections 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- 172.21.15.203:0/2447643717 wait complete. 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 Processor -- start 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.178+0000 7f8eae7b2700 1 -- start start 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:05.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea8101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea810deb0 con 0x7f8ea81003b0 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea810dd30 con 0x7f8ea8100d50 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eae7b2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea810e030 con 0x7f8ea8103930 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:49696/0 (socket says 172.21.15.203:49696) 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 -- 172.21.15.203:0/291236929 learned_addr learned my addr 172.21.15.203:0/291236929 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8ea7fff700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8ea77fe700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea8101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 -- 172.21.15.203:0/291236929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 msgr2=0x7f8ea8101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea8101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 -- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 msgr2=0x7f8ea8103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 -- 172.21.15.203:0/291236929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea8102470 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8ea7fff700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8ea77fe700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea8101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.179+0000 7f8eacd4f700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000ada0 tx=0x7f8ea000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.180+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0017020 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.180+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ea0004bb0 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.180+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea81a2680 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.180+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0005520 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.180+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea81a2b90 con 0x7f8ea8103930 2024-04-21T23:57:05.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.181+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea8068f20 con 0x7f8ea8103930 2024-04-21T23:57:05.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.182+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8ea00056c0 con 0x7f8ea8103930 2024-04-21T23:57:05.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.183+0000 7f8ea57fa700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 0x7f8e900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:05.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.183+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f8ea00ce1c0 con 0x7f8ea8103930 2024-04-21T23:57:05.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.183+0000 7f8ea7fff700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 0x7f8e900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:05.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.184+0000 7f8ea7fff700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 0x7f8e900a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800b5c0 tx=0x7f8e98005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:05.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.187+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8ea0021020 con 0x7f8ea8103930 2024-04-21T23:57:05.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:05.388+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8ea8062670 con 0x7f8e900a2c20 2024-04-21T23:57:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:05 smithi038 ceph-mon[27972]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:05.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:05 smithi203 ceph-mon[31431]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:05.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:05 smithi111 ceph-mon[31543]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:06.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:06 smithi203 ceph-mon[31431]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:06 smithi203 ceph-mon[31431]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:57:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:06 smithi038 ceph-mon[27972]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:06.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:06 smithi038 ceph-mon[27972]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:57:06.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:06 smithi111 ceph-mon[31543]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:06 smithi111 ceph-mon[31543]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:57:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:07 smithi038 ceph-mon[27972]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:07.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:07 smithi038 ceph-mon[27972]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:57:07.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:07 smithi203 ceph-mon[31431]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:07.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:07 smithi203 ceph-mon[31431]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:57:07.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:07 smithi111 ceph-mon[31543]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:07.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:07 smithi111 ceph-mon[31543]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:57:08.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:08 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/1623380081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:08 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:08 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]': finished 2024-04-21T23:57:08.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:08 smithi203 ceph-mon[31431]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:57:08.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:08 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:08.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:08 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/1623380081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:08 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:08 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]': finished 2024-04-21T23:57:08.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:08 smithi111 ceph-mon[31543]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:57:08.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:08 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:08.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:08 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/1623380081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:08 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]: dispatch 2024-04-21T23:57:08.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:08 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d85f1f1-9fed-495f-92b8-cd9f0fad42ce"}]': finished 2024-04-21T23:57:08.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:08 smithi038 ceph-mon[27972]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:57:08.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:08 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:09.728 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:09 smithi111 ceph-mon[31543]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:09.728 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/3958725128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:09.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:09 smithi203 ceph-mon[31431]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:09.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/3958725128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:09 smithi038 ceph-mon[27972]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/3958725128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:11 smithi203 ceph-mon[31431]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:11.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:11 smithi111 ceph-mon[31543]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:11.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:11 smithi038 ceph-mon[27972]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:13.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:13 smithi203 ceph-mon[31431]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:13.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:13 smithi111 ceph-mon[31543]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:13.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:13 smithi038 ceph-mon[27972]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:14.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:14 smithi111 ceph-mon[31543]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:14.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:14 smithi203 ceph-mon[31431]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:14.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:14 smithi038 ceph-mon[27972]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:16.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:16 smithi038 ceph-mon[27972]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:16.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:57:16.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:16 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:16.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:16 smithi038 ceph-mon[27972]: Deploying daemon osd.8 on smithi203 2024-04-21T23:57:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:16 smithi203 ceph-mon[31431]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:57:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:16 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:16 smithi203 ceph-mon[31431]: Deploying daemon osd.8 on smithi203 2024-04-21T23:57:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:16 smithi111 ceph-mon[31543]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:16 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:57:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:16 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:16 smithi111 ceph-mon[31543]: Deploying daemon osd.8 on smithi203 2024-04-21T23:57:18.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:18 smithi203 ceph-mon[31431]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:18.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:18 smithi038 ceph-mon[27972]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:19.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:18 smithi111 ceph-mon[31543]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:20.881 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:20 smithi203 ceph-mon[31431]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:20 smithi038 ceph-mon[27972]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:20 smithi111 ceph-mon[31543]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:22 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:22 smithi203 ceph-mon[31431]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:22.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:22 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:22.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:22 smithi111 ceph-mon[31543]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:22.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:22.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:22 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:22.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:22 smithi038 ceph-mon[27972]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:24.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:25.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:25.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:25.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:25.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:25.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.553 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-04-21T23:57:25.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.560+0000 7f8ea57fa700 1 -- 172.21.15.203:0/291236929 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8ea8062670 con 0x7f8e900a2c20 2024-04-21T23:57:25.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.562+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 msgr2=0x7f8e900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:25.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.562+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 0x7f8e900a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800b5c0 tx=0x7f8e98005fb0 comp rx=0 tx=0).stop 2024-04-21T23:57:25.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.562+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 msgr2=0x7f8ea8101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.562+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000ada0 tx=0x7f8ea000c330 comp rx=0 tx=0).stop 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 shutdown_connections 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8e900a2c20 0x7f8e900a50e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ea8103930 0x7f8ea8101bc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ea81003b0 0x7f8ea8103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 --2- 172.21.15.203:0/291236929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ea8100d50 0x7f8ea8101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 >> 172.21.15.203:0/291236929 conn(0x7f8ea80fbf70 msgr2=0x7f8ea81069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 shutdown_connections 2024-04-21T23:57:25.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:57:25.563+0000 7f8eae7b2700 1 -- 172.21.15.203:0/291236929 wait complete. 2024-04-21T23:57:25.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:25 smithi203 ceph-mon[31431]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:25.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:25 smithi203 ceph-mon[31431]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:25.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:25 smithi038 ceph-mon[27972]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:25 smithi038 ceph-mon[27972]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:25.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:25 smithi111 ceph-mon[31543]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:25 smithi111 ceph-mon[31543]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:57:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:26.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:26.245 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.8.service 2024-04-21T23:57:26.248 INFO:tasks.cephadm:Deploying osd.9 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-21T23:57:26.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:57:26.507 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:57:26.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:26.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:57:26.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:57:26.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:26.756 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:26.757 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:26 smithi203 ceph-mon[31431]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:26.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:26 smithi038 ceph-mon[27972]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: from='osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:27.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:26 smithi111 ceph-mon[31543]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:57:27.530 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8[50287]: 2024-04-21T23:57:27.414+0000 7fbe7b9d7700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T23:57:27.530 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8[50287]: 2024-04-21T23:57:27.422+0000 7fbe737c2700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: from='osd.8 ' entity='osd.8' 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] boot 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:57:27.860 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: from='osd.8 ' entity='osd.8' 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] boot 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:57:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: from='osd.8 ' entity='osd.8' 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: osd.8 [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] boot 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:57:28.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:57:28.766 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:57:28.767 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:57:28.767 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-21T23:57:28.767 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-21T23:57:28.767 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144225 s, 727 MB/s 2024-04-21T23:57:28.767 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-21T23:57:29.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-21T23:57:29.679 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:29 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:57:29.679 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:29 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:57:29.679 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:29 smithi203 ceph-mon[31431]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:29.679 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:29 smithi203 ceph-mon[31431]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:57:29.702 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:57:29.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:29 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:57:29.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:29 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:57:29.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:29 smithi038 ceph-mon[27972]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:29.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:29 smithi038 ceph-mon[27972]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:57:29.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:29 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:57:29.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:29 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:57:29.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:29 smithi111 ceph-mon[31543]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:57:29.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:29 smithi111 ceph-mon[31543]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:57:30.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:30.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi203 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:30.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:30 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi203 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-21T23:57:31.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:31.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:31.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:30 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi203 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:31.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:30 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 -- 172.21.15.38:0/794470417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40103490 msgr2=0x7f0c401038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 --2- 172.21.15.38:0/794470417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40103490 0x7f0c401038f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c340099c0 tx=0x7f0c34009cd0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 -- 172.21.15.38:0/794470417 shutdown_connections 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 --2- 172.21.15.38:0/794470417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 0x7f0c4010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 --2- 172.21.15.38:0/794470417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40102290 0x7f0c401026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 --2- 172.21.15.38:0/794470417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40103490 0x7f0c401038f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 -- 172.21.15.38:0/794470417 >> 172.21.15.38:0/794470417 conn(0x7f0c400fd830 msgr2=0x7f0c400ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 -- 172.21.15.38:0/794470417 shutdown_connections 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.543+0000 7f0c46467700 1 -- 172.21.15.38:0/794470417 wait complete. 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 Processor -- start 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 -- start start 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 0x7f0c40074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 0x7f0c40074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4010dec0 con 0x7f0c40102290 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4010dd40 con 0x7f0c40103490 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c46467700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4010e040 con 0x7f0c40103e30 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34284/0 (socket says 172.21.15.38:34284) 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 -- 172.21.15.38:0/1837503369 learned_addr learned my addr 172.21.15.38:0/1837503369 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c44a04700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 0x7f0c40074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3f7fe700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 0x7f0c40074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 -- 172.21.15.38:0/1837503369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 msgr2=0x7f0c40074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 0x7f0c40074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.544+0000 7f0c3ffff700 1 -- 172.21.15.38:0/1837503369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 msgr2=0x7f0c40074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3ffff700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 0x7f0c40074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3ffff700 1 -- 172.21.15.38:0/1837503369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c40075410 con 0x7f0c40102290 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3f7fe700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 0x7f0c40074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3ffff700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000c970 tx=0x7f0c3000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30004220 con 0x7f0c40102290 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c30007d80 con 0x7f0c40102290 2024-04-21T23:57:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c401a2660 con 0x7f0c40102290 2024-04-21T23:57:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30003e50 con 0x7f0c40102290 2024-04-21T23:57:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.545+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c401a2ba0 con 0x7f0c40102290 2024-04-21T23:57:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.546+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c40068f20 con 0x7f0c40102290 2024-04-21T23:57:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.548+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0c300078a0 con 0x7f0c40102290 2024-04-21T23:57:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.549+0000 7f0c3d7fa700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 0x7f0c280a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.549+0000 7f0c3f7fe700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 0x7f0c280a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.549+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f0c300ce660 con 0x7f0c40102290 2024-04-21T23:57:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.549+0000 7f0c3f7fe700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 0x7f0c280a5300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009fd0 tx=0x7f0c34000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:31.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.552+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0c30014030 con 0x7f0c40102290 2024-04-21T23:57:31.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:31.768+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0c40062670 con 0x7f0c280a2e40 2024-04-21T23:57:32.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:31 smithi111 ceph-mon[31543]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:57:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:32.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:31 smithi038 ceph-mon[27972]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:57:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:32.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:31 smithi203 ceph-mon[31431]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:57:32.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:32.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:32.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:32 smithi038 ceph-mon[27972]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:32 smithi038 ceph-mon[27972]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:32.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:32 smithi038 ceph-mon[27972]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:57:33.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:32 smithi111 ceph-mon[31543]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:33.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:32 smithi111 ceph-mon[31543]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:33.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:32 smithi111 ceph-mon[31543]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:57:33.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:32 smithi203 ceph-mon[31431]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:33.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:32 smithi203 ceph-mon[31431]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:33.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:32 smithi203 ceph-mon[31431]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:57:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:34 smithi111 ceph-mon[31543]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:57:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]: dispatch 2024-04-21T23:57:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]': finished 2024-04-21T23:57:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:34 smithi111 ceph-mon[31543]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:57:35.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:34 smithi038 ceph-mon[27972]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:57:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]: dispatch 2024-04-21T23:57:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]': finished 2024-04-21T23:57:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:34 smithi038 ceph-mon[27972]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:57:35.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:35.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:34 smithi203 ceph-mon[31431]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-21T23:57:35.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]: dispatch 2024-04-21T23:57:35.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1965406226' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75dc496d-7396-4c25-94c0-4976235724a8"}]': finished 2024-04-21T23:57:35.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:34 smithi203 ceph-mon[31431]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:57:35.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:36.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:35 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1353754910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:36.181 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:35 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1353754910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:36.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:35 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1353754910' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:57:37.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:36 smithi111 ceph-mon[31543]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T23:57:37.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:36 smithi038 ceph-mon[27972]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T23:57:37.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:36 smithi203 ceph-mon[31431]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T23:57:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:38 smithi111 ceph-mon[31543]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:57:39.187 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:38 smithi038 ceph-mon[27972]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:57:39.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:38 smithi203 ceph-mon[31431]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-21T23:57:41.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:40 smithi111 ceph-mon[31543]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:57:41.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:40 smithi038 ceph-mon[27972]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:57:41.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:40 smithi203 ceph-mon[31431]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:57:43.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:42 smithi038 ceph-mon[27972]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-21T23:57:43.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:57:43.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:42 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:43.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:42 smithi038 ceph-mon[27972]: Deploying daemon osd.9 on smithi038 2024-04-21T23:57:43.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:42 smithi111 ceph-mon[31543]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-21T23:57:43.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:57:43.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:42 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:43.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:42 smithi111 ceph-mon[31543]: Deploying daemon osd.9 on smithi038 2024-04-21T23:57:43.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:42 smithi203 ceph-mon[31431]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-21T23:57:43.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:57:43.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:42 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:43.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:42 smithi203 ceph-mon[31431]: Deploying daemon osd.9 on smithi038 2024-04-21T23:57:44.869 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:44 smithi038 ceph-mon[27972]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:45.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:44 smithi111 ceph-mon[31543]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:45.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:44 smithi203 ceph-mon[31431]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:47.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:46 smithi038 ceph-mon[27972]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:47.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:46 smithi203 ceph-mon[31431]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:47.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:46 smithi111 ceph-mon[31543]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:49 smithi038 ceph-mon[27972]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:49.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:49 smithi203 ceph-mon[31431]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:49.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:49.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:49 smithi111 ceph-mon[31543]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:49.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:49.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:51.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:51 smithi038 ceph-mon[27972]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:51.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:51 smithi203 ceph-mon[31431]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:51.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:51 smithi111 ceph-mon[31543]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:52.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.380+0000 7f0c3d7fa700 1 -- 172.21.15.38:0/1837503369 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0c40062670 con 0x7f0c280a2e40 2024-04-21T23:57:52.374 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 9 on host 'smithi038' 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 msgr2=0x7f0c280a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 0x7f0c280a5300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009fd0 tx=0x7f0c34000bc0 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 msgr2=0x7f0c40075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000c970 tx=0x7f0c3000cc80 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 shutdown_connections 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0c280a2e40 0x7f0c280a5300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c40103e30 0x7f0c40074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0c40102290 0x7f0c40075fd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 --2- 172.21.15.38:0/1837503369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c40103490 0x7f0c40074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 >> 172.21.15.38:0/1837503369 conn(0x7f0c400fd830 msgr2=0x7f0c40106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:57:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 shutdown_connections 2024-04-21T23:57:52.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:57:52.383+0000 7f0c46467700 1 -- 172.21.15.38:0/1837503369 wait complete. 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:52 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:52 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:52 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:57:53.160 DEBUG:teuthology.orchestra.run.smithi038:osd.9> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.9.service 2024-04-21T23:57:53.162 INFO:tasks.cephadm:Deploying osd.10 on smithi111 with /dev/vg_nvme/lv_1... 2024-04-21T23:57:53.163 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:57:53.408 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:57:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-mon[27972]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-mon[27972]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:57:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-mon[27972]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:57:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-mon[27972]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:57:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:53.709 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-21T23:57:53.488+0000 7fc987220700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T23:57:53.709 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 21 23:57:53 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-21T23:57:53.504+0000 7fc983013700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:57:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:53 smithi203 ceph-mon[31431]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:53 smithi203 ceph-mon[31431]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:57:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:53 smithi203 ceph-mon[31431]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:57:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:53 smithi203 ceph-mon[31431]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:57:53.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:53 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:53.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:53 smithi111 ceph-mon[31543]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:53.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:53 smithi111 ceph-mon[31543]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:57:53.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:53 smithi111 ceph-mon[31543]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:57:53.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:53 smithi111 ceph-mon[31543]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-21T23:57:53.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:53 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:54 smithi038 ceph-mon[27972]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:57:54.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:54 smithi038 ceph-mon[27972]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:57:54.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.609 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:54 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:54 smithi203 ceph-mon[31431]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:57:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:54 smithi203 ceph-mon[31431]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:57:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:54 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:54 smithi111 ceph-mon[31543]: from='osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-21T23:57:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:54 smithi111 ceph-mon[31543]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:57:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:54 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112074 s, 936 MB/s 2024-04-21T23:57:55.572 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] boot 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:57:55.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:55 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:55.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:57:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:57:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] boot 2024-04-21T23:57:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:57:55.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:55 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:55.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:57:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:57:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:57:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: osd.9 [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] boot 2024-04-21T23:57:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:57:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:55 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:57:56.244 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-04-21T23:57:56.484 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi038 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.762 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:56.763 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:56 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi038 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:56.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:56 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:57:56.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi038 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:57:56.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:56 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 -- 172.21.15.111:0/3651983339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102d20 msgr2=0x7f88c01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/3651983339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102d20 0x7f88c01095d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f88b800b0d0 tx=0x7f88b800b3e0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 -- 172.21.15.111:0/3651983339 shutdown_connections 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/3651983339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102d20 0x7f88c01095d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/3651983339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 0x7f88c01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/3651983339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102380 0x7f88c01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 -- 172.21.15.111:0/3651983339 >> 172.21.15.111:0/3651983339 conn(0x7f88c00fc720 msgr2=0x7f88c00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 -- 172.21.15.111:0/3651983339 shutdown_connections 2024-04-21T23:57:58.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 -- 172.21.15.111:0/3651983339 wait complete. 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.162+0000 7f88c5aa5700 1 Processor -- start 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 -- start start 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 0x7f88c01050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102d20 0x7f88c0103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f88c010cd70 con 0x7f88c0102d20 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f88c010cbf0 con 0x7f88c0102380 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88c5aa5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88c010cef0 con 0x7f88c0101180 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48420/0 (socket says 172.21.15.111:48420) 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 -- 172.21.15.111:0/2109594146 learned_addr learned my addr 172.21.15.111:0/2109594146 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88bffff700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102d20 0x7f88c0103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:58.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88bf7fe700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 0x7f88c01050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 -- 172.21.15.111:0/2109594146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 msgr2=0x7f88c01050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 0x7f88c01050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 -- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102d20 msgr2=0x7f88c0103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102d20 0x7f88c0103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 -- 172.21.15.111:0/2109594146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c0104570 con 0x7f88c0102380 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.163+0000 7f88b6ffd700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9a0 tx=0x7f88a800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.164+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800ca10 con 0x7f88c0102380 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.164+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c01a25e0 con 0x7f88c0102380 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.164+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a800cbb0 con 0x7f88c0102380 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.164+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8018680 con 0x7f88c0102380 2024-04-21T23:57:58.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.164+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c01a2b40 con 0x7f88c0102380 2024-04-21T23:57:58.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.165+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c0068f20 con 0x7f88c0102380 2024-04-21T23:57:58.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.166+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f88a8010430 con 0x7f88c0102380 2024-04-21T23:57:58.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.166+0000 7f88bd7fa700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 0x7f88ac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:57:58.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.166+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f88a80cea80 con 0x7f88c0102380 2024-04-21T23:57:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.166+0000 7f88bf7fe700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 0x7f88ac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:57:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.167+0000 7f88bf7fe700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 0x7f88ac0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88c01021e0 tx=0x7f88b000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:57:58.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.171+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f88a809c1e0 con 0x7f88c0102380 2024-04-21T23:57:58.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:57:58.377+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f88c00fe850 con 0x7f88ac0a2c20 2024-04-21T23:57:58.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:58 smithi038 ceph-mon[27972]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:58.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:58 smithi038 ceph-mon[27972]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:58.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:58.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:58.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:57:58 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:58.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:58 smithi111 ceph-mon[31543]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:58.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:58 smithi111 ceph-mon[31543]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:58.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:58.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:58.979 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:57:58 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:57:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:58 smithi203 ceph-mon[31431]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:57:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:58 smithi203 ceph-mon[31431]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:57:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:57:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:57:59.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:57:58 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:00 smithi038 ceph-mon[27972]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:00 smithi203 ceph-mon[31431]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:01.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:00 smithi111 ceph-mon[31543]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/1156187776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:01 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:01 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]': finished 2024-04-21T23:58:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:01 smithi038 ceph-mon[27972]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:58:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:01 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:02.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/1156187776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:02.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:01 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:02.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:01 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]': finished 2024-04-21T23:58:02.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:01 smithi203 ceph-mon[31431]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:58:02.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:01 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:02.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/1156187776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:02.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:01 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]: dispatch 2024-04-21T23:58:02.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:01 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9465f481-94aa-472a-993a-61d6a1946d0d"}]': finished 2024-04-21T23:58:02.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:01 smithi111 ceph-mon[31543]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:58:02.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:01 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:02.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:02 smithi038 ceph-mon[27972]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:02.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:02 smithi038 ceph-mon[27972]: from='client.? 172.21.15.111:0/895193708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:03.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:02 smithi203 ceph-mon[31431]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:03.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:02 smithi203 ceph-mon[31431]: from='client.? 172.21.15.111:0/895193708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:03.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:02 smithi111 ceph-mon[31543]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:03.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:02 smithi111 ceph-mon[31543]: from='client.? 172.21.15.111:0/895193708' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:04.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:04 smithi038 ceph-mon[27972]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:05.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:04 smithi203 ceph-mon[31431]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:05.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:04 smithi111 ceph-mon[31543]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:06 smithi111 ceph-mon[31543]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:06.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:06 smithi038 ceph-mon[27972]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:07.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:06 smithi203 ceph-mon[31431]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:08.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:08 smithi038 ceph-mon[27972]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:09.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:08 smithi203 ceph-mon[31431]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:09.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:08 smithi111 ceph-mon[31543]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:10.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:58:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:10.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:09 smithi203 ceph-mon[31431]: Deploying daemon osd.10 on smithi111 2024-04-21T23:58:10.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:58:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:10.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:09 smithi111 ceph-mon[31543]: Deploying daemon osd.10 on smithi111 2024-04-21T23:58:10.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:58:10.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:10.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:09 smithi038 ceph-mon[27972]: Deploying daemon osd.10 on smithi111 2024-04-21T23:58:11.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:10 smithi203 ceph-mon[31431]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:11.072 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:10 smithi111 ceph-mon[31543]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:11.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:10 smithi038 ceph-mon[27972]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:13.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:12 smithi203 ceph-mon[31431]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:13.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:12 smithi111 ceph-mon[31543]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:13.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:12 smithi038 ceph-mon[27972]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:15.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:15 smithi111 ceph-mon[31543]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:15 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:15 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:15 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:15 smithi038 ceph-mon[27972]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:15 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:15 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:15 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:15.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:15 smithi203 ceph-mon[31431]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:15.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:15 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:15 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:15.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:15 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:17.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:17 smithi111 ceph-mon[31543]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:17.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:17 smithi038 ceph-mon[27972]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:17.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:17 smithi203 ceph-mon[31431]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:17.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.947+0000 7f88bd7fa700 1 -- 172.21.15.111:0/2109594146 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f88c00fe850 con 0x7f88ac0a2c20 2024-04-21T23:58:17.941 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 10 on host 'smithi111' 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 msgr2=0x7f88ac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 0x7f88ac0a50e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88c01021e0 tx=0x7f88b000b410 comp rx=0 tx=0).stop 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 msgr2=0x7f88c0103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9a0 tx=0x7f88a800ecb0 comp rx=0 tx=0).stop 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 shutdown_connections 2024-04-21T23:58:17.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f88ac0a2c20 0x7f88ac0a50e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88c0102d20 0x7f88c0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88c0101180 0x7f88c01050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 --2- 172.21.15.111:0/2109594146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88c0102380 0x7f88c0103760 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 >> 172.21.15.111:0/2109594146 conn(0x7f88c00fc720 msgr2=0x7f88c00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 shutdown_connections 2024-04-21T23:58:17.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-21T23:58:17.950+0000 7f88c5aa5700 1 -- 172.21.15.111:0/2109594146 wait complete. 2024-04-21T23:58:18.664 DEBUG:teuthology.orchestra.run.smithi111:osd.10> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.10.service 2024-04-21T23:58:18.667 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-04-21T23:58:18.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:58:18.915 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:58:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:19.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:19.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:20.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-mon[31543]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:58:20.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-mon[31543]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:58:20.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:20.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-mon[31543]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:20.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-mon[31543]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:20.078 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-21T23:58:19.948+0000 7f3e70f7b700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T23:58:20.078 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 21 23:58:19 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-21T23:58:19.960+0000 7f3e6cd6e700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:58:20.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:19 smithi038 ceph-mon[27972]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:58:20.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:19 smithi038 ceph-mon[27972]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:58:20.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:19 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:20.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:19 smithi038 ceph-mon[27972]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:20.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:19 smithi038 ceph-mon[27972]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:20.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:19 smithi203 ceph-mon[31431]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:58:20.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:19 smithi203 ceph-mon[31431]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:58:20.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:19 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:20.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:19 smithi203 ceph-mon[31431]: from='osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:20.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:19 smithi203 ceph-mon[31431]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-21T23:58:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:20 smithi111 ceph-mon[31543]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:20 smithi111 ceph-mon[31543]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:58:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:20 smithi111 ceph-mon[31543]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:58:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.155 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:58:21.155 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:58:21.156 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-21T23:58:21.156 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-21T23:58:21.156 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127606 s, 822 MB/s 2024-04-21T23:58:21.156 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-21T23:58:21.201 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:20 smithi203 ceph-mon[31431]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:21.202 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:20 smithi203 ceph-mon[31431]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:58:21.202 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:20 smithi203 ceph-mon[31431]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:58:21.202 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.202 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:20 smithi038 ceph-mon[27972]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:21.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:20 smithi038 ceph-mon[27972]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-21T23:58:21.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:20 smithi038 ceph-mon[27972]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:58:21.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:21.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-04-21T23:58:22.182 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] boot 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:21 smithi038 ceph-mon[27972]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:58:22.222 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] boot 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.223 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:21 smithi111 ceph-mon[31543]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:58:22.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: osd.10 [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] boot 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:58:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:21 smithi203 ceph-mon[31431]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi111 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:58:23.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.331 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi111 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi111 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 4272M 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:23.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:23.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/79889644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103490 msgr2=0x7f9bb41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/79889644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103490 0x7f9bb41038f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009990 tx=0x7f9ba4009ca0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/79889644 shutdown_connections 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/79889644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103e30 0x7f9bb410a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/79889644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/79889644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103490 0x7f9bb41038f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/79889644 >> 172.21.15.203:0/79889644 conn(0x7f9bb40fd830 msgr2=0x7f9bb40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/79889644 shutdown_connections 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.902+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/79889644 wait complete. 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 Processor -- start 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 -- start start 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:23.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 0x7f9bb4198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb410def0 con 0x7f9bb4103490 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb410dd70 con 0x7f9bb4102290 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bbc1d2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb410e070 con 0x7f9bb4103e30 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44812/0 (socket says 172.21.15.203:44812) 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 -- 172.21.15.203:0/4043728806 learned_addr learned my addr 172.21.15.203:0/4043728806 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 -- 172.21.15.203:0/4043728806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 msgr2=0x7f9bb4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bb976d700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 0x7f9bb4198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bb9f6e700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 -- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 msgr2=0x7f9bb4198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:23.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 0x7f9bb4198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 -- 172.21.15.203:0/4043728806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb41a0190 con 0x7f9bb4103e30 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bb9f6e700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bb976d700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 0x7f9bb4198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9bba76f700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00c9a0 tx=0x7f9bac00cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.903+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac007ac0 con 0x7f9bb4103e30 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.904+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb41a0480 con 0x7f9bb4103e30 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.904+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bac004d10 con 0x7f9bb4103e30 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.904+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb41a0a90 con 0x7f9bb4103e30 2024-04-21T23:58:23.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.904+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac00f450 con 0x7f9bb4103e30 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.904+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb4068f20 con 0x7f9bb4103e30 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.906+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9bac004750 con 0x7f9bb4103e30 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.906+0000 7f9baaffd700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 0x7f9ba00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.906+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f9bac0ce9a0 con 0x7f9bb4103e30 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.907+0000 7f9bb9f6e700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 0x7f9ba00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:23.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.907+0000 7f9bb9f6e700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 0x7f9ba00a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bb41032f0 tx=0x7f9bb00073c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:23.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:23.912+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9bac014030 con 0x7f9bb4103e30 2024-04-21T23:58:24.110 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:24.117+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9bb4062670 con 0x7f9ba00a2c20 2024-04-21T23:58:24.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:24 smithi111 ceph-mon[31543]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:58:24.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:58:24.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:58:24.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:24 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:24.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:24 smithi038 ceph-mon[27972]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:58:24.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:58:24.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:58:24.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:24 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:24.723 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:24 smithi203 ceph-mon[31431]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:58:24.723 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:58:24.723 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:58:24.723 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:24 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:25 smithi111 ceph-mon[31543]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 156 KiB/s, 0 objects/s recovering 2024-04-21T23:58:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:25 smithi111 ceph-mon[31543]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:25.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:25 smithi111 ceph-mon[31543]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:58:25.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:25 smithi038 ceph-mon[27972]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 156 KiB/s, 0 objects/s recovering 2024-04-21T23:58:25.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:25 smithi038 ceph-mon[27972]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:25.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:25 smithi038 ceph-mon[27972]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:58:25.742 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:25 smithi203 ceph-mon[31431]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 156 KiB/s, 0 objects/s recovering 2024-04-21T23:58:25.742 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:25 smithi203 ceph-mon[31431]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:25.742 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:25 smithi203 ceph-mon[31431]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:58:25.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 21 23:58:25 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-21T23:58:25.587+0000 7f1b053f5700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-21T23:58:27.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-21T23:58:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/2903462192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]': finished 2024-04-21T23:58:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T23:58:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:27.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-21T23:58:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/2903462192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]': finished 2024-04-21T23:58:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T23:58:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: pgmap v246: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/2903462192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]: dispatch 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e605ac-1577-4fcf-861c-cc6278c07699"}]': finished 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T23:58:27.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:28.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.203:0/2072625933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:28.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.203:0/2072625933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:28.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.203:0/2072625933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:58:29.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:29 smithi111 ceph-mon[31543]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-21T23:58:29.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:29 smithi038 ceph-mon[27972]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-21T23:58:29.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:29 smithi203 ceph-mon[31431]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-21T23:58:31.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:31 smithi111 ceph-mon[31543]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:31.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:31 smithi038 ceph-mon[27972]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:31.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:31 smithi203 ceph-mon[31431]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:33.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:33 smithi203 ceph-mon[31431]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:33.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:33 smithi111 ceph-mon[31543]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:33.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:33 smithi038 ceph-mon[27972]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:35.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:35 smithi111 ceph-mon[31543]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:35.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:58:35.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:35.598 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:35 smithi203 ceph-mon[31431]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:35.598 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:58:35.598 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:35.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:35 smithi038 ceph-mon[27972]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:35.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:58:35.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:36.674 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:36 smithi203 ceph-mon[31431]: Deploying daemon osd.11 on smithi203 2024-04-21T23:58:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:36 smithi038 ceph-mon[27972]: Deploying daemon osd.11 on smithi203 2024-04-21T23:58:36.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:36 smithi111 ceph-mon[31543]: Deploying daemon osd.11 on smithi203 2024-04-21T23:58:37.478 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:37 smithi203 ceph-mon[31431]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:37 smithi038 ceph-mon[27972]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:37.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:37 smithi111 ceph-mon[31543]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:39.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:39 smithi038 ceph-mon[27972]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:39.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:39 smithi203 ceph-mon[31431]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:39.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:39 smithi111 ceph-mon[31543]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:41.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:41 smithi203 ceph-mon[31431]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:41.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:41 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:41 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:41 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:41.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:41 smithi111 ceph-mon[31543]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:41.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:41 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:41 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:41 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:41 smithi038 ceph-mon[27972]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:41 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:41 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:41 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:43.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:43 smithi203 ceph-mon[31431]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:43.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:43 smithi038 ceph-mon[27972]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:43.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:43 smithi111 ceph-mon[31543]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:44.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.442+0000 7f9baaffd700 1 -- 172.21.15.203:0/4043728806 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9bb4062670 con 0x7f9ba00a2c20 2024-04-21T23:58:44.434 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 msgr2=0x7f9ba00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 0x7f9ba00a50e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bb41032f0 tx=0x7f9bb00073c0 comp rx=0 tx=0).stop 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 msgr2=0x7f9bb419faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00c9a0 tx=0x7f9bac00cd60 comp rx=0 tx=0).stop 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 shutdown_connections 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9ba00a2c20 0x7f9ba00a50e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bb4103e30 0x7f9bb419faf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bb4102290 0x7f9bb4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:44.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 --2- 172.21.15.203:0/4043728806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9bb4103490 0x7f9bb4198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:44.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.444+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 >> 172.21.15.203:0/4043728806 conn(0x7f9bb40fd830 msgr2=0x7f9bb4106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:44.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.445+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 shutdown_connections 2024-04-21T23:58:44.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:44.445+0000 7f9bbc1d2700 1 -- 172.21.15.203:0/4043728806 wait complete. 2024-04-21T23:58:45.147 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.11.service 2024-04-21T23:58:45.149 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T23:58:45.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T23:58:45.411 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:45 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:45.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:45 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:45.852 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:45 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-mon[31431]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:58:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-mon[31431]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-mon[31431]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:58:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-mon[31431]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:46.530 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-21T23:58:46.415+0000 7f8a70243700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T23:58:46.531 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 21 23:58:46 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-21T23:58:46.435+0000 7f8a6c036700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:58:46.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:46 smithi038 ceph-mon[27972]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:58:46.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:46 smithi038 ceph-mon[27972]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:46.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:46 smithi038 ceph-mon[27972]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:58:46.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:46 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:46.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:46 smithi038 ceph-mon[27972]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:46.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:46 smithi111 ceph-mon[31543]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:58:46.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:46 smithi111 ceph-mon[31543]: from='osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:46.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:46 smithi111 ceph-mon[31543]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:58:46.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:46 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:46.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:46 smithi111 ceph-mon[31543]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-21T23:58:47.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- 172.21.15.38:0/107866993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8105670 msgr2=0x7f8ae8107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:47.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/107866993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8105670 0x7f8ae8107a60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009990 tx=0x7f8ad8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- 172.21.15.38:0/107866993 shutdown_connections 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/107866993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8107fa0 0x7f8ae810a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/107866993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae8105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/107866993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8105670 0x7f8ae8107a60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- 172.21.15.38:0/107866993 >> 172.21.15.38:0/107866993 conn(0x7f8ae80fc790 msgr2=0x7f8ae80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- 172.21.15.38:0/107866993 shutdown_connections 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- 172.21.15.38:0/107866993 wait complete. 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 Processor -- start 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.302+0000 7f8aefdb1700 1 -- start start 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae81026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8105670 0x7f8ae8100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae810dec0 con 0x7f8ae8107fa0 2024-04-21T23:58:47.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae810dd40 con 0x7f8ae8102d40 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae810e040 con 0x7f8ae8105670 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60110/0 (socket says 172.21.15.38:60110) 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 -- 172.21.15.38:0/721126213 learned_addr learned my addr 172.21.15.38:0/721126213 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aedb4d700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae81026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aed34c700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8105670 0x7f8ae8100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 -- 172.21.15.38:0/721126213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8105670 msgr2=0x7f8ae8100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8105670 0x7f8ae8100cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 -- 172.21.15.38:0/721126213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 msgr2=0x7f8ae81026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae81026a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 -- 172.21.15.38:0/721126213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae8101ae0 con 0x7f8ae8107fa0 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aedb4d700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae81026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:58:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aee34e700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00094f0 tx=0x7f8ae0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0004db0 con 0x7f8ae8107fa0 2024-04-21T23:58:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ae000db60 con 0x7f8ae8107fa0 2024-04-21T23:58:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae81a2620 con 0x7f8ae8107fa0 2024-04-21T23:58:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0015660 con 0x7f8ae8107fa0 2024-04-21T23:58:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.303+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae81a2b30 con 0x7f8ae8107fa0 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.304+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae8068f20 con 0x7f8ae8107fa0 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.305+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8ae0005020 con 0x7f8ae8107fa0 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.306+0000 7f8adeffd700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 0x7f8ad40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.306+0000 7f8aedb4d700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 0x7f8ad40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.306+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f8ae00ce620 con 0x7f8ae8107fa0 2024-04-21T23:58:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.307+0000 7f8aedb4d700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 0x7f8ad40a5160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4005950 tx=0x7f8ae4005f50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:47.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.311+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8ae0010070 con 0x7f8ae8107fa0 2024-04-21T23:58:47.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.519+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8ae8062e40 con 0x7f8ae8107fa0 2024-04-21T23:58:47.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.520+0000 7f8adeffd700 1 -- 172.21.15.38:0/721126213 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f8ae009bb40 con 0x7f8ae8107fa0 2024-04-21T23:58:47.513 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 msgr2=0x7f8ad40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 0x7f8ad40a5160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4005950 tx=0x7f8ae4005f50 comp rx=0 tx=0).stop 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 msgr2=0x7f8ae8101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00094f0 tx=0x7f8ae0007f90 comp rx=0 tx=0).stop 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 shutdown_connections 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8ad40a2ca0 0x7f8ad40a5160 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ae8107fa0 0x7f8ae8101230 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae8102d40 0x7f8ae81026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 --2- 172.21.15.38:0/721126213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ae8105670 0x7f8ae8100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 >> 172.21.15.38:0/721126213 conn(0x7f8ae80fc790 msgr2=0x7f8ae8103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 shutdown_connections 2024-04-21T23:58:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:47.522+0000 7f8aefdb1700 1 -- 172.21.15.38:0/721126213 wait complete. 2024-04-21T23:58:47.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:47 smithi203 ceph-mon[31431]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:47 smithi203 ceph-mon[31431]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:58:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:47 smithi203 ceph-mon[31431]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:58:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:47 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:47 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:47.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:47 smithi038 ceph-mon[27972]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:47.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:47 smithi038 ceph-mon[27972]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:58:47.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:47 smithi038 ceph-mon[27972]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:58:47.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:47 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:47.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:47 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:47.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:47 smithi111 ceph-mon[31543]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:47.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:47 smithi111 ceph-mon[31543]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-21T23:58:47.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:47 smithi111 ceph-mon[31543]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:58:47.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:47 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:47.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:47 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:48.311 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1713743927,"num_in_osds":12,"osd_in_since":1713743906,"num_remapped_pgs":0} 2024-04-21T23:58:48.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:58:48.565 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:58:48.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: purged_snaps scrub starts 2024-04-21T23:58:48.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: purged_snaps scrub ok 2024-04-21T23:58:48.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] boot 2024-04-21T23:58:48.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T23:58:48.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:48.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:48 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/721126213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: purged_snaps scrub starts 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: purged_snaps scrub ok 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] boot 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:48.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:48 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/721126213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:58:48.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: purged_snaps scrub starts 2024-04-21T23:58:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: purged_snaps scrub ok 2024-04-21T23:58:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: osd.11 [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] boot 2024-04-21T23:58:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T23:58:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:58:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:48 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/721126213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:49 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:49.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:49.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:49.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:49 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:49.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:49.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:49 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:50.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/3653376784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 msgr2=0x7fb5b4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:50.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/3653376784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b4106ec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009990 tx=0x7fb5a8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/3653376784 shutdown_connections 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/3653376784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41075a0 0x7fb5b410f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/3653376784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b406ac00 0x7fb5b406afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/3653376784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b4106ec0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/3653376784 >> 172.21.15.38:0/3653376784 conn(0x7fb5b4077180 msgr2=0x7fb5b4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/3653376784 shutdown_connections 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/3653376784 wait complete. 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.290+0000 7fb5b9fb9700 1 Processor -- start 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 -- start start 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b406ac00 0x7fb5b419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 0x7fb5b4198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b41125a0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b4112420 con 0x7fb5b406ac00 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b9fb9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b4112720 con 0x7fb5b41075a0 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60132/0 (socket says 172.21.15.38:60132) 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 -- 172.21.15.38:0/2997242506 learned_addr learned my addr 172.21.15.38:0/2997242506 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b37fe700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b406ac00 0x7fb5b419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b3fff700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 0x7fb5b4198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 -- 172.21.15.38:0/2997242506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 msgr2=0x7fb5b4198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 0x7fb5b4198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 -- 172.21.15.38:0/2997242506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b406ac00 msgr2=0x7fb5b419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b406ac00 0x7fb5b419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b2ffd700 1 -- 172.21.15.38:0/2997242506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b4198fe0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.291+0000 7fb5b3fff700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 0x7fb5b4198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b2ffd700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c00 tx=0x7fb5a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8027070 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b4199270 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a8003780 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8022db0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.292+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b41a54e0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.293+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b41039a0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.295+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb5a802a070 con 0x7fb5b406b5b0 2024-04-21T23:58:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.295+0000 7fb5b0ff9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 0x7fb59c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.295+0000 7fb5b37fe700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 0x7fb59c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.295+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb5a80cfc30 con 0x7fb5b406b5b0 2024-04-21T23:58:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.296+0000 7fb5b37fe700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 0x7fb59c0a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009a10 tx=0x7fb5a4009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:50.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.299+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb5a8013070 con 0x7fb5b406b5b0 2024-04-21T23:58:50.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.510+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5b402db30 con 0x7fb5b406b5b0 2024-04-21T23:58:50.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.511+0000 7fb5b0ff9700 1 -- 172.21.15.38:0/2997242506 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18441 (secure 0 0 0) 0x7fb5a809d0c0 con 0x7fb5b406b5b0 2024-04-21T23:58:50.504 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:58:50.504 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:58:48.433601+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:58:50.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 msgr2=0x7fb59c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:50.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 0x7fb59c0a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009a10 tx=0x7fb5a4009190 comp rx=0 tx=0).stop 2024-04-21T23:58:50.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 msgr2=0x7fb5b419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:50.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c00 tx=0x7fb5a80047b0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 shutdown_connections 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb59c0a2c20 0x7fb59c0a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b41075a0 0x7fb5b4198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b406ac00 0x7fb5b419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 --2- 172.21.15.38:0/2997242506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5b406b5b0 0x7fb5b419ebc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 >> 172.21.15.38:0/2997242506 conn(0x7fb5b4077180 msgr2=0x7fb5b40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 shutdown_connections 2024-04-21T23:58:50.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:50.513+0000 7fb5b9fb9700 1 -- 172.21.15.38:0/2997242506 wait complete. 2024-04-21T23:58:50.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:50 smithi038 ceph-mon[27972]: Detected new or changed devices on smithi203 2024-04-21T23:58:50.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:50 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-21T23:58:50.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:50.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:50.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:50 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:50.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:50 smithi203 ceph-mon[31431]: Detected new or changed devices on smithi203 2024-04-21T23:58:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:50 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-21T23:58:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:50 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:50.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:50 smithi111 ceph-mon[31543]: Detected new or changed devices on smithi203 2024-04-21T23:58:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:50 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-21T23:58:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:50 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:51.403 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:58:51.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T23:58:51.658 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:58:51.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:51 smithi038 ceph-mon[27972]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:51.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2997242506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:51 smithi203 ceph-mon[31431]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2997242506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:51 smithi111 ceph-mon[31543]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:51.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2997242506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 -- 172.21.15.38:0/1669706647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87681082a0 msgr2=0x7f8768108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1669706647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87681082a0 0x7f8768108680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8758009990 tx=0x7f8758009ca0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 -- 172.21.15.38:0/1669706647 shutdown_connections 2024-04-21T23:58:53.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1669706647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806b170 0x7f876810f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1669706647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87681082a0 0x7f8768108680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1669706647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876806a7d0 0x7f876806ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 -- 172.21.15.38:0/1669706647 >> 172.21.15.38:0/1669706647 conn(0x7f87680780a0 msgr2=0x7f87680784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 -- 172.21.15.38:0/1669706647 shutdown_connections 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.407+0000 7f876e6ae700 1 -- 172.21.15.38:0/1669706647 wait complete. 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 Processor -- start 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 -- start start 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806a7d0 0x7f87681011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:53.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 0x7f8768101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8768112500 con 0x7f876806b170 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8768112380 con 0x7f87681082a0 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.408+0000 7f876e6ae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8768112680 con 0x7f876806a7d0 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60148/0 (socket says 172.21.15.38:60148) 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 -- 172.21.15.38:0/1526447804 learned_addr learned my addr 172.21.15.38:0/1526447804 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:53.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f876cc4b700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 0x7f8768101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f8767fff700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806a7d0 0x7f87681011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 -- 172.21.15.38:0/1526447804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806a7d0 msgr2=0x7f87681011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806a7d0 0x7f87681011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 -- 172.21.15.38:0/1526447804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 msgr2=0x7f8768101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 0x7f8768101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 -- 172.21.15.38:0/1526447804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87681a6f30 con 0x7f876806b170 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f876cc4b700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 0x7f8768101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87677fe700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f875c00c920 tx=0x7f875c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c004a30 con 0x7f876806b170 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.409+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f875c005500 con 0x7f876806b170 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.410+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c007d50 con 0x7f876806b170 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.410+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87681a7220 con 0x7f876806b170 2024-04-21T23:58:53.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.410+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87681a7710 con 0x7f876806b170 2024-04-21T23:58:53.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.411+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8768068f20 con 0x7f876806b170 2024-04-21T23:58:53.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.412+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f875c004d60 con 0x7f876806b170 2024-04-21T23:58:53.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.413+0000 7f87657fa700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 0x7f87500a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:53.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.413+0000 7f8767fff700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 0x7f87500a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:53.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.413+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f875c0ce6a0 con 0x7f876806b170 2024-04-21T23:58:53.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.413+0000 7f8767fff700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 0x7f87500a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f875800b5c0 tx=0x7f8758005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:53.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.416+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f875c014030 con 0x7f876806b170 2024-04-21T23:58:53.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.620+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f876802db30 con 0x7f876806b170 2024-04-21T23:58:53.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.620+0000 7f87657fa700 1 -- 172.21.15.38:0/1526447804 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f875c09bb20 con 0x7f876806b170 2024-04-21T23:58:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-21T23:58:53.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 msgr2=0x7f87500a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:53.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 0x7f87500a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f875800b5c0 tx=0x7f8758005960 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 msgr2=0x7f8768101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f875c00c920 tx=0x7f875c00cc30 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 shutdown_connections 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f87500a2ca0 0x7f87500a5160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87681082a0 0x7f8768101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f876806a7d0 0x7f87681011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.622+0000 7f876e6ae700 1 --2- 172.21.15.38:0/1526447804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f876806b170 0x7f8768101730 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.623+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 >> 172.21.15.38:0/1526447804 conn(0x7f87680780a0 msgr2=0x7f87680ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.623+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 shutdown_connections 2024-04-21T23:58:53.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:53.623+0000 7f876e6ae700 1 -- 172.21.15.38:0/1526447804 wait complete. 2024-04-21T23:58:53.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:53 smithi038 ceph-mon[27972]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:53 smithi203 ceph-mon[31431]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:53 smithi111 ceph-mon[31543]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:54.409 INFO:tasks.cephadm:Adding mds.a on smithi038 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.d on smithi038 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.g on smithi038 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.j on smithi038 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.b on smithi111 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.e on smithi111 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.h on smithi111 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.k on smithi111 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-04-21T23:58:54.410 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-04-21T23:58:54.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-04-21T23:58:54.663 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:58:54.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:54 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1526447804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:54.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:54 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1526447804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:54.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:54 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1526447804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:55.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:55 smithi203 ceph-mon[31431]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:55.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:55 smithi111 ceph-mon[31543]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:55.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:55 smithi038 ceph-mon[27972]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:56.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 -- 172.21.15.203:0/1843709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02781023a0 msgr2=0x7f0278102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 --2- 172.21.15.203:0/1843709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02781023a0 0x7f0278102800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0268009990 tx=0x7f0268009ca0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 -- 172.21.15.203:0/1843709026 shutdown_connections 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 --2- 172.21.15.203:0/1843709026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0278102d40 0x7f02781095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 --2- 172.21.15.203:0/1843709026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781011a0 0x7f02781015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 --2- 172.21.15.203:0/1843709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02781023a0 0x7f0278102800 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 -- 172.21.15.203:0/1843709026 >> 172.21.15.203:0/1843709026 conn(0x7f02780fc760 msgr2=0x7f02780feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 -- 172.21.15.203:0/1843709026 shutdown_connections 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 -- 172.21.15.203:0/1843709026 wait complete. 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.372+0000 7f027ee14700 1 Processor -- start 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 -- start start 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02781011a0 0x7f0278198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:56.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 0x7f0278198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f027810cd90 con 0x7f02781023a0 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f027810cc10 con 0x7f02781011a0 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f027810cf10 con 0x7f0278102d40 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:56098/0 (socket says 172.21.15.203:56098) 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 -- 172.21.15.203:0/2867714528 learned_addr learned my addr 172.21.15.203:0/2867714528 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f026ffff700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 0x7f0278198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027cbb0700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02781011a0 0x7f0278198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 -- 172.21.15.203:0/2867714528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02781011a0 msgr2=0x7f0278198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02781011a0 0x7f0278198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 -- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 msgr2=0x7f0278198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:56.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 0x7f0278198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 -- 172.21.15.203:0/2867714528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02781a0180 con 0x7f0278102d40 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f026ffff700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 0x7f0278198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027d3b1700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0270007f10 tx=0x7f027000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270004bb0 con 0x7f0278102d40 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.373+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02781a0470 con 0x7f0278102d40 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.374+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0270004d50 con 0x7f0278102d40 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.374+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270005a60 con 0x7f0278102d40 2024-04-21T23:58:56.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.374+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02781a0960 con 0x7f0278102d40 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.375+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0278068f20 con 0x7f0278102d40 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.376+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f027001a020 con 0x7f0278102d40 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.376+0000 7f026dffb700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 0x7f02600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.376+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02700ce460 con 0x7f0278102d40 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.376+0000 7f027cbb0700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 0x7f02600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:56.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.377+0000 7f027cbb0700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 0x7f02600a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0274005950 tx=0x7f02740058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:56.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.381+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0270010070 con 0x7f0278102d40 2024-04-21T23:58:56.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.590+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f02780fd640 con 0x7f02600a2c20 2024-04-21T23:58:56.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.600+0000 7f026dffb700 1 -- 172.21.15.203:0/2867714528 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f02780fd640 con 0x7f02600a2c20 2024-04-21T23:58:56.592 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-04-21T23:58:56.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 msgr2=0x7f02600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 0x7f02600a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0274005950 tx=0x7f02740058e0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 msgr2=0x7f027819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0270007f10 tx=0x7f027000d3b0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 shutdown_connections 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02600a2c20 0x7f02600a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0278102d40 0x7f027819fae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02781011a0 0x7f0278198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 --2- 172.21.15.203:0/2867714528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02781023a0 0x7f0278198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 >> 172.21.15.203:0/2867714528 conn(0x7f02780fc760 msgr2=0x7f02780fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 shutdown_connections 2024-04-21T23:58:56.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-21T23:58:56.602+0000 7f027ee14700 1 -- 172.21.15.203:0/2867714528 wait complete. 2024-04-21T23:58:57.240 DEBUG:teuthology.orchestra.run.smithi038:mds.a> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.a.service 2024-04-21T23:58:57.243 DEBUG:teuthology.orchestra.run.smithi038:mds.d> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.d.service 2024-04-21T23:58:57.286 DEBUG:teuthology.orchestra.run.smithi038:mds.g> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.g.service 2024-04-21T23:58:57.328 DEBUG:teuthology.orchestra.run.smithi038:mds.j> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.j.service 2024-04-21T23:58:57.372 DEBUG:teuthology.orchestra.run.smithi111:mds.b> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.b.service 2024-04-21T23:58:57.374 DEBUG:teuthology.orchestra.run.smithi111:mds.e> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.e.service 2024-04-21T23:58:57.417 DEBUG:teuthology.orchestra.run.smithi111:mds.h> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.h.service 2024-04-21T23:58:57.460 DEBUG:teuthology.orchestra.run.smithi111:mds.k> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.k.service 2024-04-21T23:58:57.503 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.c.service 2024-04-21T23:58:57.506 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.f.service 2024-04-21T23:58:57.549 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.i.service 2024-04-21T23:58:57.592 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.l.service 2024-04-21T23:58:57.636 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T23:58:57.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:58:57.661 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:57 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:58:57.735 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:57 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 -- 172.21.15.38:0/2557660223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc264075780 msgr2=0x7fc264073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 --2- 172.21.15.38:0/2557660223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc264075780 0x7fc264073bd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc260009900 tx=0x7fc26002bc30 comp rx=0 tx=0).stop 2024-04-21T23:58:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 -- 172.21.15.38:0/2557660223 shutdown_connections 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 --2- 172.21.15.38:0/2557660223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264109650 0x7fc26410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 --2- 172.21.15.38:0/2557660223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2640741a0 0x7fc264074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 --2- 172.21.15.38:0/2557660223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc264075780 0x7fc264073bd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 -- 172.21.15.38:0/2557660223 >> 172.21.15.38:0/2557660223 conn(0x7fc2640fcd20 msgr2=0x7fc2640ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 -- 172.21.15.38:0/2557660223 shutdown_connections 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.770+0000 7fc26bb55640 1 -- 172.21.15.38:0/2557660223 wait complete. 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 Processor -- start 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 -- start start 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264075780 0x7fc26419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 0x7fc2641a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc26410f5f0 con 0x7fc2640741a0 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc26410f470 con 0x7fc264109650 2024-04-21T23:58:57.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26bb55640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc26410f770 con 0x7fc264075780 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55060/0 (socket says 172.21.15.38:55060) 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 -- 172.21.15.38:0/3288346580 learned_addr learned my addr 172.21.15.38:0/3288346580 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2690f0640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264075780 0x7fc26419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26a0f2640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 0x7fc2641a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 -- 172.21.15.38:0/3288346580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264075780 msgr2=0x7fc26419a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264075780 0x7fc26419a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 -- 172.21.15.38:0/3288346580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 msgr2=0x7fc2641a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 0x7fc2641a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc2698f1640 1 -- 172.21.15.38:0/3288346580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2641a18e0 con 0x7fc2640741a0 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.771+0000 7fc26a0f2640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 0x7fc2641a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc2698f1640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc260009230 tx=0x7fc260005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc260041070 con 0x7fc2640741a0 2024-04-21T23:58:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2600044d0 con 0x7fc2640741a0 2024-04-21T23:58:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2641a1b70 con 0x7fc2640741a0 2024-04-21T23:58:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc260034440 con 0x7fc2640741a0 2024-04-21T23:58:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.772+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2641a2060 con 0x7fc2640741a0 2024-04-21T23:58:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.773+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc264108380 con 0x7fc2640741a0 2024-04-21T23:58:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.774+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc260044070 con 0x7fc2640741a0 2024-04-21T23:58:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.774+0000 7fc25affd640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 0x7fc2500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.775+0000 7fc2690f0640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 0x7fc2500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.775+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc2600f08c0 con 0x7fc2640741a0 2024-04-21T23:58:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.775+0000 7fc2690f0640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 0x7fc2500a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc26419b1a0 tx=0x7fc25400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:57.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.779+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc2600bdc20 con 0x7fc2640741a0 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:58:57.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:57 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:58:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.965+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2640623e0 con 0x7fc2640741a0 2024-04-21T23:58:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.966+0000 7fc25affd640 1 -- 172.21.15.38:0/3288346580 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18441 (secure 0 0 0) 0x7fc2600bd520 con 0x7fc2640741a0 2024-04-21T23:58:57.959 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:58:57.959 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:58:48.433601+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 msgr2=0x7fc2500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 0x7fc2500a5050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc26419b1a0 tx=0x7fc25400b390 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 msgr2=0x7fc264199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc260009230 tx=0x7fc260005d50 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 shutdown_connections 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc2500a2b90 0x7fc2500a5050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc264109650 0x7fc2641a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc264075780 0x7fc26419a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 --2- 172.21.15.38:0/3288346580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2640741a0 0x7fc264199ca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 >> 172.21.15.38:0/3288346580 conn(0x7fc2640fcd20 msgr2=0x7fc264105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 shutdown_connections 2024-04-21T23:58:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:57.968+0000 7fc26bb55640 1 -- 172.21.15.38:0/3288346580 wait complete. 2024-04-21T23:58:57.972 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:58:57.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:58:58.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.123+0000 7fa4b8713640 1 -- 172.21.15.38:0/751839475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 msgr2=0x7fa4b0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.123+0000 7fa4b8713640 1 --2- 172.21.15.38:0/751839475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0108ac0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009ca0 tx=0x7fa4a802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- 172.21.15.38:0/751839475 shutdown_connections 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- 172.21.15.38:0/751839475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0108ac0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- 172.21.15.38:0/751839475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 0x7fa4b0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- 172.21.15.38:0/751839475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 0x7fa4b0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- 172.21.15.38:0/751839475 >> 172.21.15.38:0/751839475 conn(0x7fa4b00fbbf0 msgr2=0x7fa4b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- 172.21.15.38:0/751839475 shutdown_connections 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- 172.21.15.38:0/751839475 wait complete. 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 Processor -- start 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- start start 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 0x7fa4b0104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 0x7fa4b0102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c2e0 con 0x7fa4b0102230 2024-04-21T23:58:58.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c160 con 0x7fa4b0100680 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b8713640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b010c460 con 0x7fa4b0101870 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.124+0000 7fa4b6cb0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55066/0 (socket says 172.21.15.38:55066) 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 -- 172.21.15.38:0/1983094722 learned_addr learned my addr 172.21.15.38:0/1983094722 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b5cae640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 0x7fa4b0102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b64af640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 0x7fa4b0104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 -- 172.21.15.38:0/1983094722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 msgr2=0x7fa4b0102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 0x7fa4b0102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 -- 172.21.15.38:0/1983094722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 msgr2=0x7fa4b0104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 0x7fa4b0104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 -- 172.21.15.38:0/1983094722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b0103a20 con 0x7fa4b0102230 2024-04-21T23:58:58.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b6cb0640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8032530 tx=0x7fa4a8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:58.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8041020 con 0x7fa4b0102230 2024-04-21T23:58:58.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a8034cd0 con 0x7fa4b0102230 2024-04-21T23:58:58.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b01a1bd0 con 0x7fa4b0102230 2024-04-21T23:58:58.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8039040 con 0x7fa4b0102230 2024-04-21T23:58:58.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.125+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b01a20f0 con 0x7fa4b0102230 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.126+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b01a22c0 con 0x7fa4b0102230 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.127+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa4a8044020 con 0x7fa4b0102230 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.128+0000 7fa4a77fe640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 0x7fa49c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.128+0000 7fa4b64af640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 0x7fa49c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.128+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa4a80f0a70 con 0x7fa4b0102230 2024-04-21T23:58:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.128+0000 7fa4b64af640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 0x7fa49c0a5050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4b01016d0 tx=0x7fa4ac005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:58.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.132+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa4a804b020 con 0x7fa4b0102230 2024-04-21T23:58:58.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.319+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4b01a2490 con 0x7fa4b0102230 2024-04-21T23:58:58.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.319+0000 7fa4a77fe640 1 -- 172.21.15.38:0/1983094722 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fa4a80bde60 con 0x7fa4b0102230 2024-04-21T23:58:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-21T23:58:58.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 msgr2=0x7fa49c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 0x7fa49c0a5050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4b01016d0 tx=0x7fa4ac005f90 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 msgr2=0x7fa4b0103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8032530 tx=0x7fa4a8032f40 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 shutdown_connections 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa49c0a2b90 0x7fa49c0a5050 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4b0102230 0x7fa4b0103190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b0101870 0x7fa4b0102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 --2- 172.21.15.38:0/1983094722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4b0100680 0x7fa4b0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.321+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 >> 172.21.15.38:0/1983094722 conn(0x7fa4b00fbbf0 msgr2=0x7fa4b00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.322+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 shutdown_connections 2024-04-21T23:58:58.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.322+0000 7fa4b8713640 1 -- 172.21.15.38:0/1983094722 wait complete. 2024-04-21T23:58:58.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:58:58.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 -- 172.21.15.38:0/2408271064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938102230 msgr2=0x7f2938108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 --2- 172.21.15.38:0/2408271064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938102230 0x7f2938108ac0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f293002bc30 comp rx=0 tx=0).stop 2024-04-21T23:58:58.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 -- 172.21.15.38:0/2408271064 shutdown_connections 2024-04-21T23:58:58.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 --2- 172.21.15.38:0/2408271064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938102230 0x7f2938108ac0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 --2- 172.21.15.38:0/2408271064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938101870 0x7f2938101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 --2- 172.21.15.38:0/2408271064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 0x7f2938100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 -- 172.21.15.38:0/2408271064 >> 172.21.15.38:0/2408271064 conn(0x7f29380fbbf0 msgr2=0x7f29380fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 -- 172.21.15.38:0/2408271064 shutdown_connections 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.434+0000 7f293d94f640 1 -- 172.21.15.38:0/2408271064 wait complete. 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 Processor -- start 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 -- start start 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 0x7f2938196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938102230 0x7f293819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f293810c350 con 0x7f2938101870 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f293810c1d0 con 0x7f2938100680 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f293d94f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f293810c4d0 con 0x7f2938102230 2024-04-21T23:58:58.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f29367fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f29367fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55080/0 (socket says 172.21.15.38:55080) 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f29367fc640 1 -- 172.21.15.38:0/4107310852 learned_addr learned my addr 172.21.15.38:0/4107310852 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.435+0000 7f29377fe640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938102230 0x7f293819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f2936ffd640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 0x7f2938196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 -- 172.21.15.38:0/4107310852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938102230 msgr2=0x7f293819dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938102230 0x7f293819dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 -- 172.21.15.38:0/4107310852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 msgr2=0x7f2938196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 0x7f2938196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 -- 172.21.15.38:0/4107310852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293819e550 con 0x7f2938101870 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f29367fc640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f292000e9f0 tx=0x7f292000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292000cd80 con 0x7f2938101870 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293819e840 con 0x7f2938101870 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2920005500 con 0x7f2938101870 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920010690 con 0x7f2938101870 2024-04-21T23:58:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.436+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293819ed40 con 0x7f2938101870 2024-04-21T23:58:58.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.437+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2938107940 con 0x7f2938101870 2024-04-21T23:58:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.438+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f29200046c0 con 0x7f2938101870 2024-04-21T23:58:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.439+0000 7f293c94d640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 0x7f29240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:58:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.439+0000 7f2936ffd640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 0x7f29240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:58:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.439+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29200cf020 con 0x7f2938101870 2024-04-21T23:58:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.439+0000 7f2936ffd640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 0x7f29240a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29381016d0 tx=0x7f292800a590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:58:58.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.443+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f292009c380 con 0x7f2938101870 2024-04-21T23:58:58.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.668+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2938062340 con 0x7f2938101870 2024-04-21T23:58:58.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.668+0000 7f293c94d640 1 -- 172.21.15.38:0/4107310852 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f292009bc80 con 0x7f2938101870 2024-04-21T23:58:58.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:58:58.661 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T23:58:58.662 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 1 2024-04-21T23:58:58.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 msgr2=0x7f29240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 0x7f29240a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29381016d0 tx=0x7f292800a590 comp rx=0 tx=0).stop 2024-04-21T23:58:58.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 msgr2=0x7f2938196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:58:58.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f292000e9f0 tx=0x7f292000edf0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 shutdown_connections 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f29240a2b90 0x7f29240a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2938102230 0x7f293819dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2938101870 0x7f2938196e70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 --2- 172.21.15.38:0/4107310852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2938100680 0x7f2938196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 >> 172.21.15.38:0/4107310852 conn(0x7f29380fbbf0 msgr2=0x7f2938104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 shutdown_connections 2024-04-21T23:58:58.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:58:58.670+0000 7f293d94f640 1 -- 172.21.15.38:0/4107310852 wait complete. 2024-04-21T23:58:58.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:58 smithi038 ceph-mon[27972]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:58.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:58 smithi038 ceph-mon[27972]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-21T23:58:58.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:58 smithi038 ceph-mon[27972]: Deploying daemon mds.l on smithi203 2024-04-21T23:58:58.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3288346580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:58.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1983094722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:58.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:58 smithi203 ceph-mon[31431]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:58.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:58 smithi203 ceph-mon[31431]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-21T23:58:58.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:58 smithi203 ceph-mon[31431]: Deploying daemon mds.l on smithi203 2024-04-21T23:58:58.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3288346580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:58.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1983094722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:58 smithi111 ceph-mon[31543]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:58:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:58 smithi111 ceph-mon[31543]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-21T23:58:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:58 smithi111 ceph-mon[31543]: Deploying daemon mds.l on smithi203 2024-04-21T23:58:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3288346580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:58:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1983094722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:58:59.176 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:58 smithi203 systemd[1]: Starting Ceph mds.l for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:58:59.438 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 podman[61701]: 2024-04-21 23:58:59.184304358 +0000 UTC m=+0.110096245 container create ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-21T23:58:59.438 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 podman[61701]: 2024-04-21 23:58:59.092155807 +0000 UTC m=+0.017947695 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:58:59.438 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 podman[61701]: 2024-04-21 23:58:59.420399368 +0000 UTC m=+0.346191257 container init ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-21T23:58:59.438 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 podman[61701]: 2024-04-21 23:58:59.422976553 +0000 UTC m=+0.348768454 container start ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:58:59.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mon[31431]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4107310852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6832/1279747281,v1:172.21.15.203:6833/1279747281] up:boot 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mon[31431]: fsmap 1 up:standby 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-mds[61720]: -- 172.21.15.203:0/4064033853 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d1c53da40 con 0x558d1c510c00 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: starting mds.l at 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 bash[61701]: ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 2024-04-21T23:58:59.780 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 21 23:58:59 smithi203 systemd[1]: Started Ceph mds.l for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:58:59.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:59 smithi111 ceph-mon[31543]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:59 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4107310852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:58:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:59 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6832/1279747281,v1:172.21.15.203:6833/1279747281] up:boot 2024-04-21T23:58:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:59 smithi111 ceph-mon[31543]: fsmap 1 up:standby 2024-04-21T23:58:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:58:59 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:58:59.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:59 smithi038 ceph-mon[27972]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:58:59.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:59 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4107310852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:58:59.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:59 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6832/1279747281,v1:172.21.15.203:6833/1279747281] up:boot 2024-04-21T23:58:59.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:59 smithi038 ceph-mon[27972]: fsmap 1 up:standby 2024-04-21T23:58:59.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:58:59 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:59:00.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:00.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 -- 172.21.15.38:0/3086721952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 msgr2=0x7fd094108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:00.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 --2- 172.21.15.38:0/3086721952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd094108ae0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ca0 tx=0x7fd08c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 -- 172.21.15.38:0/3086721952 shutdown_connections 2024-04-21T23:59:00.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 --2- 172.21.15.38:0/3086721952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd094108ae0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 --2- 172.21.15.38:0/3086721952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd094101890 0x7fd094101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 --2- 172.21.15.38:0/3086721952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0941006a0 0x7fd094100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 -- 172.21.15.38:0/3086721952 >> 172.21.15.38:0/3086721952 conn(0x7fd0940fbc30 msgr2=0x7fd0940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 -- 172.21.15.38:0/3086721952 shutdown_connections 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.798+0000 7fd09abde640 1 -- 172.21.15.38:0/3086721952 wait complete. 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 Processor -- start 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 -- start start 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 0x7fd094197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd094101890 0x7fd094197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:00.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd09410d440 con 0x7fd094102250 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd09410d2c0 con 0x7fd0941006a0 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09abde640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd09410d5c0 con 0x7fd094101890 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55090/0 (socket says 172.21.15.38:55090) 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 -- 172.21.15.38:0/3818665384 learned_addr learned my addr 172.21.15.38:0/3818665384 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09897a640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 0x7fd094197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd093fff640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd094101890 0x7fd094197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 -- 172.21.15.38:0/3818665384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd094101890 msgr2=0x7fd094197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd094101890 0x7fd094197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 -- 172.21.15.38:0/3818665384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 msgr2=0x7fd094197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 0x7fd094197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 -- 172.21.15.38:0/3818665384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09419f630 con 0x7fd094102250 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09897a640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 0x7fd094197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:00.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.799+0000 7fd09917b640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c700 tx=0x7fd08c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:00.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.800+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c039040 con 0x7fd094102250 2024-04-21T23:59:00.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.800+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08c044070 con 0x7fd094102250 2024-04-21T23:59:00.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.800+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09419f8c0 con 0x7fd094102250 2024-04-21T23:59:00.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.800+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0042a0 con 0x7fd094102250 2024-04-21T23:59:00.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.800+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09419fdb0 con 0x7fd094102250 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.801+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd094107a60 con 0x7fd094102250 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.802+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd08c003970 con 0x7fd094102250 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.803+0000 7fd091ffb640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 0x7fd07c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.803+0000 7fd09897a640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 0x7fd07c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.803+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd08c0f1220 con 0x7fd094102250 2024-04-21T23:59:00.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.803+0000 7fd09897a640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 0x7fd07c0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0840058c0 tx=0x7fd084005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:00.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:00.807+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd08c0be610 con 0x7fd094102250 2024-04-21T23:59:00.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: Deploying daemon mds.k on smithi111 2024-04-21T23:59:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:00 smithi111 ceph-mon[31543]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: Deploying daemon mds.k on smithi111 2024-04-21T23:59:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:00 smithi038 ceph-mon[27972]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:01.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.031+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd09402d170 con 0x7fd094102250 2024-04-21T23:59:01.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.031+0000 7fd091ffb640 1 -- 172.21.15.38:0/3818665384 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fd08c0bdf10 con 0x7fd094102250 2024-04-21T23:59:01.024 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:01.024 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:59:01.025 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2024-04-21T23:59:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 msgr2=0x7fd07c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 0x7fd07c0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0840058c0 tx=0x7fd084005850 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 msgr2=0x7fd09419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c700 tx=0x7fd08c032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 shutdown_connections 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd07c0a2b90 0x7fd07c0a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd094102250 0x7fd09419efc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd094101890 0x7fd094197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 --2- 172.21.15.38:0/3818665384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0941006a0 0x7fd094197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 >> 172.21.15.38:0/3818665384 conn(0x7fd0940fbc30 msgr2=0x7fd094104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 shutdown_connections 2024-04-21T23:59:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:01.033+0000 7fd09abde640 1 -- 172.21.15.38:0/3818665384 wait complete. 2024-04-21T23:59:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:01.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:01.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:01.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:01.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: Deploying daemon mds.k on smithi111 2024-04-21T23:59:01.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:00 smithi203 ceph-mon[31431]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:01.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3818665384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:02.021 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3818665384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:02.021 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:01 smithi111 systemd[1]: Starting Ceph mds.k for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:02.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3818665384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:02.293 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 podman[60647]: 2024-04-21 23:59:01.93126215 +0000 UTC m=+0.017774503 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:02.293 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 podman[60647]: 2024-04-21 23:59:02.047174933 +0000 UTC m=+0.133687321 container create 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-21T23:59:02.294 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 podman[60647]: 2024-04-21 23:59:02.27485312 +0000 UTC m=+0.361365467 container init 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:59:02.294 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 podman[60647]: 2024-04-21 23:59:02.277441444 +0000 UTC m=+0.363953787 container start 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T23:59:02.294 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 ceph-mds[60666]: -- 172.21.15.111:0/2177087582 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56256fc97a40 con 0x56256fc6ac00 2024-04-21T23:59:02.576 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: starting mds.k at 2024-04-21T23:59:02.577 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 bash[60647]: 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 2024-04-21T23:59:02.577 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 21 23:59:02 smithi111 systemd[1]: Started Ceph mds.k for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:03.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:03.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 -- 172.21.15.38:0/2369926944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 msgr2=0x7f2798108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:03.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 --2- 172.21.15.38:0/2369926944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f2798108ae0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2790009ca0 tx=0x7f279002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 -- 172.21.15.38:0/2369926944 shutdown_connections 2024-04-21T23:59:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 --2- 172.21.15.38:0/2369926944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f2798108ae0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 --2- 172.21.15.38:0/2369926944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 --2- 172.21.15.38:0/2369926944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 0x7f2798100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 -- 172.21.15.38:0/2369926944 >> 172.21.15.38:0/2369926944 conn(0x7f27980fbc30 msgr2=0x7f27980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 -- 172.21.15.38:0/2369926944 shutdown_connections 2024-04-21T23:59:03.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.148+0000 7f279dc0e640 1 -- 172.21.15.38:0/2369926944 wait complete. 2024-04-21T23:59:03.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 Processor -- start 2024-04-21T23:59:03.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 -- start start 2024-04-21T23:59:03.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 0x7f2798104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:03.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:03.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:03.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f279810c320 con 0x7f2798102250 2024-04-21T23:59:03.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f279810c1a0 con 0x7f27981006a0 2024-04-21T23:59:03.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f279dc0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f279810c4a0 con 0x7f2798101890 2024-04-21T23:59:03.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f2797fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:03.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f2797fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55118/0 (socket says 172.21.15.38:55118) 2024-04-21T23:59:03.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.149+0000 7f2797fff640 1 -- 172.21.15.38:0/202237150 learned_addr learned my addr 172.21.15.38:0/202237150 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:03.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 -- 172.21.15.38:0/202237150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 msgr2=0x7f2798102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:03.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2796ffd640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:03.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f27977fe640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 0x7f2798104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:03.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 -- 172.21.15.38:0/202237150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 msgr2=0x7f2798104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:03.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 0x7f2798104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 -- 172.21.15.38:0/202237150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798103a40 con 0x7f2798102250 2024-04-21T23:59:03.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2796ffd640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:03.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2797fff640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f279002c700 tx=0x7f2790032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:03.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790039040 con 0x7f2798102250 2024-04-21T23:59:03.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2790044070 con 0x7f2798102250 2024-04-21T23:59:03.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27981a1b10 con 0x7f2798102250 2024-04-21T23:59:03.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27900042a0 con 0x7f2798102250 2024-04-21T23:59:03.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.150+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981a20b0 con 0x7f2798102250 2024-04-21T23:59:03.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.151+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27981a2280 con 0x7f2798102250 2024-04-21T23:59:03.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.152+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2790003970 con 0x7f2798102250 2024-04-21T23:59:03.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.153+0000 7f2794ff9640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 0x7f27800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:03.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.153+0000 7f27977fe640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 0x7f27800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:03.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.153+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27900f1190 con 0x7f2798102250 2024-04-21T23:59:03.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.154+0000 7f27977fe640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 0x7f27800a50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2788005fd0 tx=0x7f2788005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:03.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.158+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f27900333a0 con 0x7f2798102250 2024-04-21T23:59:03.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.383+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27981a2450 con 0x7f2798102250 2024-04-21T23:59:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.384+0000 7f2794ff9640 1 -- 172.21.15.38:0/202237150 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f27900be580 con 0x7f2798102250 2024-04-21T23:59:03.377 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:03.377 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:59:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 3 2024-04-21T23:59:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 msgr2=0x7f27800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 0x7f27800a50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2788005fd0 tx=0x7f2788005850 comp rx=0 tx=0).stop 2024-04-21T23:59:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 msgr2=0x7f27981031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f279002c700 tx=0x7f2790032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 shutdown_connections 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f27800a2c10 0x7f27800a50d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2798102250 0x7f27981031b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2798101890 0x7f2798102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 --2- 172.21.15.38:0/202237150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27981006a0 0x7f2798104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 >> 172.21.15.38:0/202237150 conn(0x7f27980fbc30 msgr2=0x7f27980fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.386+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 shutdown_connections 2024-04-21T23:59:03.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:03.387+0000 7f279dc0e640 1 -- 172.21.15.38:0/202237150 wait complete. 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:03.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6832/618180230,v1:172.21.15.111:6833/618180230] up:boot 2024-04-21T23:59:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: fsmap 2 up:standby 2024-04-21T23:59:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:59:03.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:03 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/202237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:03.726 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:03.726 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.726 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.726 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.726 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6832/618180230,v1:172.21.15.111:6833/618180230] up:boot 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: fsmap 2 up:standby 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:59:03.727 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:03 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/202237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:03.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6832/618180230,v1:172.21.15.111:6833/618180230] up:boot 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: fsmap 2 up:standby 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:59:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:03 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/202237150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:04.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:04 smithi203 ceph-mon[31431]: Deploying daemon mds.j on smithi038 2024-04-21T23:59:05.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:05.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:04 smithi038 ceph-mon[27972]: Deploying daemon mds.j on smithi038 2024-04-21T23:59:05.705 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:04 smithi111 ceph-mon[31543]: Deploying daemon mds.j on smithi038 2024-04-21T23:59:05.706 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:04 smithi038 systemd[1]: Starting Ceph mds.j for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:04 smithi038 podman[68523]: 2024-04-21 23:59:04.924207915 +0000 UTC m=+0.111708570 container create 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:04 smithi038 podman[68523]: 2024-04-21 23:59:04.840861602 +0000 UTC m=+0.028362270 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 podman[68523]: 2024-04-21 23:59:05.160519618 +0000 UTC m=+0.348020274 container init 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-820a2f8) 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 podman[68523]: 2024-04-21 23:59:05.163210942 +0000 UTC m=+0.350711598 container start 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mds[68543]: -- 172.21.15.38:0/3181040341 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56493f311a40 con 0x56493f2e4c00 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j[68539]: starting mds.j at 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 bash[68523]: 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e 2024-04-21T23:59:05.707 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 21 23:59:05 smithi038 systemd[1]: Started Ceph mds.j for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:05.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:05.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:05 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:05.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:05.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:05 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 -- 172.21.15.38:0/1402952021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264101870 msgr2=0x7f0264101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 --2- 172.21.15.38:0/1402952021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264101870 0x7f0264101cf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f024c009900 tx=0x7f024c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 -- 172.21.15.38:0/1402952021 shutdown_connections 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 --2- 172.21.15.38:0/1402952021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f0264108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 --2- 172.21.15.38:0/1402952021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264101870 0x7f0264101cf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 --2- 172.21.15.38:0/1402952021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264100680 0x7f0264100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 -- 172.21.15.38:0/1402952021 >> 172.21.15.38:0/1402952021 conn(0x7f02640fbbf0 msgr2=0x7f02640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 -- 172.21.15.38:0/1402952021 shutdown_connections 2024-04-21T23:59:05.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.823+0000 7f0269a88640 1 -- 172.21.15.38:0/1402952021 wait complete. 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 Processor -- start 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 -- start start 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264101870 0x7f0264196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f02640688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f026410c300 con 0x7f0264100680 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f026410c180 con 0x7f0264101870 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0269a88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f026410c480 con 0x7f0264102230 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f02637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0263fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f02640688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f02637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56194/0 (socket says 172.21.15.38:56194) 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f0263fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f02640688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51966/0 (socket says 172.21.15.38:51966) 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.824+0000 7f02637fe640 1 -- 172.21.15.38:0/158594709 learned_addr learned my addr 172.21.15.38:0/158594709 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f025bfff640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264101870 0x7f0264196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 -- 172.21.15.38:0/158594709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 msgr2=0x7f02640688d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f02640688d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 -- 172.21.15.38:0/158594709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264101870 msgr2=0x7f0264196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264101870 0x7f0264196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 -- 172.21.15.38:0/158594709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0264068f40 con 0x7f0264100680 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02637fe640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f025400e9c0 tx=0x7f025400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025400cd80 con 0x7f0264100680 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0264069230 con 0x7f0264100680 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02540035f0 con 0x7f0264100680 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254010690 con 0x7f0264100680 2024-04-21T23:59:05.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.825+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02640697a0 con 0x7f0264100680 2024-04-21T23:59:05.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.826+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0264068560 con 0x7f0264100680 2024-04-21T23:59:05.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.827+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f02540037a0 con 0x7f0264100680 2024-04-21T23:59:05.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.828+0000 7f02617fa640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 0x7f02500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:05.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.828+0000 7f025bfff640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 0x7f02500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:05.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.828+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02540cef70 con 0x7f0264100680 2024-04-21T23:59:05.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.829+0000 7f025bfff640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 0x7f02500a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f024c02c7b0 tx=0x7f024c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:05.832+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f025409c370 con 0x7f0264100680 2024-04-21T23:59:05.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:05.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:05 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.066+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0264062340 con 0x7f0264100680 2024-04-21T23:59:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.067+0000 7f02617fa640 1 -- 172.21.15.38:0/158594709 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f025409bc70 con 0x7f0264100680 2024-04-21T23:59:06.060 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:06.060 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:59:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 msgr2=0x7f02500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 0x7f02500a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f024c02c7b0 tx=0x7f024c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 msgr2=0x7f02641969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f025400e9c0 tx=0x7f025400ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 shutdown_connections 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02500a2b90 0x7f02500a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0264102230 0x7f02640688d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0264101870 0x7f0264196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:06.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 --2- 172.21.15.38:0/158594709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0264100680 0x7f02641969a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:06.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 >> 172.21.15.38:0/158594709 conn(0x7f02640fbbf0 msgr2=0x7f0264104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:06.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 shutdown_connections 2024-04-21T23:59:06.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:06.069+0000 7f0269a88640 1 -- 172.21.15.38:0/158594709 wait complete. 2024-04-21T23:59:06.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:06 smithi038 ceph-mon[27972]: Deploying daemon mds.d on smithi038 2024-04-21T23:59:06.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:06 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6834/425693237,v1:172.21.15.38:6835/425693237] up:boot 2024-04-21T23:59:06.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:06 smithi038 ceph-mon[27972]: fsmap 3 up:standby 2024-04-21T23:59:06.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:06 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:59:06.603 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:06 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/158594709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:06.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:06 smithi203 ceph-mon[31431]: Deploying daemon mds.d on smithi038 2024-04-21T23:59:06.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:06 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6834/425693237,v1:172.21.15.38:6835/425693237] up:boot 2024-04-21T23:59:06.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:06 smithi203 ceph-mon[31431]: fsmap 3 up:standby 2024-04-21T23:59:06.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:06 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:59:06.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:06 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/158594709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:06 smithi111 ceph-mon[31543]: Deploying daemon mds.d on smithi038 2024-04-21T23:59:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:06 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6834/425693237,v1:172.21.15.38:6835/425693237] up:boot 2024-04-21T23:59:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:06 smithi111 ceph-mon[31543]: fsmap 3 up:standby 2024-04-21T23:59:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:06 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:59:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:06 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/158594709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:07.639 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:07 smithi038 ceph-mon[27972]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:07.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:07 smithi203 ceph-mon[31431]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:07.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:07 smithi111 ceph-mon[31543]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:08.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:08.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 -- 172.21.15.38:0/1521664413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 msgr2=0x7febd8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 --2- 172.21.15.38:0/1521664413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd8108ae0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7febd0009ca0 tx=0x7febd002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 -- 172.21.15.38:0/1521664413 shutdown_connections 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 --2- 172.21.15.38:0/1521664413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd8108ae0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 --2- 172.21.15.38:0/1521664413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 0x7febd8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 --2- 172.21.15.38:0/1521664413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 0x7febd8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.187+0000 7febdf793640 1 -- 172.21.15.38:0/1521664413 >> 172.21.15.38:0/1521664413 conn(0x7febd80fbc30 msgr2=0x7febd80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- 172.21.15.38:0/1521664413 shutdown_connections 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- 172.21.15.38:0/1521664413 wait complete. 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 Processor -- start 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- start start 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 0x7febd8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 0x7febd8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7febd810d480 con 0x7febd8102250 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7febd810d300 con 0x7febd81006a0 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdf793640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febd810d600 con 0x7febd8101890 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febddd30640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febddd30640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56214/0 (socket says 172.21.15.38:56214) 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febddd30640 1 -- 172.21.15.38:0/1601856519 learned_addr learned my addr 172.21.15.38:0/1601856519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.188+0000 7febdd52f640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 0x7febd8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febdcd2e640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 0x7febd8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 -- 172.21.15.38:0/1601856519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 msgr2=0x7febd8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 0x7febd8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 -- 172.21.15.38:0/1601856519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 msgr2=0x7febd8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 0x7febd8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 -- 172.21.15.38:0/1601856519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd819f6e0 con 0x7febd8102250 2024-04-21T23:59:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febddd30640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7febd002c700 tx=0x7febd0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:08.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0039040 con 0x7febd8102250 2024-04-21T23:59:08.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd819f970 con 0x7febd8102250 2024-04-21T23:59:08.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd0044070 con 0x7febd8102250 2024-04-21T23:59:08.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd00042a0 con 0x7febd8102250 2024-04-21T23:59:08.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.189+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd819feb0 con 0x7febd8102250 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.190+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd8107960 con 0x7febd8102250 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.191+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7febd0003970 con 0x7febd8102250 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.192+0000 7febce7fc640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 0x7febc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.192+0000 7febdd52f640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 0x7febc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.192+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7febd00f10c0 con 0x7febd8102250 2024-04-21T23:59:08.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.192+0000 7febdd52f640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 0x7febc40a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febd4005fd0 tx=0x7febd400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:08.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.196+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7febd00be550 con 0x7febd8102250 2024-04-21T23:59:08.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.421+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febd802d350 con 0x7febd8102250 2024-04-21T23:59:08.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.421+0000 7febce7fc640 1 -- 172.21.15.38:0/1601856519 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7febd00bde50 con 0x7febd8102250 2024-04-21T23:59:08.414 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:08.414 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:59:08.415 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 msgr2=0x7febc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 0x7febc40a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febd4005fd0 tx=0x7febd400b410 comp rx=0 tx=0).stop 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 msgr2=0x7febd819f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7febd002c700 tx=0x7febd0032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 shutdown_connections 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7febc40a2b90 0x7febc40a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7febd8102250 0x7febd819f070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febd8101890 0x7febd8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 --2- 172.21.15.38:0/1601856519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febd81006a0 0x7febd8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 >> 172.21.15.38:0/1601856519 conn(0x7febd80fbc30 msgr2=0x7febd8104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 shutdown_connections 2024-04-21T23:59:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:08.423+0000 7febdf793640 1 -- 172.21.15.38:0/1601856519 wait complete. 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: Deploying daemon mds.e on smithi111 2024-04-21T23:59:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1601856519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6836/815557617,v1:172.21.15.38:6837/815557617] up:boot 2024-04-21T23:59:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: fsmap 4 up:standby 2024-04-21T23:59:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:09 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:09.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: Deploying daemon mds.e on smithi111 2024-04-21T23:59:09.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1601856519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:09.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6836/815557617,v1:172.21.15.38:6837/815557617] up:boot 2024-04-21T23:59:09.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: fsmap 4 up:standby 2024-04-21T23:59:09.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:09 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: Deploying daemon mds.e on smithi111 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1601856519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:09.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6836/815557617,v1:172.21.15.38:6837/815557617] up:boot 2024-04-21T23:59:09.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: fsmap 4 up:standby 2024-04-21T23:59:09.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:09 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:59:10.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:10.524 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 systemd[1]: Starting Ceph mds.e for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:10.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 -- 172.21.15.38:0/2757006180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 msgr2=0x7f19d4104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:10.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 --2- 172.21.15.38:0/2757006180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d4104df0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009900 tx=0x7f19c402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:10.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 -- 172.21.15.38:0/2757006180 shutdown_connections 2024-04-21T23:59:10.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 --2- 172.21.15.38:0/2757006180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 --2- 172.21.15.38:0/2757006180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d4104df0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 --2- 172.21.15.38:0/2757006180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 0x7f19d4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 -- 172.21.15.38:0/2757006180 >> 172.21.15.38:0/2757006180 conn(0x7f19d40fbc30 msgr2=0x7f19d40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:10.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 -- 172.21.15.38:0/2757006180 shutdown_connections 2024-04-21T23:59:10.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.550+0000 7f19dbc08640 1 -- 172.21.15.38:0/2757006180 wait complete. 2024-04-21T23:59:10.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 Processor -- start 2024-04-21T23:59:10.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 -- start start 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 0x7f19d419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19d4113a70 con 0x7f19d4102980 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f19d41138f0 con 0x7f19d4100010 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19dbc08640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19d4113bf0 con 0x7f19d4105330 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19d91a3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19d91a3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56232/0 (socket says 172.21.15.38:56232) 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.551+0000 7f19d91a3640 1 -- 172.21.15.38:0/904921972 learned_addr learned my addr 172.21.15.38:0/904921972 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d99a4640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 0x7f19d419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19da1a5640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 -- 172.21.15.38:0/904921972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 msgr2=0x7f19d41a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:10.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 -- 172.21.15.38:0/904921972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 msgr2=0x7f19d419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 0x7f19d419e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 -- 172.21.15.38:0/904921972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d41a5d00 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19da1a5640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19d91a3640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009230 tx=0x7f19c4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4041070 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19c40054e0 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d41a5f90 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4034440 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.552+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d41a63f0 con 0x7f19d4102980 2024-04-21T23:59:10.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.553+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d40fd720 con 0x7f19d4102980 2024-04-21T23:59:10.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.554+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f19c4044070 con 0x7f19d4102980 2024-04-21T23:59:10.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.555+0000 7f19caffd640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 0x7f19c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:10.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.555+0000 7f19d99a4640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 0x7f19c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:10.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.555+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f19c40f0d90 con 0x7f19d4102980 2024-04-21T23:59:10.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.555+0000 7f19d99a4640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 0x7f19c00a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005d00 tx=0x7f19d0005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:10.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.559+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f19c40be0f0 con 0x7f19d4102980 2024-04-21T23:59:10.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.785+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19d402d170 con 0x7f19d4102980 2024-04-21T23:59:10.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.786+0000 7f19caffd640 1 -- 172.21.15.38:0/904921972 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f19c40bd9f0 con 0x7f19d4102980 2024-04-21T23:59:10.779 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:10.779 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:59:10.779 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 5 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.787+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 msgr2=0x7f19c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.787+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 0x7f19c00a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005d00 tx=0x7f19d0005c50 comp rx=0 tx=0).stop 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.787+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 msgr2=0x7f19d419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.787+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009230 tx=0x7f19c4005dd0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 shutdown_connections 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19c00a2b90 0x7f19c00a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19d4105330 0x7f19d41a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19d4102980 0x7f19d419e620 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 --2- 172.21.15.38:0/904921972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19d4100010 0x7f19d419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:10.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 >> 172.21.15.38:0/904921972 conn(0x7f19d40fbc30 msgr2=0x7f19d4100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:10.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 shutdown_connections 2024-04-21T23:59:10.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:10.788+0000 7f19dbc08640 1 -- 172.21.15.38:0/904921972 wait complete. 2024-04-21T23:59:10.782 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 podman[60968]: 2024-04-21 23:59:10.434792723 +0000 UTC m=+0.017959094 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:10.782 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 podman[60968]: 2024-04-21 23:59:10.535876599 +0000 UTC m=+0.119043014 container create e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T23:59:10.782 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 podman[60968]: 2024-04-21 23:59:10.763561625 +0000 UTC m=+0.346727997 container init e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:59:10.782 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 podman[60968]: 2024-04-21 23:59:10.766048724 +0000 UTC m=+0.349215105 container start e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8) 2024-04-21T23:59:11.060 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 ceph-mds[60987]: -- 172.21.15.111:0/1030014553 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8bae27a40 con 0x55a8badfa800 2024-04-21T23:59:11.060 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: starting mds.e at 2024-04-21T23:59:11.060 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 bash[60968]: e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 2024-04-21T23:59:11.060 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 21 23:59:10 smithi111 systemd[1]: Started Ceph mds.e for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:11.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:11.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/904921972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:11.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:11 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/904921972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:11.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:11.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:11 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:11.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:11.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/904921972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:11.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:11 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:12.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:12 smithi203 ceph-mon[31431]: Deploying daemon mds.f on smithi203 2024-04-21T23:59:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:12 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6834/3450283861,v1:172.21.15.111:6835/3450283861] up:boot 2024-04-21T23:59:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:12 smithi203 ceph-mon[31431]: fsmap 5 up:standby 2024-04-21T23:59:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:12 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:59:12.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:12 smithi111 ceph-mon[31543]: Deploying daemon mds.f on smithi203 2024-04-21T23:59:12.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:12 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6834/3450283861,v1:172.21.15.111:6835/3450283861] up:boot 2024-04-21T23:59:12.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:12 smithi111 ceph-mon[31543]: fsmap 5 up:standby 2024-04-21T23:59:12.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:12 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:59:12.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:12 smithi038 ceph-mon[27972]: Deploying daemon mds.f on smithi203 2024-04-21T23:59:12.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:12 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6834/3450283861,v1:172.21.15.111:6835/3450283861] up:boot 2024-04-21T23:59:12.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:12 smithi038 ceph-mon[27972]: fsmap 5 up:standby 2024-04-21T23:59:12.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:12 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:59:12.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 -- 172.21.15.38:0/3139381149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 msgr2=0x7fcf0c109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/3139381149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c109bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009ca0 tx=0x7fcf0402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 -- 172.21.15.38:0/3139381149 shutdown_connections 2024-04-21T23:59:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/3139381149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c109bf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/3139381149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/3139381149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 0x7fcf0c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 -- 172.21.15.38:0/3139381149 >> 172.21.15.38:0/3139381149 conn(0x7fcf0c0fcd40 msgr2=0x7fcf0c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 -- 172.21.15.38:0/3139381149 shutdown_connections 2024-04-21T23:59:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.913+0000 7fcf10ff7640 1 -- 172.21.15.38:0/3139381149 wait complete. 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 Processor -- start 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 -- start start 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 0x7fcf0c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c10d430 con 0x7fcf0c103360 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c10d2b0 con 0x7fcf0c1017b0 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf10ff7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c10d5b0 con 0x7fcf0c1029a0 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf0ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf03fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf0ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56250/0 (socket says 172.21.15.38:56250) 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf0ad9d640 1 -- 172.21.15.38:0/2826824326 learned_addr learned my addr 172.21.15.38:0/2826824326 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.914+0000 7fcf0a59c640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 0x7fcf0c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 -- 172.21.15.38:0/2826824326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 msgr2=0x7fcf0c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 -- 172.21.15.38:0/2826824326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 msgr2=0x7fcf0c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 0x7fcf0c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 -- 172.21.15.38:0/2826824326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c074900 con 0x7fcf0c103360 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf03fff640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf0ad9d640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcf0402c630 tx=0x7fcf04032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04041020 con 0x7fcf0c103360 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf04034cd0 con 0x7fcf0c103360 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c1a1b60 con 0x7fcf0c103360 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04039040 con 0x7fcf0c103360 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.915+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c1a20b0 con 0x7fcf0c103360 2024-04-21T23:59:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.916+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf0c108ba0 con 0x7fcf0c103360 2024-04-21T23:59:12.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.917+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcf04044020 con 0x7fcf0c103360 2024-04-21T23:59:12.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.918+0000 7fcf037fe640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:12.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.918+0000 7fcf0a59c640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:12.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.918+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf040f0aa0 con 0x7fcf0c103360 2024-04-21T23:59:12.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.918+0000 7fcf0a59c640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 0x7fcef80a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005fd0 tx=0x7fcefc00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:12.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:12.923+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcf0404b020 con 0x7fcf0c103360 2024-04-21T23:59:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.147+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf0c062340 con 0x7fcf0c103360 2024-04-21T23:59:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.148+0000 7fcf037fe640 1 -- 172.21.15.38:0/2826824326 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fcf040bdea0 con 0x7fcf0c103360 2024-04-21T23:59:13.141 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:13.141 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T23:59:13.141 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 6 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 msgr2=0x7fcef80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 0x7fcef80a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005fd0 tx=0x7fcefc00b410 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 msgr2=0x7fcf0c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcf0402c630 tx=0x7fcf04032ea0 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 shutdown_connections 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcef80a2b90 0x7fcef80a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf0c103360 0x7fcf0c074070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c1029a0 0x7fcf0c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 --2- 172.21.15.38:0/2826824326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0c1017b0 0x7fcf0c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 >> 172.21.15.38:0/2826824326 conn(0x7fcf0c0fcd40 msgr2=0x7fcf0c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 shutdown_connections 2024-04-21T23:59:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:13.150+0000 7fcf10ff7640 1 -- 172.21.15.38:0/2826824326 wait complete. 2024-04-21T23:59:13.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:13 smithi203 ceph-mon[31431]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:13.280 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 systemd[1]: Starting Ceph mds.f for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:13.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:13 smithi038 ceph-mon[27972]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:13.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:13 smithi111 ceph-mon[31543]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:13.636 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 podman[62012]: 2024-04-21 23:59:13.265928805 +0000 UTC m=+0.018162200 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:13.636 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 podman[62012]: 2024-04-21 23:59:13.390933428 +0000 UTC m=+0.143166796 container create bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T23:59:13.636 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 podman[62012]: 2024-04-21 23:59:13.64407145 +0000 UTC m=+0.396304823 container init bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T23:59:14.029 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 podman[62012]: 2024-04-21 23:59:13.64740678 +0000 UTC m=+0.399640149 container start bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T23:59:14.030 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 ceph-mds[62031]: -- 172.21.15.203:0/677415394 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8f307ba40 con 0x55a8f304f000 2024-04-21T23:59:14.030 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: starting mds.f at 2024-04-21T23:59:14.030 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 bash[62012]: bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 2024-04-21T23:59:14.030 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 21 23:59:13 smithi203 systemd[1]: Started Ceph mds.f for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:14.432 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2826824326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:14.433 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:14 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2826824326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:14.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:14 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:14.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2826824326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:14 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:15.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- 172.21.15.38:0/2831962798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 msgr2=0x7f6d24108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2831962798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d24108ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009ca0 tx=0x7f6d1c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- 172.21.15.38:0/2831962798 shutdown_connections 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2831962798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d24108ac0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2831962798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 0x7f6d24101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2831962798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 0x7f6d24100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- 172.21.15.38:0/2831962798 >> 172.21.15.38:0/2831962798 conn(0x7f6d240fbbf0 msgr2=0x7f6d240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- 172.21.15.38:0/2831962798 shutdown_connections 2024-04-21T23:59:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- 172.21.15.38:0/2831962798 wait complete. 2024-04-21T23:59:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 Processor -- start 2024-04-21T23:59:15.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.276+0000 7f6d2a439640 1 -- start start 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 0x7f6d24197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 0x7f6d24198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2410c2e0 con 0x7f6d24102230 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2410c160 con 0x7f6d24101870 2024-04-21T23:59:15.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d2a439640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2410c460 con 0x7f6d24100680 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43638/0 (socket says 172.21.15.38:43638) 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 -- 172.21.15.38:0/2920307041 learned_addr learned my addr 172.21.15.38:0/2920307041 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d237fe640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 0x7f6d24198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d23fff640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 0x7f6d24197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 -- 172.21.15.38:0/2920307041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 msgr2=0x7f6d24197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 0x7f6d24197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 -- 172.21.15.38:0/2920307041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 msgr2=0x7f6d24198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 0x7f6d24198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 -- 172.21.15.38:0/2920307041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2419f700 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.277+0000 7f6d289d6640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c032530 tx=0x7f6d1c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.278+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c041020 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.278+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2419f990 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.278+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d1c034cd0 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.278+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c039040 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.278+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2419fe50 con 0x7f6d24102230 2024-04-21T23:59:15.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.279+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d241a00b0 con 0x7f6d24102230 2024-04-21T23:59:15.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.280+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d1c044020 con 0x7f6d24102230 2024-04-21T23:59:15.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.280+0000 7f6d217fa640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 0x7f6d0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:15.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.281+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d1c0f0b10 con 0x7f6d24102230 2024-04-21T23:59:15.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.281+0000 7f6d23fff640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 0x7f6d0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:15.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.281+0000 7f6d23fff640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 0x7f6d0c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d14005fd0 tx=0x7f6d14005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:15.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.285+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6d1c04b020 con 0x7f6d24102230 2024-04-21T23:59:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:15 smithi038 ceph-mon[27972]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:15 smithi038 ceph-mon[27972]: Deploying daemon mds.c on smithi203 2024-04-21T23:59:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:15 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:boot 2024-04-21T23:59:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:15 smithi038 ceph-mon[27972]: fsmap 6 up:standby 2024-04-21T23:59:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:15 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:59:15.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:15 smithi203 ceph-mon[31431]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:15.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:15 smithi203 ceph-mon[31431]: Deploying daemon mds.c on smithi203 2024-04-21T23:59:15.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:15 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:boot 2024-04-21T23:59:15.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:15 smithi203 ceph-mon[31431]: fsmap 6 up:standby 2024-04-21T23:59:15.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:15 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:59:15.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.507+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d241a0440 con 0x7f6d24102230 2024-04-21T23:59:15.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.507+0000 7f6d217fa640 1 -- 172.21.15.38:0/2920307041 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f6d1c0bdfa0 con 0x7f6d24102230 2024-04-21T23:59:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:15.500 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T23:59:15.501 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 7 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 msgr2=0x7f6d0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 0x7f6d0c0a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6d14005fd0 tx=0x7f6d14005850 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 msgr2=0x7f6d2419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c032530 tx=0x7f6d1c032f40 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 shutdown_connections 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d0c0a2b90 0x7f6d0c0a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d24102230 0x7f6d2419f090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d24101870 0x7f6d24198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 --2- 172.21.15.38:0/2920307041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d24100680 0x7f6d24197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 >> 172.21.15.38:0/2920307041 conn(0x7f6d240fbbf0 msgr2=0x7f6d240fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:15.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 shutdown_connections 2024-04-21T23:59:15.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:15.510+0000 7f6d2a439640 1 -- 172.21.15.38:0/2920307041 wait complete. 2024-04-21T23:59:15.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:15 smithi111 ceph-mon[31543]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:15 smithi111 ceph-mon[31543]: Deploying daemon mds.c on smithi203 2024-04-21T23:59:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:15 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:boot 2024-04-21T23:59:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:15 smithi111 ceph-mon[31543]: fsmap 6 up:standby 2024-04-21T23:59:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:15 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:59:16.030 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:15 smithi203 systemd[1]: Starting Ceph mds.c for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:16.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:16 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2920307041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:16.511 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:16 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2920307041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:16.511 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 podman[62320]: 2024-04-21 23:59:16.080111607 +0000 UTC m=+0.017939311 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:16.511 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 podman[62320]: 2024-04-21 23:59:16.199759038 +0000 UTC m=+0.137586733 container create 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:59:16.512 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 podman[62320]: 2024-04-21 23:59:16.519287305 +0000 UTC m=+0.457115000 container init 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T23:59:16.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:16 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2920307041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:16.779 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 podman[62320]: 2024-04-21 23:59:16.521933135 +0000 UTC m=+0.459760844 container start 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T23:59:16.779 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 ceph-mds[62339]: -- 172.21.15.203:0/1959171395 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d756d15a40 con 0x55d756ce8800 2024-04-21T23:59:16.780 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: starting mds.c at 2024-04-21T23:59:16.780 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 bash[62320]: 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 2024-04-21T23:59:16.780 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 21 23:59:16 smithi203 systemd[1]: Started Ceph mds.c for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:17.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:17.451 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:17.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:17.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:17.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:17.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 -- 172.21.15.38:0/3119351761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1017b0 msgr2=0x7f23ec101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 --2- 172.21.15.38:0/3119351761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1017b0 0x7f23ec101bd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009900 tx=0x7f23dc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 -- 172.21.15.38:0/3119351761 shutdown_connections 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 --2- 172.21.15.38:0/3119351761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec103360 0x7f23ec109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 --2- 172.21.15.38:0/3119351761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 --2- 172.21.15.38:0/3119351761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1017b0 0x7f23ec101bd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 -- 172.21.15.38:0/3119351761 >> 172.21.15.38:0/3119351761 conn(0x7f23ec0fcd40 msgr2=0x7f23ec0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 -- 172.21.15.38:0/3119351761 shutdown_connections 2024-04-21T23:59:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.636+0000 7f23f0b93640 1 -- 172.21.15.38:0/3119351761 wait complete. 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.637+0000 7f23f0b93640 1 Processor -- start 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.637+0000 7f23f0b93640 1 -- start start 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec1017b0 0x7f23ec0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec10d450 con 0x7f23ec103360 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec10d2d0 con 0x7f23ec1029a0 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec10d5d0 con 0x7f23ec1017b0 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43662/0 (socket says 172.21.15.38:43662) 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 -- 172.21.15.38:0/211292516 learned_addr learned my addr 172.21.15.38:0/211292516 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23e3fff640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ea59c640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec1017b0 0x7f23ec0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 -- 172.21.15.38:0/211292516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec1017b0 msgr2=0x7f23ec0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec1017b0 0x7f23ec0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 -- 172.21.15.38:0/211292516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 msgr2=0x7f23ec073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 -- 172.21.15.38:0/211292516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec074900 con 0x7f23ec103360 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23e3fff640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23ead9d640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009fd0 tx=0x7f23e400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4017070 con 0x7f23ec103360 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e4003e50 con 0x7f23ec103360 2024-04-21T23:59:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec1a1be0 con 0x7f23ec103360 2024-04-21T23:59:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.638+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40106c0 con 0x7f23ec103360 2024-04-21T23:59:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.639+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1a20e0 con 0x7f23ec103360 2024-04-21T23:59:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.639+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ec108ad0 con 0x7f23ec103360 2024-04-21T23:59:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.641+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f23e4003970 con 0x7f23ec103360 2024-04-21T23:59:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.641+0000 7f23e37fe640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 0x7f23d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.641+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f23e40ce7c0 con 0x7f23ec103360 2024-04-21T23:59:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.641+0000 7f23ea59c640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 0x7f23d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.642+0000 7f23ea59c640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 0x7f23d80a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f23dc02c170 tx=0x7f23dc02df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:17.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.646+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f23e409b570 con 0x7f23ec103360 2024-04-21T23:59:17.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.871+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23ec02d3c0 con 0x7f23ec103360 2024-04-21T23:59:17.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.872+0000 7f23e37fe640 1 -- 172.21.15.38:0/211292516 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f23e409f870 con 0x7f23ec103360 2024-04-21T23:59:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:17.865 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:59:17.865 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 8 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 msgr2=0x7f23d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 0x7f23d80a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f23dc02c170 tx=0x7f23dc02df90 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 msgr2=0x7f23ec074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009fd0 tx=0x7f23e400b260 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 shutdown_connections 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23d80a2b90 0x7f23d80a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec103360 0x7f23ec074070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec1029a0 0x7f23ec073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 --2- 172.21.15.38:0/211292516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec1017b0 0x7f23ec0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 >> 172.21.15.38:0/211292516 conn(0x7f23ec0fcd40 msgr2=0x7f23ec105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 shutdown_connections 2024-04-21T23:59:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:17.874+0000 7f23f0b93640 1 -- 172.21.15.38:0/211292516 wait complete. 2024-04-21T23:59:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:18 smithi038 ceph-mon[27972]: Deploying daemon mds.h on smithi111 2024-04-21T23:59:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:18 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:boot 2024-04-21T23:59:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:18 smithi038 ceph-mon[27972]: fsmap 7 up:standby 2024-04-21T23:59:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:18 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:59:18.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/211292516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:18.468 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:18 smithi111 ceph-mon[31543]: Deploying daemon mds.h on smithi111 2024-04-21T23:59:18.468 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:18 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:boot 2024-04-21T23:59:18.468 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:18 smithi111 ceph-mon[31543]: fsmap 7 up:standby 2024-04-21T23:59:18.468 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:18 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:59:18.468 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:18 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/211292516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:18.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:18 smithi203 ceph-mon[31431]: Deploying daemon mds.h on smithi111 2024-04-21T23:59:18.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:18 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:boot 2024-04-21T23:59:18.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:18 smithi203 ceph-mon[31431]: fsmap 7 up:standby 2024-04-21T23:59:18.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:18 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:59:18.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:18 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/211292516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:19.134 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 systemd[1]: Starting Ceph mds.h for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:19.391 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:19 smithi111 ceph-mon[31543]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:19.391 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 podman[61280]: 2024-04-21 23:59:19.159403631 +0000 UTC m=+0.017801045 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:19.391 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 podman[61280]: 2024-04-21 23:59:19.274726234 +0000 UTC m=+0.133123677 container create e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T23:59:19.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:19 smithi038 ceph-mon[27972]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:19.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:19 smithi203 ceph-mon[31431]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 podman[61280]: 2024-04-21 23:59:19.502427244 +0000 UTC m=+0.360824645 container init e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 podman[61280]: 2024-04-21 23:59:19.505148168 +0000 UTC m=+0.363545578 container start e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 ceph-mds[61299]: -- 172.21.15.111:0/2478322154 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e41254ba40 con 0x55e41251e800 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: starting mds.h at 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 bash[61280]: e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 2024-04-21T23:59:19.827 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 21 23:59:19 smithi111 systemd[1]: Started Ceph mds.h for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:19.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:19.999+0000 7fdd45c2d640 1 -- 172.21.15.38:0/39586219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40101880 msgr2=0x7fdd40101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:19.999+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/39586219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40101880 0x7fdd40101d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009900 tx=0x7fdd2802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 -- 172.21.15.38:0/39586219 shutdown_connections 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/39586219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40102240 0x7fdd40108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/39586219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40101880 0x7fdd40101d00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/39586219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40100690 0x7fdd40100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 -- 172.21.15.38:0/39586219 >> 172.21.15.38:0/39586219 conn(0x7fdd400fbc20 msgr2=0x7fdd400fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:19.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 -- 172.21.15.38:0/39586219 shutdown_connections 2024-04-21T23:59:19.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 -- 172.21.15.38:0/39586219 wait complete. 2024-04-21T23:59:19.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 Processor -- start 2024-04-21T23:59:19.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.000+0000 7fdd45c2d640 1 -- start start 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 0x7fdd40102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40102240 0x7fdd401031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010c340 con 0x7fdd40100690 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010c1c0 con 0x7fdd40101880 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010c4c0 con 0x7fdd40102240 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43684/0 (socket says 172.21.15.38:43684) 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 -- 172.21.15.38:0/538519111 learned_addr learned my addr 172.21.15.38:0/538519111 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd36ffd640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 0x7fdd40102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3ffff640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40102240 0x7fdd401031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 -- 172.21.15.38:0/538519111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40102240 msgr2=0x7fdd401031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40102240 0x7fdd401031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 -- 172.21.15.38:0/538519111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 msgr2=0x7fdd40102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 0x7fdd40102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 -- 172.21.15.38:0/538519111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40103a30 con 0x7fdd40100690 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd36ffd640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 0x7fdd40102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3f7fe640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3000cd80 con 0x7fdd40100690 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd300035f0 con 0x7fdd40100690 2024-04-21T23:59:19.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd401a1bf0 con 0x7fdd40100690 2024-04-21T23:59:19.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30010690 con 0x7fdd40100690 2024-04-21T23:59:19.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.001+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401a2110 con 0x7fdd40100690 2024-04-21T23:59:19.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.003+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd400689b0 con 0x7fdd40100690 2024-04-21T23:59:20.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.004+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdd300037a0 con 0x7fdd40100690 2024-04-21T23:59:20.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.004+0000 7fdd3d7fa640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 0x7fdd2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:20.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.004+0000 7fdd36ffd640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 0x7fdd2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:20.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.004+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdd300cf040 con 0x7fdd40100690 2024-04-21T23:59:20.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.005+0000 7fdd36ffd640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 0x7fdd2c0a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd2802c240 tx=0x7fdd2802df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:20.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.008+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdd3009c4d0 con 0x7fdd40100690 2024-04-21T23:59:20.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.230+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd401a22e0 con 0x7fdd40100690 2024-04-21T23:59:20.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.231+0000 7fdd3d7fa640 1 -- 172.21.15.38:0/538519111 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7fdd3009bdd0 con 0x7fdd40100690 2024-04-21T23:59:20.225 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:20.225 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:59:20.225 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 msgr2=0x7fdd2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 0x7fdd2c0a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdd2802c240 tx=0x7fdd2802df90 comp rx=0 tx=0).stop 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 msgr2=0x7fdd40104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 shutdown_connections 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdd2c0a2b90 0x7fdd2c0a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd40102240 0x7fdd401031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd40101880 0x7fdd40102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 --2- 172.21.15.38:0/538519111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd40100690 0x7fdd40104620 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:20.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 >> 172.21.15.38:0/538519111 conn(0x7fdd400fbc20 msgr2=0x7fdd400fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:20.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 shutdown_connections 2024-04-21T23:59:20.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:20.233+0000 7fdd45c2d640 1 -- 172.21.15.38:0/538519111 wait complete. 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:20.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: Deploying daemon mds.i on smithi203 2024-04-21T23:59:20.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:boot 2024-04-21T23:59:20.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: fsmap 8 up:standby 2024-04-21T23:59:20.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:59:20.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/538519111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:21.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: Deploying daemon mds.i on smithi203 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:boot 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: fsmap 8 up:standby 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:59:21.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:20 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/538519111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:21.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: Deploying daemon mds.i on smithi203 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:boot 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: fsmap 8 up:standby 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:59:21.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:20 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/538519111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:22.030 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:21 smithi203 systemd[1]: Starting Ceph mds.i for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:22.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:22.325 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 podman[62630]: 2024-04-21 23:59:21.977194018 +0000 UTC m=+0.018398236 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:22.325 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 podman[62630]: 2024-04-21 23:59:22.099822153 +0000 UTC m=+0.141026356 container create 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T23:59:22.325 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 podman[62630]: 2024-04-21 23:59:22.332794292 +0000 UTC m=+0.373998495 container init 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-21T23:59:22.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 -- 172.21.15.38:0/4071700994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 msgr2=0x7fbd80101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:22.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 --2- 172.21.15.38:0/4071700994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80101d10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009900 tx=0x7fbd7402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:22.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 -- 172.21.15.38:0/4071700994 shutdown_connections 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 --2- 172.21.15.38:0/4071700994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80102250 0x7fbd80108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 --2- 172.21.15.38:0/4071700994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80101d10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 --2- 172.21.15.38:0/4071700994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd801006a0 0x7fbd80100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 -- 172.21.15.38:0/4071700994 >> 172.21.15.38:0/4071700994 conn(0x7fbd800fbc30 msgr2=0x7fbd800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 -- 172.21.15.38:0/4071700994 shutdown_connections 2024-04-21T23:59:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.359+0000 7fbd849b1640 1 -- 172.21.15.38:0/4071700994 wait complete. 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 Processor -- start 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 -- start start 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd80102250 0x7fbd8019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:22.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8010c320 con 0x7fbd80101890 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8010c1a0 con 0x7fbd80102250 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd849b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8010c4a0 con 0x7fbd801006a0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd7e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd7dd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd7e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45898/0 (socket says 172.21.15.38:45898) 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.360+0000 7fbd7e59c640 1 -- 172.21.15.38:0/828806696 learned_addr learned my addr 172.21.15.38:0/828806696 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7ed9d640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd80102250 0x7fbd8019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 -- 172.21.15.38:0/828806696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd80102250 msgr2=0x7fbd8019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd80102250 0x7fbd8019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 -- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 msgr2=0x7fbd80197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 -- 172.21.15.38:0/828806696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8019f6d0 con 0x7fbd801006a0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd7e59c640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000e9c0 tx=0x7fbd7000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:22.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7000cd80 con 0x7fbd801006a0 2024-04-21T23:59:22.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8019f9c0 con 0x7fbd801006a0 2024-04-21T23:59:22.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd700035f0 con 0x7fbd801006a0 2024-04-21T23:59:22.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70010690 con 0x7fbd801006a0 2024-04-21T23:59:22.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.361+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8019fee0 con 0x7fbd801006a0 2024-04-21T23:59:22.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.362+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd801a0170 con 0x7fbd801006a0 2024-04-21T23:59:22.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.364+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbd700037a0 con 0x7fbd801006a0 2024-04-21T23:59:22.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.364+0000 7fbd6f7fe640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 0x7fbd680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:22.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.364+0000 7fbd7dd9b640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 0x7fbd680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:22.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.364+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbd700ceff0 con 0x7fbd801006a0 2024-04-21T23:59:22.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.365+0000 7fbd7dd9b640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 0x7fbd680a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbd7402c7b0 tx=0x7fbd74005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:22.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.368+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbd7009c350 con 0x7fbd801006a0 2024-04-21T23:59:22.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.608+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd801a03e0 con 0x7fbd801006a0 2024-04-21T23:59:22.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.609+0000 7fbd6f7fe640 1 -- 172.21.15.38:0/828806696 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7fbd7009bc50 con 0x7fbd801006a0 2024-04-21T23:59:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:22.603 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:59:22.603 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 msgr2=0x7fbd680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 0x7fbd680a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbd7402c7b0 tx=0x7fbd74005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 msgr2=0x7fbd80197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbd7000e9c0 tx=0x7fbd7000ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 shutdown_connections 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbd680a2b90 0x7fbd680a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd80102250 0x7fbd8019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd80101890 0x7fbd80197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 --2- 172.21.15.38:0/828806696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd801006a0 0x7fbd80197ab0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:22.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 >> 172.21.15.38:0/828806696 conn(0x7fbd800fbc30 msgr2=0x7fbd800fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:22.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 shutdown_connections 2024-04-21T23:59:22.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:22.612+0000 7fbd849b1640 1 -- 172.21.15.38:0/828806696 wait complete. 2024-04-21T23:59:22.779 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 podman[62630]: 2024-04-21 23:59:22.335546617 +0000 UTC m=+0.376750821 container start 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:59:22.780 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 ceph-mds[62649]: -- 172.21.15.203:0/2414498086 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634f4999a40 con 0x5634f496c800 2024-04-21T23:59:22.780 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: starting mds.i at 2024-04-21T23:59:22.780 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 bash[62630]: 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 2024-04-21T23:59:22.780 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 21 23:59:22 smithi203 systemd[1]: Started Ceph mds.i for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:23.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/828806696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:boot 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: fsmap 9 up:standby 2024-04-21T23:59:23.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:23 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:59:23.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:23.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/828806696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:boot 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: fsmap 9 up:standby 2024-04-21T23:59:23.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:23 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:23.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:23.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:23.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/828806696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:23.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:boot 2024-04-21T23:59:23.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: fsmap 9 up:standby 2024-04-21T23:59:23.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:23 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:59:24.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:24.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:24 smithi038 ceph-mon[27972]: Deploying daemon mds.g on smithi038 2024-04-21T23:59:24.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.744+0000 7f9bac720640 1 -- 172.21.15.38:0/916041930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4100680 msgr2=0x7f9ba4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.744+0000 7f9bac720640 1 --2- 172.21.15.38:0/916041930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4100680 0x7f9ba4100aa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009900 tx=0x7f9ba002bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 -- 172.21.15.38:0/916041930 shutdown_connections 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 --2- 172.21.15.38:0/916041930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4102230 0x7f9ba4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 --2- 172.21.15.38:0/916041930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4101870 0x7f9ba4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 --2- 172.21.15.38:0/916041930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4100680 0x7f9ba4100aa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 -- 172.21.15.38:0/916041930 >> 172.21.15.38:0/916041930 conn(0x7f9ba40fbbf0 msgr2=0x7f9ba40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 -- 172.21.15.38:0/916041930 shutdown_connections 2024-04-21T23:59:24.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 -- 172.21.15.38:0/916041930 wait complete. 2024-04-21T23:59:24.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 Processor -- start 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.745+0000 7f9bac720640 1 -- start start 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4100680 0x7f9ba410ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 0x7f9ba41094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410d420 con 0x7f9ba4102230 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410d2a0 con 0x7f9ba4101870 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410d5a0 con 0x7f9ba4100680 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47214/0 (socket says 172.21.15.38:47214) 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 -- 172.21.15.38:0/3997244840 learned_addr learned my addr 172.21.15.38:0/3997244840 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baa4bc640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4100680 0x7f9ba410ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9ba9cbb640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 0x7f9ba41094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 -- 172.21.15.38:0/3997244840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4100680 msgr2=0x7f9ba410ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:24.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4100680 0x7f9ba410ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 -- 172.21.15.38:0/3997244840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 msgr2=0x7f9ba41094c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 0x7f9ba41094c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 -- 172.21.15.38:0/3997244840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba410a290 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9ba9cbb640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 0x7f9ba41094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9baacbd640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00eaa0 tx=0x7f9b9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c00cdb0 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba41a1bf0 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c005020 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.746+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0116b0 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.747+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba41a2140 con 0x7f9ba4102230 2024-04-21T23:59:24.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.747+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba41079f0 con 0x7f9ba4102230 2024-04-21T23:59:24.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.748+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9b9c005560 con 0x7f9ba4102230 2024-04-21T23:59:24.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.749+0000 7f9b9b7fe640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 0x7f9b900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:24.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.749+0000 7f9baa4bc640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 0x7f9b900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:24.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.750+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9b9c0cee90 con 0x7f9ba4102230 2024-04-21T23:59:24.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.750+0000 7f9baa4bc640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 0x7f9b900a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0000c00 tx=0x7f9ba0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:24.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.754+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9b9c016030 con 0x7f9ba4102230 2024-04-21T23:59:24.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:24 smithi203 ceph-mon[31431]: Deploying daemon mds.g on smithi038 2024-04-21T23:59:24.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:24 smithi111 ceph-mon[31543]: Deploying daemon mds.g on smithi038 2024-04-21T23:59:24.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.983+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ba402d170 con 0x7f9ba4102230 2024-04-21T23:59:24.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.983+0000 7f9b9b7fe640 1 -- 172.21.15.38:0/3997244840 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f9b9c09c290 con 0x7f9ba4102230 2024-04-21T23:59:24.976 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:24.976 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:59:24.977 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 10 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 msgr2=0x7f9b900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 0x7f9b900a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0000c00 tx=0x7f9ba0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 msgr2=0x7f9ba4109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00eaa0 tx=0x7f9b9c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 shutdown_connections 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9b900a2b90 0x7f9b900a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ba4102230 0x7f9ba4109a00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ba4101870 0x7f9ba41094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 --2- 172.21.15.38:0/3997244840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ba4100680 0x7f9ba410ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 >> 172.21.15.38:0/3997244840 conn(0x7f9ba40fbbf0 msgr2=0x7f9ba4104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 shutdown_connections 2024-04-21T23:59:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:24.986+0000 7f9bac720640 1 -- 172.21.15.38:0/3997244840 wait complete. 2024-04-21T23:59:25.073 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 podman[69484]: 2024-04-21 23:59:25.080999848 +0000 UTC m=+0.373254710 container init 7e1a5fbaf2963811e71ccfd97611019cca9cef5c0d05ddf85908f571d374eb97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0) 2024-04-21T23:59:25.458 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 podman[69484]: 2024-04-21 23:59:25.083874886 +0000 UTC m=+0.376129748 container start 7e1a5fbaf2963811e71ccfd97611019cca9cef5c0d05ddf85908f571d374eb97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T23:59:25.458 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mds[69524]: -- 172.21.15.38:0/4003618008 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b707749a40 con 0x55b70771c800 2024-04-21T23:59:25.458 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: starting mds.g at 2024-04-21T23:59:25.458 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 bash[69484]: 7e1a5fbaf2963811e71ccfd97611019cca9cef5c0d05ddf85908f571d374eb97 2024-04-21T23:59:25.458 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 21 23:59:25 smithi038 systemd[1]: Started Ceph mds.g for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:25.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3997244840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:25.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:25 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:25.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3997244840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:25.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:25 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3997244840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:25.926 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:25 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:26.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:26 smithi038 ceph-mon[27972]: Deploying daemon mds.a on smithi038 2024-04-21T23:59:26.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:26 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6838/3894275312,v1:172.21.15.38:6839/3894275312] up:boot 2024-04-21T23:59:26.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:26 smithi038 ceph-mon[27972]: fsmap 10 up:standby 2024-04-21T23:59:26.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:59:26.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:26 smithi203 ceph-mon[31431]: Deploying daemon mds.a on smithi038 2024-04-21T23:59:26.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:26 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6838/3894275312,v1:172.21.15.38:6839/3894275312] up:boot 2024-04-21T23:59:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:26 smithi203 ceph-mon[31431]: fsmap 10 up:standby 2024-04-21T23:59:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:59:26.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:26 smithi111 ceph-mon[31543]: Deploying daemon mds.a on smithi038 2024-04-21T23:59:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:26 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6838/3894275312,v1:172.21.15.38:6839/3894275312] up:boot 2024-04-21T23:59:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:26 smithi111 ceph-mon[31543]: fsmap 10 up:standby 2024-04-21T23:59:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:59:26.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 -- 172.21.15.38:0/3618204219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc102230 msgr2=0x7fa3dc108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/3618204219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc102230 0x7fa3dc108ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009900 tx=0x7fa3d402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 -- 172.21.15.38:0/3618204219 shutdown_connections 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/3618204219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc102230 0x7fa3dc108ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/3618204219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc101870 0x7fa3dc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/3618204219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc100680 0x7fa3dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 -- 172.21.15.38:0/3618204219 >> 172.21.15.38:0/3618204219 conn(0x7fa3dc0fbbf0 msgr2=0x7fa3dc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:27.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 -- 172.21.15.38:0/3618204219 shutdown_connections 2024-04-21T23:59:27.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.267+0000 7fa3e1c20640 1 -- 172.21.15.38:0/3618204219 wait complete. 2024-04-21T23:59:27.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.268+0000 7fa3e1c20640 1 Processor -- start 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.268+0000 7fa3e1c20640 1 -- start start 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc101870 0x7fa3dc196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc102230 0x7fa3dc19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc10c350 con 0x7fa3dc100680 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc10c1d0 con 0x7fa3dc101870 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc10c4d0 con 0x7fa3dc102230 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47244/0 (socket says 172.21.15.38:47244) 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 -- 172.21.15.38:0/1379763143 learned_addr learned my addr 172.21.15.38:0/1379763143 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3dbfff640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc102230 0x7fa3dc19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3daffd640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc101870 0x7fa3dc196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 -- 172.21.15.38:0/1379763143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc102230 msgr2=0x7fa3dc19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc102230 0x7fa3dc19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 -- 172.21.15.38:0/1379763143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc101870 msgr2=0x7fa3dc196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc101870 0x7fa3dc196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 -- 172.21.15.38:0/1379763143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3dc19e550 con 0x7fa3dc100680 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3db7fe640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00ebc0 tx=0x7fa3cc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc003b70 con 0x7fa3dc100680 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3dc19e840 con 0x7fa3dc100680 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.269+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3cc003d10 con 0x7fa3dc100680 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.270+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc019980 con 0x7fa3dc100680 2024-04-21T23:59:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.270+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3dc19ede0 con 0x7fa3dc100680 2024-04-21T23:59:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.271+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3dc107940 con 0x7fa3dc100680 2024-04-21T23:59:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.272+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa3cc019b20 con 0x7fa3dc100680 2024-04-21T23:59:27.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.273+0000 7fa3d8ff9640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 0x7fa3c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:27.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.273+0000 7fa3daffd640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 0x7fa3c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:27.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.274+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa3cc0cf500 con 0x7fa3dc100680 2024-04-21T23:59:27.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.274+0000 7fa3daffd640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 0x7fa3c40a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc197e40 tx=0x7fa3d0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:27.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.278+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa3cc09c860 con 0x7fa3dc100680 2024-04-21T23:59:27.458 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 systemd[1]: Starting Ceph mds.a for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:27.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.546+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3dc02d170 con 0x7fa3dc100680 2024-04-21T23:59:27.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.547+0000 7fa3d8ff9640 1 -- 172.21.15.38:0/1379763143 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fa3dc02d170 con 0x7fa3dc100680 2024-04-21T23:59:27.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:27.540 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:59:27.540 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 11 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 msgr2=0x7fa3c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 0x7fa3c40a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc197e40 tx=0x7fa3d0005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 msgr2=0x7fa3dc196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00ebc0 tx=0x7fa3cc00c6a0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 shutdown_connections 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c40a2b90 0x7fa3c40a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3dc102230 0x7fa3dc19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3dc101870 0x7fa3dc196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 --2- 172.21.15.38:0/1379763143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3dc100680 0x7fa3dc196930 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:27.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 >> 172.21.15.38:0/1379763143 conn(0x7fa3dc0fbbf0 msgr2=0x7fa3dc104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:27.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 shutdown_connections 2024-04-21T23:59:27.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:27.549+0000 7fa3e1c20640 1 -- 172.21.15.38:0/1379763143 wait complete. 2024-04-21T23:59:27.773 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:27 smithi038 ceph-mon[27972]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:27.773 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 podman[69863]: 2024-04-21 23:59:27.550847995 +0000 UTC m=+0.018839663 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:27.773 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 podman[69863]: 2024-04-21 23:59:27.659616944 +0000 UTC m=+0.127608600 container create 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T23:59:27.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:27 smithi203 ceph-mon[31431]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:27.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:27 smithi111 ceph-mon[31543]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 podman[69863]: 2024-04-21 23:59:27.912476633 +0000 UTC m=+0.380468288 container init 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 podman[69863]: 2024-04-21 23:59:27.915104152 +0000 UTC m=+0.383095819 container start 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 ceph-mds[69884]: -- 172.21.15.38:0/3990170211 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5603f62c1a40 con 0x5603f6294800 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:27 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: starting mds.a at 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:28 smithi038 bash[69863]: 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 2024-04-21T23:59:28.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 21 23:59:28 smithi038 systemd[1]: Started Ceph mds.a for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:28.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1379763143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:28 smithi203 ceph-mon[31431]: Deploying daemon mds.b on smithi111 2024-04-21T23:59:28.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1379763143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:28.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:28 smithi111 ceph-mon[31543]: Deploying daemon mds.b on smithi111 2024-04-21T23:59:28.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1379763143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:28.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:28.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:28.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:28 smithi038 ceph-mon[27972]: Deploying daemon mds.b on smithi111 2024-04-21T23:59:29.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 -- 172.21.15.38:0/3094472001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8600101890 msgr2=0x7f8600101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 --2- 172.21.15.38:0/3094472001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8600101890 0x7f8600101d10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009900 tx=0x7f85e802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 -- 172.21.15.38:0/3094472001 shutdown_connections 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 --2- 172.21.15.38:0/3094472001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 0x7f8600108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 --2- 172.21.15.38:0/3094472001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8600101890 0x7f8600101d10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 --2- 172.21.15.38:0/3094472001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86001006a0 0x7f8600100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 -- 172.21.15.38:0/3094472001 >> 172.21.15.38:0/3094472001 conn(0x7f86000fbc30 msgr2=0x7f86000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 -- 172.21.15.38:0/3094472001 shutdown_connections 2024-04-21T23:59:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.680+0000 7f860589d640 1 -- 172.21.15.38:0/3094472001 wait complete. 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 Processor -- start 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 -- start start 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 0x7f8600102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 0x7f86001031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f860010c370 con 0x7f86001006a0 2024-04-21T23:59:29.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f860010c1f0 con 0x7f8600101890 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f860589d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f860010c4f0 con 0x7f8600102250 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47260/0 (socket says 172.21.15.38:47260) 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 -- 172.21.15.38:0/10811189 learned_addr learned my addr 172.21.15.38:0/10811189 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85ff7fe640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 0x7f86001031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85f7fff640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 0x7f8600102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 -- 172.21.15.38:0/10811189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 msgr2=0x7f86001031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 0x7f86001031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 -- 172.21.15.38:0/10811189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 msgr2=0x7f8600102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 0x7f8600102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85feffd640 1 -- 172.21.15.38:0/10811189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8600103a40 con 0x7f86001006a0 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.681+0000 7f85f7fff640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 0x7f8600102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:29.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f85feffd640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85f000e9c0 tx=0x7f85f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f000cd80 con 0x7f86001006a0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85f00035f0 con 0x7f86001006a0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0010690 con 0x7f86001006a0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86001a1bc0 con 0x7f86001006a0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.682+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86001a20c0 con 0x7f86001006a0 2024-04-21T23:59:29.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.683+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86001a22f0 con 0x7f86001006a0 2024-04-21T23:59:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.684+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f85f00037a0 con 0x7f86001006a0 2024-04-21T23:59:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.685+0000 7f860489b640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 0x7f85ec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.685+0000 7f85f7fff640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 0x7f85ec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.685+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f85f00cf150 con 0x7f86001006a0 2024-04-21T23:59:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.685+0000 7f85f7fff640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 0x7f85ec0a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f85e80096c0 tx=0x7f85e802df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:29.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.688+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f85f009c5e0 con 0x7f86001006a0 2024-04-21T23:59:29.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:29 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6840/3401657425,v1:172.21.15.38:6841/3401657425] up:boot 2024-04-21T23:59:29.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:29 smithi203 ceph-mon[31431]: fsmap 11 up:standby 2024-04-21T23:59:29.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:29 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:59:29.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:29 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6840/3401657425,v1:172.21.15.38:6841/3401657425] up:boot 2024-04-21T23:59:29.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:29 smithi111 ceph-mon[31543]: fsmap 11 up:standby 2024-04-21T23:59:29.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:29 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:59:29.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.913+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86001a24c0 con 0x7f86001006a0 2024-04-21T23:59:29.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.914+0000 7f860489b640 1 -- 172.21.15.38:0/10811189 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f85f009bee0 con 0x7f86001006a0 2024-04-21T23:59:29.906 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:29.907 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:59:29.907 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 12 2024-04-21T23:59:29.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 msgr2=0x7f85ec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 0x7f85ec0a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f85e80096c0 tx=0x7f85e802df90 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 msgr2=0x7f8600104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85f000e9c0 tx=0x7f85f000ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 shutdown_connections 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f85ec0a2c10 0x7f85ec0a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8600102250 0x7f86001031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8600101890 0x7f8600102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 --2- 172.21.15.38:0/10811189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f86001006a0 0x7f8600104630 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 >> 172.21.15.38:0/10811189 conn(0x7f86000fbc30 msgr2=0x7f86000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 shutdown_connections 2024-04-21T23:59:29.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:29.916+0000 7f860589d640 1 -- 172.21.15.38:0/10811189 wait complete. 2024-04-21T23:59:29.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:29 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6840/3401657425,v1:172.21.15.38:6841/3401657425] up:boot 2024-04-21T23:59:29.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:29 smithi038 ceph-mon[27972]: fsmap 11 up:standby 2024-04-21T23:59:29.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:29 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:59:30.524 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 systemd[1]: Starting Ceph mds.b for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-21T23:59:30.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:30 smithi203 ceph-mon[31431]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:30 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/10811189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:30.786 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:30 smithi111 ceph-mon[31543]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:30.786 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:30 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/10811189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:30.786 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 podman[61589]: 2024-04-21 23:59:30.541515893 +0000 UTC m=+0.113413578 container create bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:59:30.787 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 podman[61589]: 2024-04-21 23:59:30.446136369 +0000 UTC m=+0.018034057 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:59:30.787 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 podman[61589]: 2024-04-21 23:59:30.769215482 +0000 UTC m=+0.341113167 container init bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T23:59:30.787 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 podman[61589]: 2024-04-21 23:59:30.771944189 +0000 UTC m=+0.343841876 container start bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T23:59:30.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:30 smithi038 ceph-mon[27972]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:30.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:30 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/10811189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:31.076 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 ceph-mds[61608]: -- 172.21.15.111:0/1416523006 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559210019a40 con 0x55920ffec800 2024-04-21T23:59:31.076 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: starting mds.b at 2024-04-21T23:59:31.077 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 bash[61589]: bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 2024-04-21T23:59:31.077 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:30 smithi111 systemd[1]: Started Ceph mds.b for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-21T23:59:31.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.034+0000 7fe5a91bf640 1 -- 172.21.15.38:0/614865115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 msgr2=0x7fe5a4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.034+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/614865115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4101cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe598009900 tx=0x7fe59802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 -- 172.21.15.38:0/614865115 shutdown_connections 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/614865115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/614865115 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4101cf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/614865115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 -- 172.21.15.38:0/614865115 >> 172.21.15.38:0/614865115 conn(0x7fe5a40fbbf0 msgr2=0x7fe5a40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 -- 172.21.15.38:0/614865115 shutdown_connections 2024-04-21T23:59:32.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 -- 172.21.15.38:0/614865115 wait complete. 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.035+0000 7fe5a91bf640 1 Processor -- start 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 -- start start 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a410c350 con 0x7fe5a4102230 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a410c1d0 con 0x7fe5a4101870 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a91bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a410c4d0 con 0x7fe5a4100680 2024-04-21T23:59:32.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47282/0 (socket says 172.21.15.38:47282) 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a359e640 1 -- 172.21.15.38:0/719884643 learned_addr learned my addr 172.21.15.38:0/719884643 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a259c640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 -- 172.21.15.38:0/719884643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 msgr2=0x7fe5a4102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 -- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 msgr2=0x7fe5a4103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a2d9d640 1 -- 172.21.15.38:0/719884643 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a4103a20 con 0x7fe5a4100680 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.036+0000 7fe5a359e640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe5a2d9d640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4104610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe59400e9c0 tx=0x7fe59400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59400ccc0 con 0x7fe5a4100680 2024-04-21T23:59:32.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5940035f0 con 0x7fe5a4100680 2024-04-21T23:59:32.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a41a1c00 con 0x7fe5a4100680 2024-04-21T23:59:32.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59400ce60 con 0x7fe5a4100680 2024-04-21T23:59:32.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.037+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a41a20d0 con 0x7fe5a4100680 2024-04-21T23:59:32.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.038+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a4068560 con 0x7fe5a4100680 2024-04-21T23:59:32.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.039+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe5940037a0 con 0x7fe5a4100680 2024-04-21T23:59:32.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.039+0000 7fe58bfff640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 0x7fe58c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.040+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe5940cf150 con 0x7fe5a4100680 2024-04-21T23:59:32.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.040+0000 7fe5a259c640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 0x7fe58c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.040+0000 7fe5a259c640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 0x7fe58c0a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe59802c240 tx=0x7fe59802df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.043+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe59409c4b0 con 0x7fe5a4100680 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:boot 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: fsmap 12 up:standby 2024-04-21T23:59:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:31 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:boot 2024-04-21T23:59:32.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: fsmap 12 up:standby 2024-04-21T23:59:32.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:31 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:59:32.261 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.261 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.261 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:boot 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: fsmap 12 up:standby 2024-04-21T23:59:32.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:31 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:59:32.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.275+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5a41a22a0 con 0x7fe5a4100680 2024-04-21T23:59:32.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.276+0000 7fe58bfff640 1 -- 172.21.15.38:0/719884643 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7fe59409bdb0 con 0x7fe5a4100680 2024-04-21T23:59:32.269 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:32.269 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T23:59:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 13 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 msgr2=0x7fe58c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 0x7fe58c0a5050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe59802c240 tx=0x7fe59802df90 comp rx=0 tx=0).stop 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 msgr2=0x7fe5a4104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4104610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe59400e9c0 tx=0x7fe59400ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 shutdown_connections 2024-04-21T23:59:32.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe58c0a2b90 0x7fe58c0a5050 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4102230 0x7fe5a4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5a4101870 0x7fe5a4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 --2- 172.21.15.38:0/719884643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5a4100680 0x7fe5a4104610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 >> 172.21.15.38:0/719884643 conn(0x7fe5a40fbbf0 msgr2=0x7fe5a40fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 shutdown_connections 2024-04-21T23:59:32.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.278+0000 7fe5a91bf640 1 -- 172.21.15.38:0/719884643 wait complete. 2024-04-21T23:59:32.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 -- 172.21.15.38:0/4118371323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb50102250 msgr2=0x7fbb50108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 --2- 172.21.15.38:0/4118371323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb50102250 0x7fbb50108ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009ca0 tx=0x7fbb4002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 -- 172.21.15.38:0/4118371323 shutdown_connections 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 --2- 172.21.15.38:0/4118371323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb50102250 0x7fbb50108ae0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 --2- 172.21.15.38:0/4118371323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 0x7fbb50101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 --2- 172.21.15.38:0/4118371323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb501006a0 0x7fbb50100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 -- 172.21.15.38:0/4118371323 >> 172.21.15.38:0/4118371323 conn(0x7fbb500fbc30 msgr2=0x7fbb500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 -- 172.21.15.38:0/4118371323 shutdown_connections 2024-04-21T23:59:32.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 -- 172.21.15.38:0/4118371323 wait complete. 2024-04-21T23:59:32.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.394+0000 7fbb5518a640 1 Processor -- start 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 -- start start 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 0x7fbb5019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb50102250 0x7fbb5019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010c300 con 0x7fbb501006a0 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010c180 con 0x7fbb50102250 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb5518a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010c480 con 0x7fbb50101890 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47298/0 (socket says 172.21.15.38:47298) 2024-04-21T23:59:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 -- 172.21.15.38:0/739840555 learned_addr learned my addr 172.21.15.38:0/739840555 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4e59c640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 0x7fbb5019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4f59e640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb50102250 0x7fbb5019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 -- 172.21.15.38:0/739840555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 msgr2=0x7fbb5019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 0x7fbb5019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 -- 172.21.15.38:0/739840555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb50102250 msgr2=0x7fbb5019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb50102250 0x7fbb5019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 -- 172.21.15.38:0/739840555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5019f670 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.395+0000 7fbb4ed9d640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c970 tx=0x7fbb4800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.396+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004220 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.396+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5019f960 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.396+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb480043c0 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.396+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4800f450 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.396+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5019ff00 con 0x7fbb501006a0 2024-04-21T23:59:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.397+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb500689b0 con 0x7fbb501006a0 2024-04-21T23:59:32.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.398+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbb480079b0 con 0x7fbb501006a0 2024-04-21T23:59:32.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.399+0000 7fbb47fff640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 0x7fbb3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.399+0000 7fbb4e59c640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 0x7fbb3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.399+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbb480ced20 con 0x7fbb501006a0 2024-04-21T23:59:32.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.399+0000 7fbb4e59c640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 0x7fbb3c0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb5006a6e0 tx=0x7fbb3800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.402+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbb48014030 con 0x7fbb501006a0 2024-04-21T23:59:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.599+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb5002d170 con 0x7fbb501006a0 2024-04-21T23:59:32.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.600+0000 7fbb47fff640 1 -- 172.21.15.38:0/739840555 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18441 (secure 0 0 0) 0x7fbb4809ba30 con 0x7fbb501006a0 2024-04-21T23:59:32.594 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:32.594 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":81,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:58:48.433601+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 msgr2=0x7fbb3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 0x7fbb3c0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb5006a6e0 tx=0x7fbb3800b390 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 msgr2=0x7fbb5006a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c970 tx=0x7fbb4800ce30 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 shutdown_connections 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbb3c0a2b90 0x7fbb3c0a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb50102250 0x7fbb5019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb50101890 0x7fbb5019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 --2- 172.21.15.38:0/739840555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb501006a0 0x7fbb5006a970 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 >> 172.21.15.38:0/739840555 conn(0x7fbb500fbc30 msgr2=0x7fbb50104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 shutdown_connections 2024-04-21T23:59:32.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.603+0000 7fbb5518a640 1 -- 172.21.15.38:0/739840555 wait complete. 2024-04-21T23:59:32.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:59:32.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:59:32.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 -- 172.21.15.38:0/3135037334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f4101890 msgr2=0x7fa6f4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 --2- 172.21.15.38:0/3135037334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f4101890 0x7fa6f4101d10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009900 tx=0x7fa6e802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:32.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 -- 172.21.15.38:0/3135037334 shutdown_connections 2024-04-21T23:59:32.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 --2- 172.21.15.38:0/3135037334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 0x7fa6f4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 --2- 172.21.15.38:0/3135037334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f4101890 0x7fa6f4101d10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 --2- 172.21.15.38:0/3135037334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f41006a0 0x7fa6f4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 -- 172.21.15.38:0/3135037334 >> 172.21.15.38:0/3135037334 conn(0x7fa6f40fbc30 msgr2=0x7fa6f40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 -- 172.21.15.38:0/3135037334 shutdown_connections 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 -- 172.21.15.38:0/3135037334 wait complete. 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.724+0000 7fa6faa39640 1 Processor -- start 2024-04-21T23:59:32.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 -- start start 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 0x7fa6f4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 0x7fa6f41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f410c320 con 0x7fa6f41006a0 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f410c1a0 con 0x7fa6f4101890 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6faa39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f410c4a0 con 0x7fa6f4102250 2024-04-21T23:59:32.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47318/0 (socket says 172.21.15.38:47318) 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 -- 172.21.15.38:0/1253512046 learned_addr learned my addr 172.21.15.38:0/1253512046 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f37fe640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 0x7fa6f4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f8fd6640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 0x7fa6f41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 -- 172.21.15.38:0/1253512046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 msgr2=0x7fa6f41031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 0x7fa6f41031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 -- 172.21.15.38:0/1253512046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 msgr2=0x7fa6f4102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 0x7fa6f4102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f3fff640 1 -- 172.21.15.38:0/1253512046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f4103a40 con 0x7fa6f41006a0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.725+0000 7fa6f37fe640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 0x7fa6f4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6f3fff640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400e9c0 tx=0x7fa6e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400cd80 con 0x7fa6f41006a0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f41a1c60 con 0x7fa6f41006a0 2024-04-21T23:59:32.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6e40035f0 con 0x7fa6f41006a0 2024-04-21T23:59:32.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4010690 con 0x7fa6f41006a0 2024-04-21T23:59:32.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.726+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f41a21d0 con 0x7fa6f41006a0 2024-04-21T23:59:32.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.727+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f41a23a0 con 0x7fa6f41006a0 2024-04-21T23:59:32.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.728+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa6e40037a0 con 0x7fa6f41006a0 2024-04-21T23:59:32.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.729+0000 7fa6f17fa640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 0x7fa6dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:32.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.729+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6e40cf050 con 0x7fa6f41006a0 2024-04-21T23:59:32.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.729+0000 7fa6f37fe640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 0x7fa6dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:32.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.730+0000 7fa6f37fe640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 0x7fa6dc0a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009230 tx=0x7fa6e8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:32.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.735+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa6e409c3b0 con 0x7fa6f41006a0 2024-04-21T23:59:32.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.931+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa6f41a2570 con 0x7fa6f41006a0 2024-04-21T23:59:32.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.936+0000 7fa6f17fa640 1 -- 172.21.15.38:0/1253512046 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fa6e409bcb0 con 0x7fa6f41006a0 2024-04-21T23:59:32.929 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-21T23:59:32.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.938+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 msgr2=0x7fa6dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.938+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 0x7fa6dc0a5050 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009230 tx=0x7fa6e8005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.938+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 msgr2=0x7fa6f4104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.938+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400e9c0 tx=0x7fa6e400ee80 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 shutdown_connections 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa6dc0a2b90 0x7fa6dc0a5050 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6f4102250 0x7fa6f41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f4101890 0x7fa6f4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 --2- 172.21.15.38:0/1253512046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f41006a0 0x7fa6f4104630 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 >> 172.21.15.38:0/1253512046 conn(0x7fa6f40fbc30 msgr2=0x7fa6f40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 shutdown_connections 2024-04-21T23:59:32.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:32.939+0000 7fa6faa39640 1 -- 172.21.15.38:0/1253512046 wait complete. 2024-04-21T23:59:32.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.055+0000 7f7c94035640 1 -- 172.21.15.38:0/3424500770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 msgr2=0x7f7c8c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.055+0000 7f7c94035640 1 --2- 172.21.15.38:0/3424500770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c108ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009be0 tx=0x7f7c8402bef0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.055+0000 7f7c94035640 1 -- 172.21.15.38:0/3424500770 shutdown_connections 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.055+0000 7f7c94035640 1 --2- 172.21.15.38:0/3424500770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c108ac0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 --2- 172.21.15.38:0/3424500770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 0x7f7c8c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 --2- 172.21.15.38:0/3424500770 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- 172.21.15.38:0/3424500770 >> 172.21.15.38:0/3424500770 conn(0x7f7c8c0fbbf0 msgr2=0x7f7c8c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- 172.21.15.38:0/3424500770 shutdown_connections 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- 172.21.15.38:0/3424500770 wait complete. 2024-04-21T23:59:33.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 Processor -- start 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- start start 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 0x7f7c8c196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10c300 con 0x7f7c8c100680 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10c180 con 0x7f7c8c102230 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c94035640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10c480 con 0x7f7c8c101870 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.056+0000 7f7c91dd1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47336/0 (socket says 172.21.15.38:47336) 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 -- 172.21.15.38:0/2350452790 learned_addr learned my addr 172.21.15.38:0/2350452790 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c925d2640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c915d0640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 0x7f7c8c196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 -- 172.21.15.38:0/2350452790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 msgr2=0x7f7c8c196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 0x7f7c8c196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 -- 172.21.15.38:0/2350452790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 msgr2=0x7f7c8c19def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c19def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 -- 172.21.15.38:0/2350452790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c19e560 con 0x7f7c8c100680 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c925d2640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c19def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c91dd1640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9a0 tx=0x7f7c8800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8800cd80 con 0x7f7c8c100680 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c19e850 con 0x7f7c8c100680 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c88010910 con 0x7f7c8c100680 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88018a40 con 0x7f7c8c100680 2024-04-21T23:59:33.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.057+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c19edf0 con 0x7f7c8c100680 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.058+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c8c1079a0 con 0x7f7c8c100680 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.060+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7c88010ac0 con 0x7f7c8c100680 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.060+0000 7f7c82ffd640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 0x7f7c780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.060+0000 7f7c915d0640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 0x7f7c780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.060+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7c880cef80 con 0x7f7c8c100680 2024-04-21T23:59:33.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.061+0000 7f7c915d0640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 0x7f7c780a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c197e50 tx=0x7f7c7c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:33.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.068+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7c880536f0 con 0x7f7c8c100680 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/719884643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/739840555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:32 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1253512046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:33.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.258+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7c8c02d380 con 0x7f7c8c100680 2024-04-21T23:59:33.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.258+0000 7f7c82ffd640 1 -- 172.21.15.38:0/2350452790 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f7c8809c380 con 0x7f7c8c100680 2024-04-21T23:59:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:33.251 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2024-04-21T23:59:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-21T23:59:33.251 INFO:teuthology.orchestra.run.smithi038.stdout: "poolnum": 1, 2024-04-21T23:59:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "poolname": ".mgr" 2024-04-21T23:59:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-21T23:59:33.252 INFO:teuthology.orchestra.run.smithi038.stdout:] 2024-04-21T23:59:33.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 msgr2=0x7f7c780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 0x7f7c780a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c197e50 tx=0x7f7c7c005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 msgr2=0x7f7c8c196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9a0 tx=0x7f7c8800ee60 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 shutdown_connections 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c780a2b90 0x7f7c780a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c102230 0x7f7c8c19def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c101870 0x7f7c8c196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 --2- 172.21.15.38:0/2350452790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c100680 0x7f7c8c196940 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.260+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 >> 172.21.15.38:0/2350452790 conn(0x7f7c8c0fbbf0 msgr2=0x7f7c8c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.261+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 shutdown_connections 2024-04-21T23:59:33.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.261+0000 7f7c94035640 1 -- 172.21.15.38:0/2350452790 wait complete. 2024-04-21T23:59:33.264 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T23:59:33.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T23:59:33.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:33.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/719884643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/739840555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:32 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1253512046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:33.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/719884643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/739840555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:33.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:32 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1253512046' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 -- 172.21.15.38:0/3920752315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 msgr2=0x7f839c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 --2- 172.21.15.38:0/3920752315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 0x7f839c101cf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8384009900 tx=0x7f838402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 -- 172.21.15.38:0/3920752315 shutdown_connections 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 --2- 172.21.15.38:0/3920752315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c102230 0x7f839c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 --2- 172.21.15.38:0/3920752315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c101870 0x7f839c101cf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 --2- 172.21.15.38:0/3920752315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c100680 0x7f839c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.378+0000 7f83a27b8640 1 -- 172.21.15.38:0/3920752315 >> 172.21.15.38:0/3920752315 conn(0x7f839c0fbbf0 msgr2=0x7f839c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- 172.21.15.38:0/3920752315 shutdown_connections 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- 172.21.15.38:0/3920752315 wait complete. 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 Processor -- start 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- start start 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 0x7f839c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c300 con 0x7f839c100680 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c180 con 0x7f839c102230 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.379+0000 7f83a27b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10c480 con 0x7f839c101870 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47354/0 (socket says 172.21.15.38:47354) 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 -- 172.21.15.38:0/4199793915 learned_addr learned my addr 172.21.15.38:0/4199793915 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839b7fe640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83a0d55640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 0x7f839c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 -- 172.21.15.38:0/4199793915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 msgr2=0x7f839c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 -- 172.21.15.38:0/4199793915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 msgr2=0x7f839c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 0x7f839c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 -- 172.21.15.38:0/4199793915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c19f760 con 0x7f839c100680 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f839bfff640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f838c00d770 tx=0x7f838c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c017070 con 0x7f839c100680 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c19f9f0 con 0x7f839c100680 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838c009c30 con 0x7f839c100680 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c003dd0 con 0x7f839c100680 2024-04-21T23:59:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.380+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c19ff60 con 0x7f839c100680 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.381+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c1079c0 con 0x7f839c100680 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.382+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f838c003f70 con 0x7f839c100680 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.383+0000 7f83997fa640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 0x7f83880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.383+0000 7f839b7fe640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 0x7f83880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.383+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f838c0ce550 con 0x7f839c100680 2024-04-21T23:59:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.384+0000 7f839b7fe640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 0x7f83880a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f838402c7b0 tx=0x7f8384005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:33.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.388+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f838c009de0 con 0x7f839c100680 2024-04-21T23:59:33.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:33.577+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f839c062340 con 0x7f839c100680 2024-04-21T23:59:34.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2350452790' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:59:34.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:34.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:34 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:34.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2350452790' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:59:34.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:34.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:34 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2350452790' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:34 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:34.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.588+0000 7f83997fa640 1 -- 172.21.15.38:0/4199793915 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f838c09b260 con 0x7f839c100680 2024-04-21T23:59:34.581 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' created 2024-04-21T23:59:34.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.590+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 msgr2=0x7f83880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:34.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.590+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 0x7f83880a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f838402c7b0 tx=0x7f8384005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.590+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 msgr2=0x7f839c069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.590+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f838c00d770 tx=0x7f838c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 shutdown_connections 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f83880a2b90 0x7f83880a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f839c102230 0x7f839c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f839c101870 0x7f839c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 --2- 172.21.15.38:0/4199793915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f839c100680 0x7f839c069440 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 >> 172.21.15.38:0/4199793915 conn(0x7f839c0fbbf0 msgr2=0x7f839c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 shutdown_connections 2024-04-21T23:59:34.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.591+0000 7f83a27b8640 1 -- 172.21.15.38:0/4199793915 wait complete. 2024-04-21T23:59:34.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 -- 172.21.15.38:0/2698043505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 msgr2=0x7f2bc0101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/2698043505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0101b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009900 tx=0x7f2bbc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 -- 172.21.15.38:0/2698043505 shutdown_connections 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/2698043505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 0x7f2bc0109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/2698043505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 0x7f2bc0102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/2698043505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0101b90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 -- 172.21.15.38:0/2698043505 >> 172.21.15.38:0/2698043505 conn(0x7f2bc00fcd00 msgr2=0x7f2bc00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:34.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 -- 172.21.15.38:0/2698043505 shutdown_connections 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.718+0000 7f2bc87d7640 1 -- 172.21.15.38:0/2698043505 wait complete. 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 Processor -- start 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 -- start start 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 0x7f2bc0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 0x7f2bc019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010d3c0 con 0x7f2bc0101770 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010d240 con 0x7f2bc0102960 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc87d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010d540 con 0x7f2bc0103320 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43088/0 (socket says 172.21.15.38:43088) 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 -- 172.21.15.38:0/3489501544 learned_addr learned my addr 172.21.15.38:0/3489501544 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6d74640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 0x7f2bc019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:34.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc5d72640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 0x7f2bc0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 -- 172.21.15.38:0/3489501544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 msgr2=0x7f2bc019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 0x7f2bc019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 -- 172.21.15.38:0/3489501544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 msgr2=0x7f2bc0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 0x7f2bc0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 -- 172.21.15.38:0/3489501544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc019f680 con 0x7f2bc0101770 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.719+0000 7f2bc6573640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005ab0 tx=0x7f2bbc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.720+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc041070 con 0x7f2bc0101770 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.720+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bbc0054e0 con 0x7f2bc0101770 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.720+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc034440 con 0x7f2bc0101770 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.720+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc019f910 con 0x7f2bc0101770 2024-04-21T23:59:34.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.720+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc019fe50 con 0x7f2bc0101770 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.721+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc0108b10 con 0x7f2bc0101770 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.722+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2bbc044070 con 0x7f2bc0101770 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.722+0000 7f2bb77fe640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 0x7f2bac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.723+0000 7f2bc5d72640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 0x7f2bac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.723+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f2bbc0f09f0 con 0x7f2bc0101770 2024-04-21T23:59:34.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.723+0000 7f2bc5d72640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 0x7f2bac0a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0198f70 tx=0x7f2bb000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:34.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.726+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2bbc0bd630 con 0x7f2bc0101770 2024-04-21T23:59:34.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:34.915+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f2bc002d170 con 0x7f2bc0101770 2024-04-21T23:59:35.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.594+0000 7f2bb77fe640 1 -- 172.21.15.38:0/3489501544 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f2bbc0c1930 con 0x7f2bc0101770 2024-04-21T23:59:35.668 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' created 2024-04-21T23:59:35.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 msgr2=0x7f2bac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 0x7f2bac0a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0198f70 tx=0x7f2bb000b410 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 msgr2=0x7f2bc0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005ab0 tx=0x7f2bbc005dc0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 shutdown_connections 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2bac0a2c10 0x7f2bac0a50d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc0103320 0x7f2bc019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc0102960 0x7f2bc0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 --2- 172.21.15.38:0/3489501544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2bc0101770 0x7f2bc0197a60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 >> 172.21.15.38:0/3489501544 conn(0x7f2bc00fcd00 msgr2=0x7f2bc0105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 shutdown_connections 2024-04-21T23:59:35.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.597+0000 7f2bc87d7640 1 -- 172.21.15.38:0/3489501544 wait complete. 2024-04-21T23:59:35.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T23:59:35.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:35.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:59:35.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:35.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:35.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:59:35.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:35 smithi038 ceph-mon[27972]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:59:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:59:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:35.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:59:35.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:35 smithi203 ceph-mon[31431]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 -- 172.21.15.38:0/3733664580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 msgr2=0x7fd5fc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 --2- 172.21.15.38:0/3733664580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc101bb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009900 tx=0x7fd5ec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 -- 172.21.15.38:0/3733664580 shutdown_connections 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 --2- 172.21.15.38:0/3733664580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc103340 0x7fd5fc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 --2- 172.21.15.38:0/3733664580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc102980 0x7fd5fc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 --2- 172.21.15.38:0/3733664580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc101bb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 -- 172.21.15.38:0/3733664580 >> 172.21.15.38:0/3733664580 conn(0x7fd5fc0fcd20 msgr2=0x7fd5fc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 -- 172.21.15.38:0/3733664580 shutdown_connections 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.793+0000 7fd601e20640 1 -- 172.21.15.38:0/3733664580 wait complete. 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 Processor -- start 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 -- start start 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 0x7fd5fc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 0x7fd5fc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d450 con 0x7fd5fc101790 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d2d0 con 0x7fd5fc102980 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd601e20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc10d5d0 con 0x7fd5fc103340 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:35.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43112/0 (socket says 172.21.15.38:43112) 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 -- 172.21.15.38:0/240443801 learned_addr learned my addr 172.21.15.38:0/240443801 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5faffd640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 0x7fd5fc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fbfff640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 0x7fd5fc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 -- 172.21.15.38:0/240443801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 msgr2=0x7fd5fc074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 0x7fd5fc074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 -- 172.21.15.38:0/240443801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 msgr2=0x7fd5fc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 0x7fd5fc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fb7fe640 1 -- 172.21.15.38:0/240443801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc074900 con 0x7fd5fc101790 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5fbfff640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 0x7fd5fc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:35.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.794+0000 7fd5faffd640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 0x7fd5fc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd5fb7fe640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec043020 con 0x7fd5fc101790 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc1a1c40 con 0x7fd5fc101790 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec0054e0 con 0x7fd5fc101790 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec003ce0 con 0x7fd5fc101790 2024-04-21T23:59:35.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.795+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc1a2160 con 0x7fd5fc101790 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.796+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc108bd0 con 0x7fd5fc101790 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.798+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd5ec046020 con 0x7fd5fc101790 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.798+0000 7fd5f8ff9640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.799+0000 7fd5faffd640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.799+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fd5ec0f14d0 con 0x7fd5fc101790 2024-04-21T23:59:35.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.799+0000 7fd5faffd640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 0x7fd5e80a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc075260 tx=0x7fd5e4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:35.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:35.802+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd5ec04d020 con 0x7fd5fc101790 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4199793915' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:59:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:35 smithi111 ceph-mon[31543]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:59:36.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.027+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd5fc02d170 con 0x7fd5fc101790 2024-04-21T23:59:36.681 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:59:36.681 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:59:36.681 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:59:36.681 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:59:36.682 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:36 smithi203 ceph-mon[31431]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:59:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:36 smithi038 ceph-mon[27972]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:59:36.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.773+0000 7fd5f8ff9640 1 -- 172.21.15.38:0/240443801 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fd5ec0be5e0 con 0x7fd5fc101790 2024-04-21T23:59:36.766 INFO:teuthology.orchestra.run.smithi038.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 msgr2=0x7fd5e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 0x7fd5e80a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc075260 tx=0x7fd5e4005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 msgr2=0x7fd5fc0754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec02c6c0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 shutdown_connections 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5fc103340 0x7fd5fc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5fc102980 0x7fd5fc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 --2- 172.21.15.38:0/240443801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5fc101790 0x7fd5fc0754f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 >> 172.21.15.38:0/240443801 conn(0x7fd5fc0fcd20 msgr2=0x7fd5fc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 shutdown_connections 2024-04-21T23:59:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.776+0000 7fd601e20640 1 -- 172.21.15.38:0/240443801 wait complete. 2024-04-21T23:59:36.785 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T23:59:36.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3489501544' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:59:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:59:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-21T23:59:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:59:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:59:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:36 smithi111 ceph-mon[31543]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:59:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 -- 172.21.15.38:0/3603176677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 msgr2=0x7f9d50100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3603176677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d50100dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009900 tx=0x7f9d4002bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 -- 172.21.15.38:0/3603176677 shutdown_connections 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3603176677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 0x7f9d50101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3603176677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d50100dd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3603176677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 -- 172.21.15.38:0/3603176677 >> 172.21.15.38:0/3603176677 conn(0x7f9d500fbc30 msgr2=0x7f9d500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 -- 172.21.15.38:0/3603176677 shutdown_connections 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.907+0000 7f9d581e7640 1 -- 172.21.15.38:0/3603176677 wait complete. 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 Processor -- start 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 -- start start 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 0x7f9d501a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5010f5e0 con 0x7f9d50100970 2024-04-21T23:59:36.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5010f460 con 0x7f9d50101310 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d581e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5010f760 con 0x7f9d50100010 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43124/0 (socket says 172.21.15.38:43124) 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 -- 172.21.15.38:0/3566920764 learned_addr learned my addr 172.21.15.38:0/3566920764 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d56784640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 0x7f9d501a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55f83640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 -- 172.21.15.38:0/3566920764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 msgr2=0x7f9d50199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 -- 172.21.15.38:0/3566920764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 msgr2=0x7f9d501a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 0x7f9d501a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 -- 172.21.15.38:0/3566920764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d501a18d0 con 0x7f9d50100970 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55f83640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:36.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.908+0000 7f9d55782640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009230 tx=0x7f9d40004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.909+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40041070 con 0x7f9d50100970 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.909+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d501a1b60 con 0x7f9d50100970 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.909+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d400054e0 con 0x7f9d50100970 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.909+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d400346a0 con 0x7f9d50100970 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.909+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d501a2050 con 0x7f9d50100970 2024-04-21T23:59:36.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.910+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5010aee0 con 0x7f9d50100970 2024-04-21T23:59:36.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.911+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9d40044070 con 0x7f9d50100970 2024-04-21T23:59:36.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.911+0000 7f9d46ffd640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 0x7f9d3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:36.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.911+0000 7f9d55f83640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 0x7f9d3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:36.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.911+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f9d400f0cf0 con 0x7f9d50100970 2024-04-21T23:59:36.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.912+0000 7f9d55f83640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 0x7f9d3c0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c005fd0 tx=0x7f9d4c008310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:36.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:36.916+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9d400f5070 con 0x7f9d50100970 2024-04-21T23:59:37.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.109+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f9d500623e0 con 0x7f9d50100970 2024-04-21T23:59:37.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.234+0000 7f9d46ffd640 1 -- 172.21.15.38:0/3566920764 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f9d400bdda0 con 0x7f9d50100970 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 msgr2=0x7f9d3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 0x7f9d3c0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c005fd0 tx=0x7f9d4c008310 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 msgr2=0x7f9d5019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009230 tx=0x7f9d40004040 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 shutdown_connections 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9d3c0a2b90 0x7f9d3c0a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d50101310 0x7f9d501a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9d50100970 0x7f9d5019a1d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 --2- 172.21.15.38:0/3566920764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d50100010 0x7f9d50199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 >> 172.21.15.38:0/3566920764 conn(0x7f9d500fbc30 msgr2=0x7f9d500fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 shutdown_connections 2024-04-21T23:59:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.236+0000 7f9d581e7640 1 -- 172.21.15.38:0/3566920764 wait complete. 2024-04-21T23:59:37.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T23:59:37.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 -- 172.21.15.38:0/2186077870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed41017b0 msgr2=0x7f0ed4101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/2186077870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed41017b0 0x7f0ed4101bd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009900 tx=0x7f0ed002bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 -- 172.21.15.38:0/2186077870 shutdown_connections 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/2186077870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed4103360 0x7f0ed4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/2186077870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 0x7f0ed4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/2186077870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed41017b0 0x7f0ed4101bd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 -- 172.21.15.38:0/2186077870 >> 172.21.15.38:0/2186077870 conn(0x7f0ed40fcd40 msgr2=0x7f0ed40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 -- 172.21.15.38:0/2186077870 shutdown_connections 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.358+0000 7f0edc0a7640 1 -- 172.21.15.38:0/2186077870 wait complete. 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 Processor -- start 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 -- start start 2024-04-21T23:59:37.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed41017b0 0x7f0ed4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 0x7f0ed4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed410d400 con 0x7f0ed4103360 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed410d280 con 0x7f0ed41017b0 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0edc0a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed410d580 con 0x7f0ed41029a0 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43138/0 (socket says 172.21.15.38:43138) 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 -- 172.21.15.38:0/1607882730 learned_addr learned my addr 172.21.15.38:0/1607882730 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0ed9e43640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed41017b0 0x7f0ed4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0ed9642640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 0x7f0ed4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 -- 172.21.15.38:0/1607882730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 msgr2=0x7f0ed4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 0x7f0ed4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 -- 172.21.15.38:0/1607882730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed41017b0 msgr2=0x7f0ed4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:37.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed41017b0 0x7f0ed4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:37.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.359+0000 7f0eda644640 1 -- 172.21.15.38:0/1607882730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed419f6f0 con 0x7f0ed4103360 2024-04-21T23:59:37.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.360+0000 7f0eda644640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00cad0 tx=0x7f0ecc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:37.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.366+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00ae20 con 0x7f0ed4103360 2024-04-21T23:59:37.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.367+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc005020 con 0x7f0ed4103360 2024-04-21T23:59:37.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.367+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00f6f0 con 0x7f0ed4103360 2024-04-21T23:59:37.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.367+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed419f9e0 con 0x7f0ed4103360 2024-04-21T23:59:37.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.367+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed419ff30 con 0x7f0ed4103360 2024-04-21T23:59:37.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.368+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed4108b70 con 0x7f0ed4103360 2024-04-21T23:59:37.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.369+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0ecc0051d0 con 0x7f0ed4103360 2024-04-21T23:59:37.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.369+0000 7f0ecaffd640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 0x7f0ec00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:37.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.369+0000 7f0ed9e43640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 0x7f0ec00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:37.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.369+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f0ecc0cefe0 con 0x7f0ed4103360 2024-04-21T23:59:37.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.370+0000 7f0ed9e43640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 0x7f0ec00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000c00 tx=0x7f0ed0004f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:37.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.374+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0ecc014030 con 0x7f0ed4103360 2024-04-21T23:59:37.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:37.558+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0ed402d3c0 con 0x7f0ed4103360 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: pgmap v288: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:59:37.698 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:37 smithi203 ceph-mon[31431]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:59:37.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: pgmap v288: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:59:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:37 smithi038 ceph-mon[27972]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:59:37.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: pgmap v288: 97 pgs: 9 creating+peering, 1 active+clean, 87 unknown; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/240443801' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3566920764' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:59:37.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:37 smithi111 ceph-mon[31543]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:59:38.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:38 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:59:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:38 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:active 2024-04-21T23:59:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:38 smithi038 ceph-mon[27972]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:38.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:38 smithi038 ceph-mon[27972]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:38.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:38 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:59:38.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:38 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:active 2024-04-21T23:59:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:38 smithi203 ceph-mon[31431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:38 smithi203 ceph-mon[31431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:38.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:38 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:59:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:38 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] up:active 2024-04-21T23:59:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:38 smithi111 ceph-mon[31543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:38 smithi111 ceph-mon[31543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.373+0000 7f0ecaffd640 1 -- 172.21.15.38:0/1607882730 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f0ecc09bfb0 con 0x7f0ed4103360 2024-04-21T23:59:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data_ec' created 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 msgr2=0x7f0ec00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 0x7f0ec00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000c00 tx=0x7f0ed0004f10 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 msgr2=0x7f0ed419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00cad0 tx=0x7f0ecc00cf90 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 shutdown_connections 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ec00a2c10 0x7f0ec00a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed4103360 0x7f0ed419f080 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ed41029a0 0x7f0ed4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 --2- 172.21.15.38:0/1607882730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed41017b0 0x7f0ed4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 >> 172.21.15.38:0/1607882730 conn(0x7f0ed40fcd40 msgr2=0x7f0ed4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:39.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 shutdown_connections 2024-04-21T23:59:39.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.376+0000 7f0edc0a7640 1 -- 172.21.15.38:0/1607882730 wait complete. 2024-04-21T23:59:39.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:39 smithi038 ceph-mon[27972]: pgmap v291: 97 pgs: 19 creating+peering, 31 active+clean, 47 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-21T23:59:39.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:39 smithi038 ceph-mon[27972]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-21T23:59:39.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:39 smithi038 ceph-mon[27972]: Cluster is now healthy 2024-04-21T23:59:39.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:39 smithi038 ceph-mon[27972]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:59:39.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- 172.21.15.38:0/3215608804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 msgr2=0x7f8b68100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/3215608804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b68100ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009900 tx=0x7f8b5802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- 172.21.15.38:0/3215608804 shutdown_connections 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/3215608804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68102250 0x7f8b68108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/3215608804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68101890 0x7f8b68101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/3215608804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b68100ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- 172.21.15.38:0/3215608804 >> 172.21.15.38:0/3215608804 conn(0x7f8b680fbc30 msgr2=0x7f8b680fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- 172.21.15.38:0/3215608804 shutdown_connections 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- 172.21.15.38:0/3215608804 wait complete. 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 Processor -- start 2024-04-21T23:59:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.502+0000 7f8b6dba0640 1 -- start start 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101890 0x7f8b6819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68102250 0x7f8b6819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6810c2f0 con 0x7f8b681006a0 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6810c170 con 0x7f8b68102250 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6810c470 con 0x7f8b68101890 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43160/0 (socket says 172.21.15.38:43160) 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 -- 172.21.15.38:0/2909331129 learned_addr learned my addr 172.21.15.38:0/2909331129 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b67fff640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68102250 0x7f8b6819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 -- 172.21.15.38:0/2909331129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101890 msgr2=0x7f8b6819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101890 0x7f8b6819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 -- 172.21.15.38:0/2909331129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68102250 msgr2=0x7f8b6819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68102250 0x7f8b6819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 -- 172.21.15.38:0/2909331129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6819f740 con 0x7f8b681006a0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b677fe640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005ab0 tx=0x7f8b58005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58041070 con 0x7f8b681006a0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6819f970 con 0x7f8b681006a0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b580054e0 con 0x7f8b681006a0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.503+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58034440 con 0x7f8b681006a0 2024-04-21T23:59:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.504+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6819fe60 con 0x7f8b681006a0 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.505+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b68107ae0 con 0x7f8b681006a0 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.506+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8b58044070 con 0x7f8b681006a0 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.506+0000 7f8b64ff9640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 0x7f8b500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.506+0000 7f8b66ffd640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 0x7f8b500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.506+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f8b580f1170 con 0x7f8b681006a0 2024-04-21T23:59:39.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.507+0000 7f8b66ffd640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 0x7f8b500a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c005fd0 tx=0x7f8b5c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:39.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.510+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8b580bdeb0 con 0x7f8b681006a0 2024-04-21T23:59:39.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:39.694+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f8b6802d330 con 0x7f8b681006a0 2024-04-21T23:59:39.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:39 smithi203 ceph-mon[31431]: pgmap v291: 97 pgs: 19 creating+peering, 31 active+clean, 47 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-21T23:59:39.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:39 smithi203 ceph-mon[31431]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-21T23:59:39.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:39 smithi203 ceph-mon[31431]: Cluster is now healthy 2024-04-21T23:59:39.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:39 smithi203 ceph-mon[31431]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:59:39.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:39 smithi111 ceph-mon[31543]: pgmap v291: 97 pgs: 19 creating+peering, 31 active+clean, 47 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-21T23:59:39.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:39 smithi111 ceph-mon[31543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-21T23:59:39.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:39 smithi111 ceph-mon[31543]: Cluster is now healthy 2024-04-21T23:59:39.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:39 smithi111 ceph-mon[31543]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:59:40.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.388+0000 7f8b64ff9640 1 -- 172.21.15.38:0/2909331129 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f8b580bd7b0 con 0x7f8b681006a0 2024-04-21T23:59:40.381 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T23:59:40.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 msgr2=0x7f8b500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:40.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 0x7f8b500a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c005fd0 tx=0x7f8b5c005850 comp rx=0 tx=0).stop 2024-04-21T23:59:40.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 msgr2=0x7f8b6806a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:40.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b58005ab0 tx=0x7f8b58005dc0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 shutdown_connections 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8b500a2b90 0x7f8b500a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b68102250 0x7f8b6819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b68101890 0x7f8b6819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 --2- 172.21.15.38:0/2909331129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b681006a0 0x7f8b6806a8b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 >> 172.21.15.38:0/2909331129 conn(0x7f8b680fbc30 msgr2=0x7f8b68104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 shutdown_connections 2024-04-21T23:59:40.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.391+0000 7f8b6dba0640 1 -- 172.21.15.38:0/2909331129 wait complete. 2024-04-21T23:59:40.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T23:59:40.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 -- 172.21.15.38:0/515935266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 msgr2=0x7f621c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:40.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 --2- 172.21.15.38:0/515935266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c108ae0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6214009ca0 tx=0x7f621402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 -- 172.21.15.38:0/515935266 shutdown_connections 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 --2- 172.21.15.38:0/515935266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c108ae0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 --2- 172.21.15.38:0/515935266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c101890 0x7f621c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 --2- 172.21.15.38:0/515935266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c1006a0 0x7f621c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 -- 172.21.15.38:0/515935266 >> 172.21.15.38:0/515935266 conn(0x7f621c0fbc30 msgr2=0x7f621c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 -- 172.21.15.38:0/515935266 shutdown_connections 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.526+0000 7f6222adf640 1 -- 172.21.15.38:0/515935266 wait complete. 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 Processor -- start 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 -- start start 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c1006a0 0x7f621c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c101890 0x7f621c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f621c10d440 con 0x7f621c102250 2024-04-21T23:59:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f621c10d2c0 con 0x7f621c101890 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f6222adf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f621c10d5c0 con 0x7f621c1006a0 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f622107c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f622107c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43184/0 (socket says 172.21.15.38:43184) 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f622107c640 1 -- 172.21.15.38:0/2929335725 learned_addr learned my addr 172.21.15.38:0/2929335725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f622087b640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c1006a0 0x7f621c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.527+0000 7f621bfff640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c101890 0x7f621c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 -- 172.21.15.38:0/2929335725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c1006a0 msgr2=0x7f621c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c1006a0 0x7f621c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 -- 172.21.15.38:0/2929335725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c101890 msgr2=0x7f621c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c101890 0x7f621c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 -- 172.21.15.38:0/2929335725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621c19f610 con 0x7f621c102250 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f622107c640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f621402c700 tx=0x7f6214032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214039040 con 0x7f621c102250 2024-04-21T23:59:40.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6214044070 con 0x7f621c102250 2024-04-21T23:59:40.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62140042a0 con 0x7f621c102250 2024-04-21T23:59:40.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621c19f8a0 con 0x7f621c102250 2024-04-21T23:59:40.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.528+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621c19fd90 con 0x7f621c102250 2024-04-21T23:59:40.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.529+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f621c107960 con 0x7f621c102250 2024-04-21T23:59:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.530+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6214032690 con 0x7f621c102250 2024-04-21T23:59:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.531+0000 7f6219ffb640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 0x7f62040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.531+0000 7f622087b640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 0x7f62040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.531+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f62140f16d0 con 0x7f621c102250 2024-04-21T23:59:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.531+0000 7f622087b640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 0x7f62040a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f620c005fd0 tx=0x7f620c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:40.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.535+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f62140be440 con 0x7f621c102250 2024-04-21T23:59:40.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:40 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:59:40.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:40 smithi038 ceph-mon[27972]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:59:40.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:40 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:59:40.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:40.762+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f621c062340 con 0x7f621c102250 2024-04-21T23:59:40.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:40 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:59:40.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:40 smithi203 ceph-mon[31431]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:59:40.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:40 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:59:40.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:40 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1607882730' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:59:40.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:40 smithi111 ceph-mon[31543]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:59:40.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:40 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:59:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.399+0000 7f6219ffb640 1 -- 172.21.15.38:0/2929335725 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f62140bdd40 con 0x7f621c102250 2024-04-21T23:59:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:added data pool 4 to fsmap 2024-04-21T23:59:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 msgr2=0x7f62040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 0x7f62040a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f620c005fd0 tx=0x7f620c00b410 comp rx=0 tx=0).stop 2024-04-21T23:59:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 msgr2=0x7f621c19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f621402c700 tx=0x7f6214032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 shutdown_connections 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62040a2b90 0x7f62040a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c102250 0x7f621c19efa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f621c101890 0x7f621c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 --2- 172.21.15.38:0/2929335725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f621c1006a0 0x7f621c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 >> 172.21.15.38:0/2929335725 conn(0x7f621c0fbc30 msgr2=0x7f621c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.402+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 shutdown_connections 2024-04-21T23:59:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.403+0000 7f6222adf640 1 -- 172.21.15.38:0/2929335725 wait complete. 2024-04-21T23:59:41.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:41.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 -- 172.21.15.38:0/1165216843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 msgr2=0x7f4f30108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 --2- 172.21.15.38:0/1165216843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f30108ae0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009ca0 tx=0x7f4f2802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 -- 172.21.15.38:0/1165216843 shutdown_connections 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 --2- 172.21.15.38:0/1165216843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f30108ae0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 --2- 172.21.15.38:0/1165216843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 --2- 172.21.15.38:0/1165216843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f30100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 -- 172.21.15.38:0/1165216843 >> 172.21.15.38:0/1165216843 conn(0x7f4f300fbc30 msgr2=0x7f4f300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 -- 172.21.15.38:0/1165216843 shutdown_connections 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.524+0000 7f4f384ae640 1 -- 172.21.15.38:0/1165216843 wait complete. 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 Processor -- start 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 -- start start 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f301045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3010c320 con 0x7f4f30102250 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3010c1a0 con 0x7f4f30101890 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f384ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3010c4a0 con 0x7f4f301006a0 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43196/0 (socket says 172.21.15.38:43196) 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 -- 172.21.15.38:0/503876731 learned_addr learned my addr 172.21.15.38:0/503876731 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f35a49640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f3624a640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f301045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 -- 172.21.15.38:0/503876731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 msgr2=0x7f4f301045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f301045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 -- 172.21.15.38:0/503876731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 msgr2=0x7f4f30102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 -- 172.21.15.38:0/503876731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f30103a70 con 0x7f4f30102250 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f35a49640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:41.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f3624a640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f301045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.525+0000 7f4f36a4b640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f2802c700 tx=0x7f4f28032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.526+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28039040 con 0x7f4f30102250 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.526+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f28044070 con 0x7f4f30102250 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.526+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f301a1b50 con 0x7f4f30102250 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.526+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f280042a0 con 0x7f4f30102250 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.526+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f301a2060 con 0x7f4f30102250 2024-04-21T23:59:41.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.527+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f301a2230 con 0x7f4f30102250 2024-04-21T23:59:41.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.528+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4f28003970 con 0x7f4f30102250 2024-04-21T23:59:41.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.528+0000 7f4f277fe640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 0x7f4f1c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.528+0000 7f4f3624a640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 0x7f4f1c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.528+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4f280f17b0 con 0x7f4f30102250 2024-04-21T23:59:41.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.529+0000 7f4f3624a640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 0x7f4f1c0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4f301016f0 tx=0x7f4f2c005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:41.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.533+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4f280333a0 con 0x7f4f30102250 2024-04-21T23:59:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:41 smithi038 ceph-mon[27972]: pgmap v294: 129 pgs: 21 creating+peering, 65 active+clean, 43 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-04-21T23:59:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:41 smithi038 ceph-mon[27972]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:41 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:59:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:41 smithi038 ceph-mon[27972]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:59:41.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:41 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:59:41.728 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-21T23:59:41.543+0000 7f8b4e317700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T23:59:41.728 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-mon[31543]: pgmap v294: 129 pgs: 21 creating+peering, 65 active+clean, 43 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-04-21T23:59:41.728 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-mon[31543]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:41.728 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:59:41.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-mon[31543]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:59:41.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:41 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:59:41.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.757+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f301a2400 con 0x7f4f30102250 2024-04-21T23:59:41.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.758+0000 7f4f277fe640 1 -- 172.21.15.38:0/503876731 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f4f280be560 con 0x7f4f30102250 2024-04-21T23:59:41.751 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:41.751 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-21T23:59:40.763972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24500,"qdb_cluster":[24500]},"id":1}]} 2024-04-21T23:59:41.751 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 msgr2=0x7f4f1c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 0x7f4f1c0a50d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4f301016f0 tx=0x7f4f2c005e70 comp rx=0 tx=0).stop 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 msgr2=0x7f4f301031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f2802c700 tx=0x7f4f28032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 shutdown_connections 2024-04-21T23:59:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4f1c0a2c10 0x7f4f1c0a50d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f30102250 0x7f4f301031b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f30101890 0x7f4f30102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 --2- 172.21.15.38:0/503876731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f301006a0 0x7f4f301045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 >> 172.21.15.38:0/503876731 conn(0x7f4f300fbc30 msgr2=0x7f4f300fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 shutdown_connections 2024-04-21T23:59:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.760+0000 7f4f384ae640 1 -- 172.21.15.38:0/503876731 wait complete. 2024-04-21T23:59:41.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:41.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:41 smithi203 ceph-mon[31431]: pgmap v294: 129 pgs: 21 creating+peering, 65 active+clean, 43 unknown; 578 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-04-21T23:59:41.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:41 smithi203 ceph-mon[31431]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:59:41.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:41 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2909331129' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:59:41.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:41 smithi203 ceph-mon[31431]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:59:41.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:41 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:59:41.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- 172.21.15.38:0/1187718687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31081006a0 msgr2=0x7f3108100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- 172.21.15.38:0/1187718687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31081006a0 0x7f3108100ac0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3100009ca0 tx=0x7f310002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- 172.21.15.38:0/1187718687 shutdown_connections 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- 172.21.15.38:0/1187718687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3108102250 0x7f3108108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- 172.21.15.38:0/1187718687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108101890 0x7f3108101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- 172.21.15.38:0/1187718687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31081006a0 0x7f3108100ac0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- 172.21.15.38:0/1187718687 >> 172.21.15.38:0/1187718687 conn(0x7f31080fbc30 msgr2=0x7f31080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- 172.21.15.38:0/1187718687 shutdown_connections 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- 172.21.15.38:0/1187718687 wait complete. 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 Processor -- start 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- start start 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31081006a0 0x7f3108197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108102250 0x7f310819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f310810c320 con 0x7f3108101890 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f310810c1a0 con 0x7f31081006a0 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.876+0000 7f310ec39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f310810c4a0 con 0x7f3108102250 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43214/0 (socket says 172.21.15.38:43214) 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 -- 172.21.15.38:0/1838328214 learned_addr learned my addr 172.21.15.38:0/1838328214 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f310c9d5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31081006a0 0x7f3108197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f310d1d6640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108102250 0x7f310819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 -- 172.21.15.38:0/1838328214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108102250 msgr2=0x7f310819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108102250 0x7f310819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 -- 172.21.15.38:0/1838328214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31081006a0 msgr2=0x7f3108197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31081006a0 0x7f3108197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 -- 172.21.15.38:0/1838328214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310819f760 con 0x7f3108101890 2024-04-21T23:59:41.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3107fff640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00c970 tx=0x7f30fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc007bb0 con 0x7f3108101890 2024-04-21T23:59:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30fc007d50 con 0x7f3108101890 2024-04-21T23:59:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310819fa50 con 0x7f3108101890 2024-04-21T23:59:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc00f450 con 0x7f3108101890 2024-04-21T23:59:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.877+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310819ff50 con 0x7f3108101890 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.878+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31081a01e0 con 0x7f3108101890 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.879+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f30fc005020 con 0x7f3108101890 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.880+0000 7f3105ffb640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 0x7f30f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.880+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30fc0cf0d0 con 0x7f3108101890 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.880+0000 7f310c9d5640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 0x7f30f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.881+0000 7f310c9d5640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 0x7f30f00a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f310002c630 tx=0x7f3100009360 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:41.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:41.885+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f30fc014030 con 0x7f3108101890 2024-04-21T23:59:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.071+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31081a03b0 con 0x7f3108101890 2024-04-21T23:59:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.073+0000 7f3105ffb640 1 -- 172.21.15.38:0/1838328214 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7f30fc09bd90 con 0x7f3108101890 2024-04-21T23:59:42.066 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:42.066 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 msgr2=0x7f30f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 0x7f30f00a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f310002c630 tx=0x7f3100009360 comp rx=0 tx=0).stop 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 msgr2=0x7f3108198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00c970 tx=0x7f30fc00ce30 comp rx=0 tx=0).stop 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 shutdown_connections 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30f00a2b90 0x7f30f00a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3108102250 0x7f310819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3108101890 0x7f3108198080 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 --2- 172.21.15.38:0/1838328214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31081006a0 0x7f3108197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 >> 172.21.15.38:0/1838328214 conn(0x7f31080fbc30 msgr2=0x7f31080fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 shutdown_connections 2024-04-21T23:59:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.075+0000 7f310ec39640 1 -- 172.21.15.38:0/1838328214 wait complete. 2024-04-21T23:59:42.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- 172.21.15.38:0/4280373554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2584101890 msgr2=0x7f2584101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 --2- 172.21.15.38:0/4280373554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2584101890 0x7f2584101d10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2574009900 tx=0x7f257402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- 172.21.15.38:0/4280373554 shutdown_connections 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 --2- 172.21.15.38:0/4280373554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 0x7f2584108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 --2- 172.21.15.38:0/4280373554 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2584101890 0x7f2584101d10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 --2- 172.21.15.38:0/4280373554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25841006a0 0x7f2584100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- 172.21.15.38:0/4280373554 >> 172.21.15.38:0/4280373554 conn(0x7f25840fbc30 msgr2=0x7f25840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- 172.21.15.38:0/4280373554 shutdown_connections 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- 172.21.15.38:0/4280373554 wait complete. 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 Processor -- start 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.189+0000 7f258a435640 1 -- start start 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 0x7f2584109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 0x7f2584109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f258410d440 con 0x7f25841006a0 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f258410d2c0 con 0x7f2584102250 2024-04-21T23:59:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f258410d5c0 con 0x7f2584101890 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43244/0 (socket says 172.21.15.38:43244) 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 -- 172.21.15.38:0/2305561879 learned_addr learned my addr 172.21.15.38:0/2305561879 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25889d2640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 0x7f2584109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25837fe640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 0x7f2584109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 -- 172.21.15.38:0/2305561879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 msgr2=0x7f2584109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 0x7f2584109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 -- 172.21.15.38:0/2305561879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 msgr2=0x7f2584109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 0x7f2584109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 -- 172.21.15.38:0/2305561879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258410a430 con 0x7f25841006a0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25837fe640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 0x7f2584109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f2583fff640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c970 tx=0x7f256c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c007c10 con 0x7f25841006a0 2024-04-21T23:59:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256c007db0 con 0x7f25841006a0 2024-04-21T23:59:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25841a1b00 con 0x7f25841006a0 2024-04-21T23:59:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.190+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c00f940 con 0x7f25841006a0 2024-04-21T23:59:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.191+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25841a2070 con 0x7f25841006a0 2024-04-21T23:59:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.191+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2584107960 con 0x7f25841006a0 2024-04-21T23:59:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.193+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f256c00fae0 con 0x7f25841006a0 2024-04-21T23:59:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.193+0000 7f25817fa640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 0x7f25700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.193+0000 7f25837fe640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 0x7f25700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.193+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f256c0cf640 con 0x7f25841006a0 2024-04-21T23:59:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.194+0000 7f25837fe640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 0x7f25700a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2574006010 tx=0x7f257402c8a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.198+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f256c09c3f0 con 0x7f25841006a0 2024-04-21T23:59:42.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.383+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f258402d3d0 con 0x7f25841006a0 2024-04-21T23:59:42.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.384+0000 7f25817fa640 1 -- 172.21.15.38:0/2305561879 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7f256c09bcf0 con 0x7f25841006a0 2024-04-21T23:59:42.377 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:42.377 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 msgr2=0x7f25700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 0x7f25700a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2574006010 tx=0x7f257402c8a0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 msgr2=0x7f258410b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c970 tx=0x7f256c00ce30 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 shutdown_connections 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f25700a2c10 0x7f25700a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2584102250 0x7f2584109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2584101890 0x7f2584109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 --2- 172.21.15.38:0/2305561879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25841006a0 0x7f258410b020 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 >> 172.21.15.38:0/2305561879 conn(0x7f25840fbc30 msgr2=0x7f2584104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 shutdown_connections 2024-04-21T23:59:42.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.386+0000 7f258a435640 1 -- 172.21.15.38:0/2305561879 wait complete. 2024-04-21T23:59:42.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:59:42.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:59:42.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:42.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/503876731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.384 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1838328214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:59:42.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 -- 172.21.15.38:0/3782984612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b4101890 msgr2=0x7f16b4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 --2- 172.21.15.38:0/3782984612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b4101890 0x7f16b4101d10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009900 tx=0x7f16a802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 -- 172.21.15.38:0/3782984612 shutdown_connections 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 --2- 172.21.15.38:0/3782984612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4102250 0x7f16b4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 --2- 172.21.15.38:0/3782984612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b4101890 0x7f16b4101d10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 --2- 172.21.15.38:0/3782984612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b41006a0 0x7f16b4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 -- 172.21.15.38:0/3782984612 >> 172.21.15.38:0/3782984612 conn(0x7f16b40fbc30 msgr2=0x7f16b40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 -- 172.21.15.38:0/3782984612 shutdown_connections 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 -- 172.21.15.38:0/3782984612 wait complete. 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.517+0000 7f16b9c78640 1 Processor -- start 2024-04-21T23:59:42.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 -- start start 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4101890 0x7f16b4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 0x7f16b419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16b410c380 con 0x7f16b41006a0 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16b410c200 con 0x7f16b4102250 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16b410c500 con 0x7f16b4101890 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43262/0 (socket says 172.21.15.38:43262) 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 -- 172.21.15.38:0/1163611642 learned_addr learned my addr 172.21.15.38:0/1163611642 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b3fff640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 0x7f16b419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b2ffd640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4101890 0x7f16b4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 -- 172.21.15.38:0/1163611642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4101890 msgr2=0x7f16b4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4101890 0x7f16b4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 -- 172.21.15.38:0/1163611642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 msgr2=0x7f16b419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 0x7f16b419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 -- 172.21.15.38:0/1163611642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b419f740 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b3fff640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 0x7f16b419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b37fe640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f16a400d950 tx=0x7f16a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4017020 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a4009c30 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b419fa30 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.518+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a400f740 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.519+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b419ff50 con 0x7f16b41006a0 2024-04-21T23:59:42.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.519+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b40689b0 con 0x7f16b41006a0 2024-04-21T23:59:42.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.520+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f16a401a020 con 0x7f16b41006a0 2024-04-21T23:59:42.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.521+0000 7f16b0ff9640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 0x7f169c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.521+0000 7f16b2ffd640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 0x7f169c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.521+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f16a40ced30 con 0x7f16b41006a0 2024-04-21T23:59:42.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.522+0000 7f16b2ffd640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 0x7f169c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006010 tx=0x7f16a8000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.525+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f16a4021070 con 0x7f16b41006a0 2024-04-21T23:59:42.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:42 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2305561879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.751+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16b41a01a0 con 0x7f16b41006a0 2024-04-21T23:59:42.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.752+0000 7f16b0ff9640 1 -- 172.21.15.38:0/1163611642 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f16a4009de0 con 0x7f16b41006a0 2024-04-21T23:59:42.745 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:42.745 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24395,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24409,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-21T23:59:40.763972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24500,"qdb_cluster":[24500]},"id":1}]} 2024-04-21T23:59:42.745 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 msgr2=0x7f169c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 0x7f169c0a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006010 tx=0x7f16a8000bc0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 msgr2=0x7f16b4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f16a400d950 tx=0x7f16a400de10 comp rx=0 tx=0).stop 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 shutdown_connections 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f169c0a2b90 0x7f169c0a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b4102250 0x7f16b419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16b4101890 0x7f16b4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 --2- 172.21.15.38:0/1163611642 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16b41006a0 0x7f16b4197b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 >> 172.21.15.38:0/1163611642 conn(0x7f16b40fbc30 msgr2=0x7f16b4069dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 shutdown_connections 2024-04-21T23:59:42.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.754+0000 7f16b9c78640 1 -- 172.21.15.38:0/1163611642 wait complete. 2024-04-21T23:59:42.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:42.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:59:42.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:59:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/503876731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1838328214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:42 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2305561879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2929335725' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/503876731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1838328214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:42 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2305561879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:42.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- 172.21.15.38:0/3069660658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 msgr2=0x7fb5d0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/3069660658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0100ac0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009900 tx=0x7fb5c802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- 172.21.15.38:0/3069660658 shutdown_connections 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/3069660658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0102250 0x7fb5d0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/3069660658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 0x7fb5d0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/3069660658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0100ac0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- 172.21.15.38:0/3069660658 >> 172.21.15.38:0/3069660658 conn(0x7fb5d00fbc30 msgr2=0x7fb5d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- 172.21.15.38:0/3069660658 shutdown_connections 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- 172.21.15.38:0/3069660658 wait complete. 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 Processor -- start 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.868+0000 7fb5d56c9640 1 -- start start 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 0x7fb5d01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0102250 0x7fb5d019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010c2f0 con 0x7fb5d01006a0 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010c170 con 0x7fb5d0101890 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010c470 con 0x7fb5d0102250 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43280/0 (socket says 172.21.15.38:43280) 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 -- 172.21.15.38:0/2541392001 learned_addr learned my addr 172.21.15.38:0/2541392001 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ce7fc640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 0x7fb5d01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 -- 172.21.15.38:0/2541392001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0102250 msgr2=0x7fb5d019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0102250 0x7fb5d019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 -- 172.21.15.38:0/2541392001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 msgr2=0x7fb5d01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 0x7fb5d01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 -- 172.21.15.38:0/2541392001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d019f7c0 con 0x7fb5d01006a0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5ceffd640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8041070 con 0x7fb5d01006a0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c80044d0 con 0x7fb5d01006a0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d019fa50 con 0x7fb5d01006a0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8034440 con 0x7fb5d01006a0 2024-04-21T23:59:42.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.869+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d019ff40 con 0x7fb5d01006a0 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.870+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d01a0170 con 0x7fb5d01006a0 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.872+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb5c8044070 con 0x7fb5d01006a0 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.872+0000 7fb5b7fff640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 0x7fb5b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.873+0000 7fb5ce7fc640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 0x7fb5b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.873+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb5c80f0ec0 con 0x7fb5d01006a0 2024-04-21T23:59:42.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.873+0000 7fb5ce7fc640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 0x7fb5b80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5d01990b0 tx=0x7fb5c4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:42.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:42.877+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb5c80bdbe0 con 0x7fb5d01006a0 2024-04-21T23:59:43.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.062+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5d01a0910 con 0x7fb5d01006a0 2024-04-21T23:59:43.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.064+0000 7fb5b7fff640 1 -- 172.21.15.38:0/2541392001 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fb5c80bd4e0 con 0x7fb5d01006a0 2024-04-21T23:59:43.057 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:43.057 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 msgr2=0x7fb5b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 0x7fb5b80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5d01990b0 tx=0x7fb5c4005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 msgr2=0x7fb5d0197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8005d50 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 shutdown_connections 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5b80a2b90 0x7fb5b80a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0102250 0x7fb5d019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d0101890 0x7fb5d01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 --2- 172.21.15.38:0/2541392001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5d01006a0 0x7fb5d0197ba0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 >> 172.21.15.38:0/2541392001 conn(0x7fb5d00fbc30 msgr2=0x7fb5d00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 shutdown_connections 2024-04-21T23:59:43.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.066+0000 7fb5d56c9640 1 -- 172.21.15.38:0/2541392001 wait complete. 2024-04-21T23:59:43.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T23:59:36.029248+0000', 'modified': '2024-04-21T23:59:40.763972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3890542559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3890542559}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3890542559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24500, 'qdb_cluster': [24500]} 2024-04-21T23:59:43.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T23:59:43.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:59:43.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 -- 172.21.15.38:0/1109691241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102230 msgr2=0x7fd6d4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/1109691241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102230 0x7fd6d4108ac0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009ca0 tx=0x7fd6cc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 -- 172.21.15.38:0/1109691241 shutdown_connections 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/1109691241 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4102230 0x7fd6d4108ac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/1109691241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4101870 0x7fd6d4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/1109691241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 0x7fd6d4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 -- 172.21.15.38:0/1109691241 >> 172.21.15.38:0/1109691241 conn(0x7fd6d40fbbf0 msgr2=0x7fd6d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.180+0000 7fd6dacd4640 1 -- 172.21.15.38:0/1109691241 shutdown_connections 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 -- 172.21.15.38:0/1109691241 wait complete. 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 Processor -- start 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 -- start start 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 0x7fd6d4199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102230 0x7fd6d41a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410c300 con 0x7fd6d4101870 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410c180 con 0x7fd6d4100680 2024-04-21T23:59:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6dacd4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410c480 con 0x7fd6d4102230 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43296/0 (socket says 172.21.15.38:43296) 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.181+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4111968748 learned_addr learned my addr 172.21.15.38:0/4111968748 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 0x7fd6d4199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d9271640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102230 0x7fd6d41a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4111968748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102230 msgr2=0x7fd6d41a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102230 0x7fd6d41a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4111968748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 msgr2=0x7fd6d4199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 0x7fd6d4199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4111968748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41a1850 con 0x7fd6d4101870 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d770 tx=0x7fd6c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d1ffb640 1 -- 172.21.15.38:0/4111968748 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0017070 con 0x7fd6d4101870 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d41a1b40 con 0x7fd6d4101870 2024-04-21T23:59:43.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d1ffb640 1 -- 172.21.15.38:0/4111968748 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c0005500 con 0x7fd6d4101870 2024-04-21T23:59:43.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6d1ffb640 1 -- 172.21.15.38:0/4111968748 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0004e00 con 0x7fd6d4101870 2024-04-21T23:59:43.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.182+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d41a2040 con 0x7fd6d4101870 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.184+0000 7fd6d1ffb640 1 -- 172.21.15.38:0/4111968748 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6c0004fa0 con 0x7fd6d4101870 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 Processor -- start 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 -- start start 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d1ffb640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 0x7fd6c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd0201490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0209060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558cd0165df0 con 0x558cd0201a60 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558cd010e640 con 0x558cd0201070 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558cd0161db0 con 0x558cd0206bd0 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d9271640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 0x7fd6c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43298/0 (socket says 172.21.15.38:43298) 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d3fff640 1 -- 172.21.15.38:0/3460855038 learned_addr learned my addr 172.21.15.38:0/3460855038 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d1ffb640 1 -- 172.21.15.38:0/4111968748 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6c00cedf0 con 0x7fd6d4101870 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.185+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd0201490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d9271640 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0209060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 -- 172.21.15.38:0/3460855038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 msgr2=0x558cd0209060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0209060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 -- 172.21.15.38:0/3460855038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 msgr2=0x558cd0201490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd0201490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 -- 172.21.15.38:0/3460855038 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ccfe7a2d0 con 0x558cd0201a60 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d9271640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 0x7fd6c40a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc006010 tx=0x7fd6cc02cf40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0005760 tx=0x7fd6c0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d73e793d31aec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6beffd640 1 -- 172.21.15.38:0/3460855038 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0020070 con 0x558cd0201a60 2024-04-21T23:59:43.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6beffd640 1 -- 172.21.15.38:0/3460855038 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c00046c0 con 0x558cd0201a60 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.186+0000 7fd6beffd640 1 -- 172.21.15.38:0/3460855038 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00cc760 con 0x558cd0201a60 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3460855038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 msgr2=0x558cd0206560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0005760 tx=0x7fd6c0009c30 comp rx=0 tx=0).stop 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3460855038 shutdown_connections 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0209060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd0206560 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3460855038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd0201490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3460855038 >> 172.21.15.38:0/3460855038 conn(0x558cd01fc5e0 msgr2=0x558cd01fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3460855038 shutdown_connections 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3460855038 wait complete. 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- start start 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 Processor -- start 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201a60 0x558cd020ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0212a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.187+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558cd0209ac0 con 0x558cd0201070 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558cd0121be0 con 0x558cd0201a60 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558cd0209c40 con 0x558cd0206bd0 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 10 client.0 _ll_register_callbacks cb 0x558cd01215d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43306/0 (socket says 172.21.15.38:43306) 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 -- 172.21.15.38:0/1603387354 learned_addr learned my addr 172.21.15.38:0/1603387354 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201a60 0x558cd020ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 -- 172.21.15.38:0/1603387354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 msgr2=0x558cd0212a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0212a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 -- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201a60 msgr2=0x558cd020ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201a60 0x558cd020ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 -- 172.21.15.38:0/1603387354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ccff446b0 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8010820 tx=0x7fd6c8010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9a50f9f00d0ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c801e060 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6c800fb30 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558cd0173330 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd6c8025050 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 10 client.14748 fetch_fsmap learned FSMap version 17 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.188+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558cd0215b70 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558cd0173330 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fd6c8018370 con 0x558cd0201070 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 10 client.14748 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 10 client.14748 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 20 client.14748 populate_metadata read hostname 'smithi038' 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0218270 0x558cd02196e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 4 client.14748 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cd016d680 con 0x558cd0218270 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.189+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0218270 0x558cd02196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0218270 0x558cd02196e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b6fbc237b93e0bd server_cookie=2c09c15338c00b76 in_seq=0 out_seq=0 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6b3fff640 10 client.14748 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6b3fff640 1 -- 172.21.15.38:0/1603387354 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558cd016d680 con 0x558cd0218270 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6b3fff640 10 client.14748 handle_command_reply: tid=0 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 client.14748 shutdown 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 msgr2=0x558cd020b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8010820 tx=0x7fd6c8010ce0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 shutdown_connections 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0212a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201a60 0x558cd020ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0218270 0x558cd02196e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/1603387354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201070 0x558cd020b4f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6b3fff640 0 client.14748 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-21T23:59:43.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.190+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 >> 172.21.15.38:0/1603387354 conn(0x558cd01fc5e0 msgr2=0x558cd01fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.191+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 shutdown_connections 2024-04-21T23:59:43.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.191+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/1603387354 wait complete. 2024-04-21T23:59:43.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.191+0000 7fd6dd9a0740 20 client.14748 trim_cache size 0 max 16384 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 Processor -- start 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 -- start start 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd01fae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0201a60 0x558cd0212990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558cd0217a20 con 0x558cd0206bd0 2024-04-21T23:59:43.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558cd02180f0 con 0x558cd0201070 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558cd02187c0 con 0x558cd0201a60 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43314/0 (socket says 172.21.15.38:43314) 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 -- 172.21.15.38:0/3675206006 learned_addr learned my addr 172.21.15.38:0/3675206006 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd01fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 -- 172.21.15.38:0/3675206006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0201a60 msgr2=0x558cd0212990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0201a60 0x558cd0212990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 -- 172.21.15.38:0/3675206006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 msgr2=0x558cd01fae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd01fae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 -- 172.21.15.38:0/3675206006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd01fac20 con 0x558cd0206bd0 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd01fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.209+0000 7fd6d9271640 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005cb0 tx=0x7fd6cc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756d2ea95071fa31 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6be7fc640 1 -- 172.21.15.38:0/3675206006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc052070 con 0x558cd0206bd0 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6be7fc640 1 -- 172.21.15.38:0/3675206006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6cc003b70 con 0x558cd0206bd0 2024-04-21T23:59:43.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6be7fc640 1 -- 172.21.15.38:0/3675206006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc039040 con 0x558cd0206bd0 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3675206006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 msgr2=0x558cd01fb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005cb0 tx=0x7fd6cc02e050 comp rx=0 tx=0).stop 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3675206006 shutdown_connections 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0206bd0 0x558cd01fb3d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0201a60 0x558cd0212990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/3675206006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd01fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3675206006 >> 172.21.15.38:0/3675206006 conn(0x558cd01fc5e0 msgr2=0x558cd01f7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3675206006 shutdown_connections 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.210+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/3675206006 wait complete. 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 -- start start 2024-04-21T23:59:43.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 Processor -- start 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd020e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0211820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558cd0219560 con 0x558cd0201a60 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558cd0218e90 con 0x558cd0201070 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558cd0219c30 con 0x558cd0206bd0 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6dd9a0740 10 client.0 _ll_register_callbacks cb 0x7fd6c800aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43316/0 (socket says 172.21.15.38:43316) 2024-04-21T23:59:43.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4214804719 learned_addr learned my addr 172.21.15.38:0/4214804719 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd020e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d9271640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0211820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4214804719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 msgr2=0x558cd0211820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0211820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 msgr2=0x558cd020e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd020e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.211+0000 7fd6d3fff640 1 -- 172.21.15.38:0/4214804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd0211e10 con 0x558cd0201a60 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6d3fff640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0097780 tx=0x7fd6c0005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03b8ce1ca51d5d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b4060 con 0x558cd0201a60 2024-04-21T23:59:43.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6cc003b70 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6c0009cc0 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd6c00c3e60 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558cd027e400 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6cc003b70 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fd6c00b4200 con 0x558cd0201a60 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:59:43.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 20 client.14754 populate_metadata read hostname 'smithi038' 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0282a60 0x558cd0281770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 4 client.14754 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "status"}) -- 0x558cd00eba70 con 0x558cd0282a60 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.212+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0282a60 0x558cd0281770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6b3fff640 10 client.14754 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6d8a70640 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0282a60 0x558cd0281770 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da8f841a9e1f991 server_cookie=767bd29a1c91704b in_seq=0 out_seq=0 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6b3fff640 1 -- 172.21.15.38:0/4214804719 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558cd00eba70 con 0x558cd0282a60 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6b3fff640 10 client.14754 handle_command_reply: tid=0 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 client.14754 shutdown 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 msgr2=0x558cd020e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0097780 tx=0x7fd6c0005bb0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 shutdown_connections 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558cd0206bd0 0x558cd0211820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558cd0282a60 0x558cd0281770 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558cd0201a60 0x558cd020e800 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 --2- 172.21.15.38:0/4214804719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cd0201070 0x558cd020e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6b3fff640 0 client.14754 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 >> 172.21.15.38:0/4214804719 conn(0x558cd01fc5e0 msgr2=0x558cd01fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 shutdown_connections 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 1 -- 172.21.15.38:0/4214804719 wait complete. 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.213+0000 7fd6dd9a0740 20 client.14754 trim_cache size 0 max 16384 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24500, 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-21T23:59:43.218 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 6.4388943809999999, 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 17, 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 12.416571129999999 2024-04-21T23:59:43.219 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 msgr2=0x7fd6c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 0x7fd6c40a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc006010 tx=0x7fd6cc02cf40 comp rx=0 tx=0).stop 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 msgr2=0x7fd6d419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000d770 tx=0x7fd6c000dc30 comp rx=0 tx=0).stop 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 shutdown_connections 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6c40a2b90 0x7fd6c40a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6d4102230 0x7fd6d41a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6d4101870 0x7fd6d419a180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 --2- 172.21.15.38:0/4111968748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6d4100680 0x7fd6d4199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:43.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 >> 172.21.15.38:0/4111968748 conn(0x7fd6d40fbbf0 msgr2=0x7fd6d40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:43.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 shutdown_connections 2024-04-21T23:59:43.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:43.214+0000 7fd6dacd4640 1 -- 172.21.15.38:0/4111968748 wait complete. 2024-04-21T23:59:43.223 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T23:59:43.223 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:43.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-21T23:59:43.468 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:59:43.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:43 smithi038 ceph-mon[27972]: pgmap v297: 129 pgs: 7 creating+peering, 113 active+clean, 9 unknown; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:59:43.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:43 smithi038 ceph-mon[27972]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:59:43.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:43 smithi038 ceph-mon[27972]: Cluster is now healthy 2024-04-21T23:59:43.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:43 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1163611642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:43.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:43 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2541392001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:43.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:43 smithi203 ceph-mon[31431]: pgmap v297: 129 pgs: 7 creating+peering, 113 active+clean, 9 unknown; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:59:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:43 smithi203 ceph-mon[31431]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:59:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:43 smithi203 ceph-mon[31431]: Cluster is now healthy 2024-04-21T23:59:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:43 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1163611642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:43 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2541392001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:43.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:43 smithi111 ceph-mon[31543]: pgmap v297: 129 pgs: 7 creating+peering, 113 active+clean, 9 unknown; 579 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:59:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:43 smithi111 ceph-mon[31543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:59:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:43 smithi111 ceph-mon[31543]: Cluster is now healthy 2024-04-21T23:59:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:43 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1163611642' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:59:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:43 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2541392001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/383430303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f41013b0 msgr2=0x7fe5f4101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/383430303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f41013b0 0x7fe5f4101790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40099c0 tx=0x7fe5e4009cd0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/383430303 shutdown_connections 2024-04-21T23:59:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/383430303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4106490 0x7fe5f410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/383430303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f41013b0 0x7fe5f4101790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/383430303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/383430303 >> 172.21.15.38:0/383430303 conn(0x7fe5f40781d0 msgr2=0x7fe5f40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.288+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/383430303 shutdown_connections 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.289+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/383430303 wait complete. 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.289+0000 7fe5fa3ea700 1 Processor -- start 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.289+0000 7fe5fa3ea700 1 -- start start 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 0x7fe5f41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:45.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f4111470 con 0x7fe5f4106490 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41112f0 con 0x7fe5f4101d60 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41115f0 con 0x7fe5f41013b0 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56352/0 (socket says 172.21.15.38:56352) 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 -- 172.21.15.38:0/4169873382 learned_addr learned my addr 172.21.15.38:0/4169873382 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f37fe700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f3fff700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 0x7fe5f41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 -- 172.21.15.38:0/4169873382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 msgr2=0x7fe5f41998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 0x7fe5f41998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 -- 172.21.15.38:0/4169873382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 msgr2=0x7fe5f4199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 -- 172.21.15.38:0/4169873382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f41a4e90 con 0x7fe5f4106490 2024-04-21T23:59:45.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f37fe700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f3fff700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 0x7fe5f41998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f8987700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00ada0 tx=0x7fe5ec00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec017020 con 0x7fe5f4106490 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f41a5060 con 0x7fe5f4106490 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5ec004bb0 con 0x7fe5f4106490 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.290+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec005520 con 0x7fe5f4106490 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.291+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f41a5570 con 0x7fe5f4106490 2024-04-21T23:59:45.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.291+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f4068f20 con 0x7fe5f4106490 2024-04-21T23:59:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.293+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe5ec0056c0 con 0x7fe5f4106490 2024-04-21T23:59:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.293+0000 7fe5f17fa700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 0x7fe5dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.293+0000 7fe5f3fff700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 0x7fe5dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.293+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe5ec0cef80 con 0x7fe5f4106490 2024-04-21T23:59:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.294+0000 7fe5f3fff700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 0x7fe5dc0a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009750 tx=0x7fe5e4009f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:45.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.297+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe5ec021020 con 0x7fe5f4106490 2024-04-21T23:59:45.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:45 smithi038 ceph-mon[27972]: pgmap v298: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-21T23:59:45.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.551+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe5f402db30 con 0x7fe5f4106490 2024-04-21T23:59:45.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.552+0000 7fe5f17fa700 1 -- 172.21.15.38:0/4169873382 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fe5ec09bd30 con 0x7fe5f4106490 2024-04-21T23:59:45.545 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 msgr2=0x7fe5dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 0x7fe5dc0a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009750 tx=0x7fe5e4009f10 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 msgr2=0x7fe5f41a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00ada0 tx=0x7fe5ec00c330 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 shutdown_connections 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe5dc0a2c20 0x7fe5dc0a50e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5f4106490 0x7fe5f41a4950 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f41013b0 0x7fe5f41998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 --2- 172.21.15.38:0/4169873382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5f4101d60 0x7fe5f4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 >> 172.21.15.38:0/4169873382 conn(0x7fe5f40781d0 msgr2=0x7fe5f4107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 shutdown_connections 2024-04-21T23:59:45.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:45.554+0000 7fe5fa3ea700 1 -- 172.21.15.38:0/4169873382 wait complete. 2024-04-21T23:59:45.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:45 smithi203 ceph-mon[31431]: pgmap v298: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-21T23:59:45.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:45 smithi111 ceph-mon[31543]: pgmap v298: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-21T23:59:46.462 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T23:59:46.463 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:59:46.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:46.463 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-04-21T23:59:46.481 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T23:59:46.481 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T23:59:46.481 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T23:59:46.481 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:59:46.481 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T23:59:46.482 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 42156sec preferred_lft 42156sec 2024-04-21T23:59:46.483 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-21T23:59:46.483 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:59:46.483 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:59:46.483 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-04-21T23:59:46.483 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-04-21T23:59:46.483 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T23:59:46.484 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-21T23:59:46.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:59:46.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4169873382' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:59:46.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:46.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:46.671 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T23:59:46.695 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-04-21T23:59:46.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:46.696 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-04-21T23:59:46.764 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-04-21T23:59:46.764 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T23:59:46.764 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T23:59:46.764 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T23:59:46.764 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T23:59:46.765 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-21T23:59:46.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4169873382' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:59:46.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4169873382' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:59:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:59:47.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:47.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:47.125 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-21T23:59:47.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:47.140 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-04-21T23:59:47.196 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:59:47.196 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-21T23:59:47.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:59:47.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T23:59:47.196 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-21T23:59:47.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:59:47.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:47.314 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T23:59:47.314 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T23:59:47.315 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-21T23:59:47.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:59:47.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:47 smithi038 ceph-mon[27972]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 2 op/s 2024-04-21T23:59:47.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:47.575 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:59:47.575 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-21T23:59:47.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-04-21T23:59:47.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T23:59:47.575 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-21T23:59:47.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:59:47.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:47.652 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T23:59:47.652 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:59:47.653 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:47.668 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:59:47.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:59:47.668 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:47.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2024-04-21T23:59:47.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:47 smithi203 ceph-mon[31431]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 2 op/s 2024-04-21T23:59:47.788 DEBUG:teuthology.orchestra.run.smithi038:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:59:47.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:47 smithi111 ceph-mon[31543]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 2 op/s 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/proc 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/sys 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/dev 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/security 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/shm 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/pts 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/run 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/cgroup 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/pstore 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/bpf 2024-04-21T23:59:47.845 INFO:teuthology.orchestra.run.smithi038.stdout:/ 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/selinux 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/mqueue 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/hugepages 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/debug 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/tracing 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/config 2024-04-21T23:59:47.846 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:59:47.847 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/fuse/connections 2024-04-21T23:59:47.847 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:59:47.847 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:59:47.847 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:59:47.847 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/1000 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/ceph 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/bef4ccde65fa12340daf1b2e32cad42c4a14cf4bd1e93e2bd7c610acb257636c/merged 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/facbc9808cec058ec5468b97bf2c9bedfa2e7951685d6bf8818f2108ba0889bb/merged 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/0 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:59:47.848 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/c6dd5b740b8ddaca3124168d55d03a1e47713d5b626329172edaa31c4c4cd536/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/d3dc8f6c3c39c3a9725b45ee643de4d560253371918400b46da5dc35afe86434/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/9defa69cc312a1bc6b0e830693ea07db7344c0cb9e1fb52633aca9f7e50d287b/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/92a3bcc783cd48fb285b65a3e5bd5e787b967b6ee64e49598203b58b07ed8209/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/efbee1b29ab77e1f8df937d9efe8672ae6fc27babd0f2688a0b86f2e9cd3e4d9/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b99a021e9c7538c165cd37c35355e3cbd4de5ae769b1c4cc9ff13eb7787822b4/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/f5fa3f295671a1fdafd333cd7de44179265de8088fcf39861e9aa86fe24586b1/merged 2024-04-21T23:59:47.849 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/90ca01f8306ba79ed95ef3704aeb6bfa2270126e60c4300b8b5b4f70438c7319/merged 2024-04-21T23:59:47.850 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns 2024-04-21T23:59:47.850 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:59:47.850 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:59:47.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:47.850 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2024-04-21T23:59:47.900 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T23:59:47.901 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T23:59:47.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2024-04-21T23:59:47.968 DEBUG:teuthology.orchestra.run.smithi038:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:59:48.024 INFO:teuthology.orchestra.run.smithi038.stdout:/proc 2024-04-21T23:59:48.024 INFO:teuthology.orchestra.run.smithi038.stdout:/sys 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/dev 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/security 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/shm 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/pts 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/run 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/cgroup 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/pstore 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/bpf 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/ 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/selinux 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/mqueue 2024-04-21T23:59:48.025 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/hugepages 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/debug 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/tracing 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/kernel/config 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/sys/fs/fuse/connections 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/1000 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/ceph 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:59:48.026 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/bef4ccde65fa12340daf1b2e32cad42c4a14cf4bd1e93e2bd7c610acb257636c/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/facbc9808cec058ec5468b97bf2c9bedfa2e7951685d6bf8818f2108ba0889bb/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/run/user/0 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/c6dd5b740b8ddaca3124168d55d03a1e47713d5b626329172edaa31c4c4cd536/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/d3dc8f6c3c39c3a9725b45ee643de4d560253371918400b46da5dc35afe86434/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/9defa69cc312a1bc6b0e830693ea07db7344c0cb9e1fb52633aca9f7e50d287b/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/92a3bcc783cd48fb285b65a3e5bd5e787b967b6ee64e49598203b58b07ed8209/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/efbee1b29ab77e1f8df937d9efe8672ae6fc27babd0f2688a0b86f2e9cd3e4d9/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/b99a021e9c7538c165cd37c35355e3cbd4de5ae769b1c4cc9ff13eb7787822b4/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/f5fa3f295671a1fdafd333cd7de44179265de8088fcf39861e9aa86fe24586b1/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/var/lib/containers/storage/overlay/90ca01f8306ba79ed95ef3704aeb6bfa2270126e60c4300b8b5b4f70438c7319/merged 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:59:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:59:48.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:48.028 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2024-04-21T23:59:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:2024-04-21T23:59:48.043+0000 7fa193c3a480 -1 init, newargv = 0x556302a7e050 newargc=15 2024-04-21T23:59:48.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[72099]: starting ceph client 2024-04-21T23:59:48.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[72099]: starting fuse 2024-04-21T23:59:48.052 INFO:teuthology.orchestra.run.smithi038.stdout:115 2024-04-21T23:59:48.052 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T23:59:48.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo stdin-killer -- python3 -c ' 2024-04-21T23:59:48.052 DEBUG:teuthology.orchestra.run.smithi038:> import glob 2024-04-21T23:59:48.052 DEBUG:teuthology.orchestra.run.smithi038:> import re 2024-04-21T23:59:48.052 DEBUG:teuthology.orchestra.run.smithi038:> import os 2024-04-21T23:59:48.052 DEBUG:teuthology.orchestra.run.smithi038:> import subprocess 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> def _find_admin_socket(client_name): 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> files = glob.glob(asok_path) 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> # Given a non-glob path, it better be there 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> if "*" not in asok_path: 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> assert(len(files) == 1) 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> return files[0] 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> for f in files: 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T23:59:48.053 DEBUG:teuthology.orchestra.run.smithi038:> contents = proc_f.read() 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> if mountpoint in contents: 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> return f 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> print(_find_admin_socket("client.admin")) 2024-04-21T23:59:48.054 DEBUG:teuthology.orchestra.run.smithi038:> ' 2024-04-21T23:59:48.172 INFO:teuthology.orchestra.run.smithi038.stdout:/var/run/ceph/ceph-client.admin.72099.asok 2024-04-21T23:59:48.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:48.183 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72099.asok 2024-04-21T23:59:48.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:48.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72099.asok status 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "metadata": { 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "entity_id": "admin", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": "smithi038", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "pid": "72099", 2024-04-21T23:59:48.320 INFO:teuthology.orchestra.run.smithi038.stdout: "root": "/" 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_count": 0, 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_pinned_count": 0, 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 14766, 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "inst": { 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": { 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "client", 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 14766 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3399836102 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-21T23:59:48.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3399836102 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "inst_str": "client.14766 192.168.0.1:0/3399836102", 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "addr_str": "192.168.0.1:0/3399836102", 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "inode_count": 1, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_epoch": 17, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch": 89, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch_barrier": 0, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklisted": false, 2024-04-21T23:59:48.322 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs" 2024-04-21T23:59:48.323 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-21T23:59:48.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:48.328 DEBUG:teuthology.orchestra.run.smithi038:> stat --file-system '--printf=%T 2024-04-21T23:59:48.328 DEBUG:teuthology.orchestra.run.smithi038:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:48.344 INFO:teuthology.orchestra.run.smithi038.stdout:fuseblk 2024-04-21T23:59:48.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:48.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:48.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:48.410 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T23:59:48.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:59:48.556 INFO:teuthology.orchestra.run.smithi038.stdout:# file: . 2024-04-21T23:59:48.556 INFO:teuthology.orchestra.run.smithi038.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T23:59:48.556 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:48.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:59:48.562 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:59:48.562 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:59:48.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:48.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 -- 172.21.15.38:0/352906872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960107580 msgr2=0x7f8960109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:48.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 --2- 172.21.15.38:0/352906872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960107580 0x7f8960109970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8958009ca0 tx=0x7f895802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 -- 172.21.15.38:0/352906872 shutdown_connections 2024-04-21T23:59:48.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 --2- 172.21.15.38:0/352906872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960107580 0x7f8960109970 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 --2- 172.21.15.38:0/352906872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960104c50 0x7f8960107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 --2- 172.21.15.38:0/352906872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960102320 0x7f8960104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 -- 172.21.15.38:0/352906872 >> 172.21.15.38:0/352906872 conn(0x7f89600fbc90 msgr2=0x7f89600fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:48.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.728+0000 7f8967096640 1 -- 172.21.15.38:0/352906872 shutdown_connections 2024-04-21T23:59:48.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 -- 172.21.15.38:0/352906872 wait complete. 2024-04-21T23:59:48.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 Processor -- start 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 -- start start 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 0x7f8960198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960107580 0x7f896019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f896010d430 con 0x7f8960102320 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f896010d2b0 con 0x7f8960107580 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8967096640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f896010d5b0 con 0x7f8960104c50 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.729+0000 7f8964e32640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56380/0 (socket says 172.21.15.38:56380) 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 -- 172.21.15.38:0/357229891 learned_addr learned my addr 172.21.15.38:0/357229891 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8965633640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960107580 0x7f896019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:48.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f895ffff640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 0x7f8960198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 -- 172.21.15.38:0/357229891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 msgr2=0x7f8960198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 0x7f8960198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 -- 172.21.15.38:0/357229891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960107580 msgr2=0x7f896019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960107580 0x7f896019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 -- 172.21.15.38:0/357229891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896019f6a0 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f895ffff640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 0x7f8960198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8964e32640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f895000d770 tx=0x7f895000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950017070 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8950005500 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896019f990 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950004e00 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.730+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896019fe90 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.732+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89600fd780 con 0x7f8960102320 2024-04-21T23:59:48.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.733+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8950004fa0 con 0x7f8960102320 2024-04-21T23:59:48.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.733+0000 7f895dffb640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 0x7f894c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:48.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.733+0000 7f895ffff640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 0x7f894c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:48.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.733+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f89500ced00 con 0x7f8960102320 2024-04-21T23:59:48.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.734+0000 7f895ffff640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 0x7f894c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8948005d00 tx=0x7f8948009550 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:48.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.738+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f895009ba20 con 0x7f8960102320 2024-04-21T23:59:48.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.924+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8960062340 con 0x7f8960102320 2024-04-21T23:59:48.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.926+0000 7f895dffb640 1 -- 172.21.15.38:0/357229891 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7f895009b320 con 0x7f8960102320 2024-04-21T23:59:48.919 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:48.919 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 msgr2=0x7f894c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 0x7f894c0a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8948005d00 tx=0x7f8948009550 comp rx=0 tx=0).stop 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 msgr2=0x7f8960197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f895000d770 tx=0x7f895000dc30 comp rx=0 tx=0).stop 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 shutdown_connections 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f894c0a2b90 0x7f894c0a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8960107580 0x7f896019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8960104c50 0x7f8960198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 --2- 172.21.15.38:0/357229891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8960102320 0x7f8960197ad0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:48.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 >> 172.21.15.38:0/357229891 conn(0x7f89600fbc90 msgr2=0x7f8960102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:48.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 shutdown_connections 2024-04-21T23:59:48.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:48.928+0000 7f8967096640 1 -- 172.21.15.38:0/357229891 wait complete. 2024-04-21T23:59:48.932 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:59:48.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:59:49.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.044+0000 7fc634468640 1 -- 172.21.15.38:0/782804068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 msgr2=0x7fc62c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.044+0000 7fc634468640 1 --2- 172.21.15.38:0/782804068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c100ac0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc628009900 tx=0x7fc62802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:49.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- 172.21.15.38:0/782804068 shutdown_connections 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- 172.21.15.38:0/782804068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c102250 0x7fc62c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- 172.21.15.38:0/782804068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c101890 0x7fc62c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- 172.21.15.38:0/782804068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c100ac0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- 172.21.15.38:0/782804068 >> 172.21.15.38:0/782804068 conn(0x7fc62c0fbc30 msgr2=0x7fc62c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- 172.21.15.38:0/782804068 shutdown_connections 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- 172.21.15.38:0/782804068 wait complete. 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 Processor -- start 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- start start 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c101890 0x7fc62c196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c102250 0x7fc62c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c10c360 con 0x7fc62c1006a0 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c10c1e0 con 0x7fc62c101890 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc634468640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c10c4e0 con 0x7fc62c102250 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc632204640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc632204640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56394/0 (socket says 172.21.15.38:56394) 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.045+0000 7fc632204640 1 -- 172.21.15.38:0/2758714490 learned_addr learned my addr 172.21.15.38:0/2758714490 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632a05640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c102250 0x7fc62c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc631a03640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c101890 0x7fc62c196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 -- 172.21.15.38:0/2758714490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c102250 msgr2=0x7fc62c19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c102250 0x7fc62c19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 -- 172.21.15.38:0/2758714490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c101890 msgr2=0x7fc62c196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c101890 0x7fc62c196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 -- 172.21.15.38:0/2758714490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c19e650 con 0x7fc62c1006a0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc632204640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc628000c00 tx=0x7fc62802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628004450 con 0x7fc62c1006a0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6280054e0 con 0x7fc62c1006a0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6280476e0 con 0x7fc62c1006a0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c069d40 con 0x7fc62c1006a0 2024-04-21T23:59:49.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.046+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62c06a210 con 0x7fc62c1006a0 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.047+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc62c10cb60 con 0x7fc62c1006a0 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.048+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc6280035f0 con 0x7fc62c1006a0 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.049+0000 7fc6237fe640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.049+0000 7fc631a03640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.049+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc6280f3320 con 0x7fc62c1006a0 2024-04-21T23:59:49.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.049+0000 7fc631a03640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 0x7fc6180a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc62c197f40 tx=0x7fc61c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.053+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc628003e20 con 0x7fc62c1006a0 2024-04-21T23:59:49.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.237+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc62c107ad0 con 0x7fc62c1006a0 2024-04-21T23:59:49.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.238+0000 7fc6237fe640 1 -- 172.21.15.38:0/2758714490 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc6280c0040 con 0x7fc62c1006a0 2024-04-21T23:59:49.231 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 msgr2=0x7fc6180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 0x7fc6180a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc62c197f40 tx=0x7fc61c005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 msgr2=0x7fc62c196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc628000c00 tx=0x7fc62802c6c0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 shutdown_connections 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc62c102250 0x7fc62c19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc62c101890 0x7fc62c196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 --2- 172.21.15.38:0/2758714490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc62c1006a0 0x7fc62c196a30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 >> 172.21.15.38:0/2758714490 conn(0x7fc62c0fbc30 msgr2=0x7fc62c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 shutdown_connections 2024-04-21T23:59:49.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.240+0000 7fc634468640 1 -- 172.21.15.38:0/2758714490 wait complete. 2024-04-21T23:59:49.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:59:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- 172.21.15.38:0/1597362118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 msgr2=0x7f8c10102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- 172.21.15.38:0/1597362118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10102f70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009930 tx=0x7f8c0402bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- 172.21.15.38:0/1597362118 shutdown_connections 2024-04-21T23:59:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- 172.21.15.38:0/1597362118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 0x7f8c10109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- 172.21.15.38:0/1597362118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10102f70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- 172.21.15.38:0/1597362118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 0x7f8c100ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- 172.21.15.38:0/1597362118 >> 172.21.15.38:0/1597362118 conn(0x7f8c100fb370 msgr2=0x7f8c100fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- 172.21.15.38:0/1597362118 shutdown_connections 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- 172.21.15.38:0/1597362118 wait complete. 2024-04-21T23:59:49.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 Processor -- start 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- start start 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 0x7f8c10197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 0x7f8c1019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d430 con 0x7f8c10102af0 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d2b0 con 0x7f8c101034b0 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.356+0000 7f8c1759a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1010d5b0 con 0x7f8c100ffa80 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56414/0 (socket says 172.21.15.38:56414) 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 -- 172.21.15.38:0/692177026 learned_addr learned my addr 172.21.15.38:0/692177026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c15b37640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 0x7f8c1019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c15336640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 0x7f8c10197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 -- 172.21.15.38:0/692177026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 msgr2=0x7f8c10197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 0x7f8c10197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 -- 172.21.15.38:0/692177026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 msgr2=0x7f8c1019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 0x7f8c1019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 -- 172.21.15.38:0/692177026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1019f750 con 0x7f8c10102af0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c14b35640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000c00 tx=0x7f8c0402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04041070 con 0x7f8c10102af0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c04004450 con 0x7f8c10102af0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1019f9e0 con 0x7f8c10102af0 2024-04-21T23:59:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04034440 con 0x7f8c10102af0 2024-04-21T23:59:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.357+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1019fed0 con 0x7f8c10102af0 2024-04-21T23:59:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.358+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c10108c40 con 0x7f8c10102af0 2024-04-21T23:59:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.359+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8c04044070 con 0x7f8c10102af0 2024-04-21T23:59:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.360+0000 7f8c0e7fc640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.360+0000 7f8c15336640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.360+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8c040f0e20 con 0x7f8c10102af0 2024-04-21T23:59:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.360+0000 7f8c15336640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 0x7f8bf80a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c10069a80 tx=0x7f8c00005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.364+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8c040bdb70 con 0x7f8c10102af0 2024-04-21T23:59:49.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.551+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c1002d390 con 0x7f8c10102af0 2024-04-21T23:59:49.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.552+0000 7f8c0e7fc640 1 -- 172.21.15.38:0/692177026 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f8c040bd470 con 0x7f8c10102af0 2024-04-21T23:59:49.545 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-21T23:59:49.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 msgr2=0x7f8bf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 0x7f8bf80a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c10069a80 tx=0x7f8c00005f90 comp rx=0 tx=0).stop 2024-04-21T23:59:49.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 msgr2=0x7f8c10198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000c00 tx=0x7f8c0402c640 comp rx=0 tx=0).stop 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 shutdown_connections 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8bf80a2b90 0x7f8bf80a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c101034b0 0x7f8c1019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c10102af0 0x7f8c10198070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 --2- 172.21.15.38:0/692177026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c100ffa80 0x7f8c10197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 >> 172.21.15.38:0/692177026 conn(0x7f8c100fb370 msgr2=0x7f8c10105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 shutdown_connections 2024-04-21T23:59:49.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.554+0000 7f8c1759a640 1 -- 172.21.15.38:0/692177026 wait complete. 2024-04-21T23:59:49.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:49 smithi038 ceph-mon[27972]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-04-21T23:59:49.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:49 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/357229891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:49.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:49 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2758714490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:49.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:59:49.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- 172.21.15.38:0/3280011804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 msgr2=0x7fc994108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 --2- 172.21.15.38:0/3280011804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc994108ae0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc984009ca0 tx=0x7fc98402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- 172.21.15.38:0/3280011804 shutdown_connections 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 --2- 172.21.15.38:0/3280011804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc994108ae0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 --2- 172.21.15.38:0/3280011804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc994101890 0x7fc994101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 --2- 172.21.15.38:0/3280011804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9941006a0 0x7fc994100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- 172.21.15.38:0/3280011804 >> 172.21.15.38:0/3280011804 conn(0x7fc9940fbc10 msgr2=0x7fc9940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- 172.21.15.38:0/3280011804 shutdown_connections 2024-04-21T23:59:49.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- 172.21.15.38:0/3280011804 wait complete. 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 Processor -- start 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.670+0000 7fc99baf4640 1 -- start start 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9941006a0 0x7fc994104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc994101890 0x7fc994102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc99410c320 con 0x7fc994102250 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc99410c1a0 con 0x7fc9941006a0 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc99410c4a0 con 0x7fc994101890 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56420/0 (socket says 172.21.15.38:56420) 2024-04-21T23:59:49.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 -- 172.21.15.38:0/878060868 learned_addr learned my addr 172.21.15.38:0/878060868 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc999890640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9941006a0 0x7fc994104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99908f640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc994101890 0x7fc994102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 -- 172.21.15.38:0/878060868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc994101890 msgr2=0x7fc994102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc994101890 0x7fc994102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 -- 172.21.15.38:0/878060868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9941006a0 msgr2=0x7fc994104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9941006a0 0x7fc994104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 -- 172.21.15.38:0/878060868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc994103a40 con 0x7fc994102250 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99a091640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc98402c700 tx=0x7fc984032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984039040 con 0x7fc994102250 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9941a1b70 con 0x7fc994102250 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984044070 con 0x7fc994102250 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840042a0 con 0x7fc994102250 2024-04-21T23:59:49.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.671+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9941a2110 con 0x7fc994102250 2024-04-21T23:59:49.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.672+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9941a2340 con 0x7fc994102250 2024-04-21T23:59:49.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.673+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc984003970 con 0x7fc994102250 2024-04-21T23:59:49.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.674+0000 7fc992ffd640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 0x7fc97c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.674+0000 7fc999890640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 0x7fc97c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.674+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc9840f1ea0 con 0x7fc994102250 2024-04-21T23:59:49.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.675+0000 7fc999890640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 0x7fc97c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc98c005fd0 tx=0x7fc98c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.677+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc9840bebf0 con 0x7fc994102250 2024-04-21T23:59:49.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:49 smithi203 ceph-mon[31431]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-04-21T23:59:49.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:49 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/357229891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:49.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:49 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2758714490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:49.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:49 smithi111 ceph-mon[31543]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-04-21T23:59:49.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:49 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/357229891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:49.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:49 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2758714490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:49.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.860+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc9941a2510 con 0x7fc994102250 2024-04-21T23:59:49.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.860+0000 7fc992ffd640 1 -- 172.21.15.38:0/878060868 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fc9840be4f0 con 0x7fc994102250 2024-04-21T23:59:49.853 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 msgr2=0x7fc97c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 0x7fc97c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc98c005fd0 tx=0x7fc98c005850 comp rx=0 tx=0).stop 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 msgr2=0x7fc9941031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc98402c700 tx=0x7fc984032f20 comp rx=0 tx=0).stop 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 shutdown_connections 2024-04-21T23:59:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc97c0a2b90 0x7fc97c0a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc994102250 0x7fc9941031b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc994101890 0x7fc994102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 --2- 172.21.15.38:0/878060868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9941006a0 0x7fc994104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 >> 172.21.15.38:0/878060868 conn(0x7fc9940fbc10 msgr2=0x7fc9940fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 shutdown_connections 2024-04-21T23:59:49.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.862+0000 7fc99baf4640 1 -- 172.21.15.38:0/878060868 wait complete. 2024-04-21T23:59:49.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2144217402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04102250 msgr2=0x7f6c04108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2144217402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04102250 0x7f6c04108ae0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009ca0 tx=0x7f6bf402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2144217402 shutdown_connections 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2144217402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04102250 0x7f6c04108ae0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2144217402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04101890 0x7f6c04101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2144217402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 0x7f6c04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2144217402 >> 172.21.15.38:0/2144217402 conn(0x7f6c040fbc30 msgr2=0x7f6c040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:49.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2144217402 shutdown_connections 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.978+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2144217402 wait complete. 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 Processor -- start 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 -- start start 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 0x7f6c0410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04102250 0x7f6c04109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0410d440 con 0x7f6c04101890 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0410d2c0 con 0x7f6c04102250 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0c0b5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0410d5c0 con 0x7f6c041006a0 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56448/0 (socket says 172.21.15.38:56448) 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 -- 172.21.15.38:0/2825943692 learned_addr learned my addr 172.21.15.38:0/2825943692 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:49.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09e51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 0x7f6c0410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c0a652640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04102250 0x7f6c04109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 -- 172.21.15.38:0/2825943692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 msgr2=0x7f6c0410b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 0x7f6c0410b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 -- 172.21.15.38:0/2825943692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04102250 msgr2=0x7f6c04109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04102250 0x7f6c04109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.979+0000 7f6c09650640 1 -- 172.21.15.38:0/2825943692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0410a430 con 0x7f6c04101890 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c09650640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800c920 tx=0x7f6bf800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8007c70 con 0x7f6c04101890 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf8007e10 con 0x7f6c04101890 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c041a1b00 con 0x7f6c04101890 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8018890 con 0x7f6c04101890 2024-04-21T23:59:49.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.980+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c041a20a0 con 0x7f6c04101890 2024-04-21T23:59:49.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.981+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c041079c0 con 0x7f6c04101890 2024-04-21T23:59:49.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.982+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6bf8018a70 con 0x7f6c04101890 2024-04-21T23:59:49.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.982+0000 7f6c02ffd640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 0x7f6bec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:49.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.983+0000 7f6c09e51640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 0x7f6bec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:49.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.983+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6bf80cf360 con 0x7f6c04101890 2024-04-21T23:59:49.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.983+0000 7f6c09e51640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 0x7f6bec0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6c041016f0 tx=0x7f6bfc00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:49.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:49.987+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6bf8014030 con 0x7f6c04101890 2024-04-21T23:59:50.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.174+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6c040623e0 con 0x7f6c04101890 2024-04-21T23:59:50.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.174+0000 7f6c02ffd640 1 -- 172.21.15.38:0/2825943692 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f6bf809c020 con 0x7f6c04101890 2024-04-21T23:59:50.168 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-21T23:59:50.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 msgr2=0x7f6bec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 0x7f6bec0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6c041016f0 tx=0x7f6bfc00b390 comp rx=0 tx=0).stop 2024-04-21T23:59:50.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 msgr2=0x7f6c04109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800c920 tx=0x7f6bf800cde0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 shutdown_connections 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6bec0a2c10 0x7f6bec0a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c04102250 0x7f6c04109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c04101890 0x7f6c04109660 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 --2- 172.21.15.38:0/2825943692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c041006a0 0x7f6c0410b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.176+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 >> 172.21.15.38:0/2825943692 conn(0x7f6c040fbc30 msgr2=0x7f6c04104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.177+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 shutdown_connections 2024-04-21T23:59:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.177+0000 7f6c0c0b5640 1 -- 172.21.15.38:0/2825943692 wait complete. 2024-04-21T23:59:50.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:59:50.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- 172.21.15.38:0/209590444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa498104bd0 msgr2=0x7fa498106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/209590444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa498104bd0 0x7fa498106fc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009900 tx=0x7fa48c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- 172.21.15.38:0/209590444 shutdown_connections 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/209590444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 0x7fa4981098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/209590444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa498104bd0 0x7fa498106fc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/209590444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4981022a0 0x7fa498104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- 172.21.15.38:0/209590444 >> 172.21.15.38:0/209590444 conn(0x7fa4980fbc30 msgr2=0x7fa4980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- 172.21.15.38:0/209590444 shutdown_connections 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- 172.21.15.38:0/209590444 wait complete. 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 Processor -- start 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.291+0000 7fa49dbf7640 1 -- start start 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa498104bd0 0x7fa498100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 0x7fa498100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d3f0 con 0x7fa4981022a0 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d270 con 0x7fa498107500 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d570 con 0x7fa498104bd0 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56468/0 (socket says 172.21.15.38:56468) 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 -- 172.21.15.38:0/980131782 learned_addr learned my addr 172.21.15.38:0/980131782 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa497fff640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 0x7fa498100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa496ffd640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa498104bd0 0x7fa498100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 -- 172.21.15.38:0/980131782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa498104bd0 msgr2=0x7fa498100250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa498104bd0 0x7fa498100250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 -- 172.21.15.38:0/980131782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 msgr2=0x7fa498100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 0x7fa498100790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 -- 172.21.15.38:0/980131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498101020 con 0x7fa4981022a0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa4977fe640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e9f0 tx=0x7fa48800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48800cd80 con 0x7fa4981022a0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4981a1bf0 con 0x7fa4981022a0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa488010910 con 0x7fa4981022a0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.292+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488018950 con 0x7fa4981022a0 2024-04-21T23:59:50.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.293+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4981a2160 con 0x7fa4981022a0 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.294+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4980fd720 con 0x7fa4981022a0 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.295+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa488010ac0 con 0x7fa4981022a0 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.295+0000 7fa494ff9640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 0x7fa4800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.295+0000 7fa496ffd640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 0x7fa4800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.295+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4880cf630 con 0x7fa4981022a0 2024-04-21T23:59:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.296+0000 7fa496ffd640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 0x7fa4800a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa48c000c00 tx=0x7fa48c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.300+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa48809bd00 con 0x7fa4981022a0 2024-04-21T23:59:50.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:50 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/692177026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:50 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/878060868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:50 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2825943692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.500+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa49802d170 con 0x7fa4981022a0 2024-04-21T23:59:50.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.500+0000 7fa494ff9640 1 -- 172.21.15.38:0/980131782 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7fa49802d170 con 0x7fa4981022a0 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4082720653 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1866190732 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1458203717 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/71250327 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/71250327 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/247200712 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:50.493 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4200150941 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:50.494 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2769068060 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:50.494 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2024-04-21T23:59:50.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 msgr2=0x7fa4800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 0x7fa4800a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa48c000c00 tx=0x7fa48c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 msgr2=0x7fa498101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa48800e9f0 tx=0x7fa48800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 shutdown_connections 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4800a2b90 0x7fa4800a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa498107500 0x7fa498100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa498104bd0 0x7fa498100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 --2- 172.21.15.38:0/980131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4981022a0 0x7fa498101c10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 >> 172.21.15.38:0/980131782 conn(0x7fa4980fbc30 msgr2=0x7fa498102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 shutdown_connections 2024-04-21T23:59:50.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.502+0000 7fa49dbf7640 1 -- 172.21.15.38:0/980131782 wait complete. 2024-04-21T23:59:50.506 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-21T23:59:50.506 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:59:50.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2925088485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d8102250 msgr2=0x7fa3d8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2925088485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d8102250 0x7fa3d8108ae0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009ca0 tx=0x7fa3c802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2925088485 shutdown_connections 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2925088485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d8102250 0x7fa3d8108ae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2925088485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8101890 0x7fa3d8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2925088485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d81006a0 0x7fa3d8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2925088485 >> 172.21.15.38:0/2925088485 conn(0x7fa3d80fbc30 msgr2=0x7fa3d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2925088485 shutdown_connections 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.672+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2925088485 wait complete. 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 Processor -- start 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 -- start start 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d8101890 0x7fa3d819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8102250 0x7fa3d819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d810c300 con 0x7fa3d81006a0 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d810c180 con 0x7fa3d8101890 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3dcc6b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d810c480 con 0x7fa3d8102250 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56480/0 (socket says 172.21.15.38:56480) 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 -- 172.21.15.38:0/2389213088 learned_addr learned my addr 172.21.15.38:0/2389213088 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d5d9b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d8101890 0x7fa3d819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d6d9d640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8102250 0x7fa3d819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 -- 172.21.15.38:0/2389213088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8102250 msgr2=0x7fa3d819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8102250 0x7fa3d819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 -- 172.21.15.38:0/2389213088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d8101890 msgr2=0x7fa3d819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d8101890 0x7fa3d819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.673+0000 7fa3d659c640 1 -- 172.21.15.38:0/2389213088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d819f5b0 con 0x7fa3d81006a0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3d659c640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d950 tx=0x7fa3d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0017020 con 0x7fa3d81006a0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3d0009c30 con 0x7fa3d81006a0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000f740 con 0x7fa3d81006a0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d819f8a0 con 0x7fa3d81006a0 2024-04-21T23:59:50.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.674+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d819fda0 con 0x7fa3d81006a0 2024-04-21T23:59:50.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.675+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d81078f0 con 0x7fa3d81006a0 2024-04-21T23:59:50.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.676+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa3d001a020 con 0x7fa3d81006a0 2024-04-21T23:59:50.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.676+0000 7fa3c77fe640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 0x7fa3c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.677+0000 7fa3d5d9b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 0x7fa3c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.677+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa3d00ced00 con 0x7fa3d81006a0 2024-04-21T23:59:50.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.677+0000 7fa3d5d9b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 0x7fa3c00a50d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3d806a6a0 tx=0x7fa3cc00c4a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.681+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa3d0021070 con 0x7fa3d81006a0 2024-04-21T23:59:50.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:50 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/692177026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:50 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/878060868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:50 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2825943692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:50 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/692177026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:50 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/878060868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:50 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2825943692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:50.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.866+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3d80623e0 con 0x7fa3d81006a0 2024-04-21T23:59:50.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.867+0000 7fa3c77fe640 1 -- 172.21.15.38:0/2389213088 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fa3d0009de0 con 0x7fa3d81006a0 2024-04-21T23:59:50.860 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:50.860 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:50.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 msgr2=0x7fa3c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 0x7fa3c00a50d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3d806a6a0 tx=0x7fa3cc00c4a0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 msgr2=0x7fa3d806a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d950 tx=0x7fa3d000de10 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 shutdown_connections 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa3c00a2c10 0x7fa3c00a50d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3d8102250 0x7fa3d819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3d8101890 0x7fa3d819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 --2- 172.21.15.38:0/2389213088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3d81006a0 0x7fa3d806a930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 >> 172.21.15.38:0/2389213088 conn(0x7fa3d80fbc30 msgr2=0x7fa3d8104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 shutdown_connections 2024-04-21T23:59:50.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.869+0000 7fa3dcc6b640 1 -- 172.21.15.38:0/2389213088 wait complete. 2024-04-21T23:59:50.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:59:50.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/2759171542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98101310 msgr2=0x7fdb98101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/2759171542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98101310 0x7fdb98101790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009ca0 tx=0x7fdb8802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/2759171542 shutdown_connections 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/2759171542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98101310 0x7fdb98101790 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/2759171542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98100970 0x7fdb98100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/2759171542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 0x7fdb98100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.986+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/2759171542 >> 172.21.15.38:0/2759171542 conn(0x7fdb980fbc30 msgr2=0x7fdb980fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/2759171542 shutdown_connections 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/2759171542 wait complete. 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 Processor -- start 2024-04-21T23:59:50.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- start start 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 0x7fdb98199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98101310 0x7fdb981a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9810f610 con 0x7fdb98100970 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9810f490 con 0x7fdb98100010 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9f5bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9810f790 con 0x7fdb98101310 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9cb5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9cb5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56506/0 (socket says 172.21.15.38:56506) 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.987+0000 7fdb9cb5a640 1 -- 172.21.15.38:0/1677209682 learned_addr learned my addr 172.21.15.38:0/1677209682 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9db5c640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98101310 0x7fdb981a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9d35b640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 0x7fdb98199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 -- 172.21.15.38:0/1677209682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98101310 msgr2=0x7fdb981a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98101310 0x7fdb981a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 -- 172.21.15.38:0/1677209682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 msgr2=0x7fdb98199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 0x7fdb98199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 -- 172.21.15.38:0/1677209682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb981a1890 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9cb5a640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00d770 tx=0x7fdb8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c017070 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb981a1b80 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb8c005500 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c004e00 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.988+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb981a2080 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.989+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb9810ad60 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.990+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdb8c004fa0 con 0x7fdb98100970 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.991+0000 7fdb967fc640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 0x7fdb800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.991+0000 7fdb9d35b640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 0x7fdb800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:50.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.991+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdb8c0cf1f0 con 0x7fdb98100970 2024-04-21T23:59:50.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.991+0000 7fdb9d35b640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 0x7fdb800a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdb90005fd0 tx=0x7fdb90005e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:50.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:50.994+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdb8c09bf10 con 0x7fdb98100970 2024-04-21T23:59:51.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.179+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb98062340 con 0x7fdb98100970 2024-04-21T23:59:51.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.179+0000 7fdb967fc640 1 -- 172.21.15.38:0/1677209682 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fdb8c09b810 con 0x7fdb98100970 2024-04-21T23:59:51.172 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-21T23:59:51.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 msgr2=0x7fdb800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 0x7fdb800a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdb90005fd0 tx=0x7fdb90005e40 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 msgr2=0x7fdb9819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c00d770 tx=0x7fdb8c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 shutdown_connections 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdb800a2b90 0x7fdb800a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb98101310 0x7fdb981a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb98100970 0x7fdb9819a1c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 --2- 172.21.15.38:0/1677209682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb98100010 0x7fdb98199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.181+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 >> 172.21.15.38:0/1677209682 conn(0x7fdb980fbc30 msgr2=0x7fdb980fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.182+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 shutdown_connections 2024-04-21T23:59:51.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.182+0000 7fdb9f5bf640 1 -- 172.21.15.38:0/1677209682 wait complete. 2024-04-21T23:59:51.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:59:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 -- 172.21.15.38:0/798798916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c102d90 msgr2=0x7fe69c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/798798916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c102d90 0x7fe69c1031b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009900 tx=0x7fe68c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 -- 172.21.15.38:0/798798916 shutdown_connections 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/798798916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c1001c0 0x7fe69c109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/798798916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c0ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/798798916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c102d90 0x7fe69c1031b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 -- 172.21.15.38:0/798798916 >> 172.21.15.38:0/798798916 conn(0x7fe69c0fb380 msgr2=0x7fe69c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 -- 172.21.15.38:0/798798916 shutdown_connections 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.311+0000 7fe6a10aa640 1 -- 172.21.15.38:0/798798916 wait complete. 2024-04-21T23:59:51.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 Processor -- start 2024-04-21T23:59:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 -- start start 2024-04-21T23:59:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c102d90 0x7fe69c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c10d460 con 0x7fe69c1001c0 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c10d2e0 con 0x7fe69c102d90 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe6a10aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c10d5e0 con 0x7fe69c0ff800 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56530/0 (socket says 172.21.15.38:56530) 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 -- 172.21.15.38:0/1631210014 learned_addr learned my addr 172.21.15.38:0/1631210014 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:51.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69ad9d640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69b59e640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c102d90 0x7fe69c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 -- 172.21.15.38:0/1631210014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 msgr2=0x7fe69c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 -- 172.21.15.38:0/1631210014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c102d90 msgr2=0x7fe69c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c102d90 0x7fe69c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69a59c640 1 -- 172.21.15.38:0/1631210014 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c19f740 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.312+0000 7fe69ad9d640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe69a59c640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c970 tx=0x7fe69000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690007bb0 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69c19fa30 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe690007d50 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69000f450 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.313+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69c19ff80 con 0x7fe69c1001c0 2024-04-21T23:59:51.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.314+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69c108d20 con 0x7fe69c1001c0 2024-04-21T23:59:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.315+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe690005020 con 0x7fe69c1001c0 2024-04-21T23:59:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.315+0000 7fe683fff640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 0x7fe6840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.316+0000 7fe69ad9d640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 0x7fe6840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.316+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe6900cf090 con 0x7fe69c1001c0 2024-04-21T23:59:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.316+0000 7fe69ad9d640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 0x7fe6840a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0096c0 tx=0x7fe68c02df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.320+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe690014030 con 0x7fe69c1001c0 2024-04-21T23:59:51.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:51 smithi038 ceph-mon[27972]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 1 op/s 2024-04-21T23:59:51.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/980131782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:51.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2389213088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:51.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1677209682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:51.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.506+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe69c062340 con 0x7fe69c1001c0 2024-04-21T23:59:51.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.506+0000 7fe683fff640 1 -- 172.21.15.38:0/1631210014 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fe69009bdb0 con 0x7fe69c1001c0 2024-04-21T23:59:51.500 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-21T23:59:51.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 msgr2=0x7fe6840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 0x7fe6840a50d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0096c0 tx=0x7fe68c02df90 comp rx=0 tx=0).stop 2024-04-21T23:59:51.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 msgr2=0x7fe69c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c970 tx=0x7fe69000ce30 comp rx=0 tx=0).stop 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 shutdown_connections 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe6840a2c10 0x7fe6840a50d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe69c102d90 0x7fe69c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe69c1001c0 0x7fe69c198060 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 --2- 172.21.15.38:0/1631210014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe69c0ff800 0x7fe69c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 >> 172.21.15.38:0/1631210014 conn(0x7fe69c0fb380 msgr2=0x7fe69c105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 shutdown_connections 2024-04-21T23:59:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.508+0000 7fe6a10aa640 1 -- 172.21.15.38:0/1631210014 wait complete. 2024-04-21T23:59:51.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:59:51.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 -- 172.21.15.38:0/4149273101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6264101890 msgr2=0x7f6264101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 --2- 172.21.15.38:0/4149273101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6264101890 0x7f6264101d10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6258009900 tx=0x7f625802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:51.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 -- 172.21.15.38:0/4149273101 shutdown_connections 2024-04-21T23:59:51.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 --2- 172.21.15.38:0/4149273101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264102250 0x7f6264108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 --2- 172.21.15.38:0/4149273101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6264101890 0x7f6264101d10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 --2- 172.21.15.38:0/4149273101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62641006a0 0x7f6264100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 -- 172.21.15.38:0/4149273101 >> 172.21.15.38:0/4149273101 conn(0x7f62640fbc30 msgr2=0x7f62640fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 -- 172.21.15.38:0/4149273101 shutdown_connections 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 -- 172.21.15.38:0/4149273101 wait complete. 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.626+0000 7f6269fa7640 1 Processor -- start 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 -- start start 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264101890 0x7f6264197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 0x7f626419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f626410d440 con 0x7f62641006a0 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f626410d2c0 con 0x7f6264101890 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6269fa7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f626410d5c0 con 0x7f6264102250 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56540/0 (socket says 172.21.15.38:56540) 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 -- 172.21.15.38:0/192143272 learned_addr learned my addr 172.21.15.38:0/192143272 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6263fff640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 0x7f626419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6262ffd640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264101890 0x7f6264197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 -- 172.21.15.38:0/192143272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 msgr2=0x7f626419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 0x7f626419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 -- 172.21.15.38:0/192143272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264101890 msgr2=0x7f6264197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264101890 0x7f6264197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 -- 172.21.15.38:0/192143272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626419f660 con 0x7f62641006a0 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f6263fff640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 0x7f626419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.627+0000 7f62637fe640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f625400d950 tx=0x7f625400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.628+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254017020 con 0x7f62641006a0 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.628+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6254009c30 con 0x7f62641006a0 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.628+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626419f950 con 0x7f62641006a0 2024-04-21T23:59:51.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.628+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625400f740 con 0x7f62641006a0 2024-04-21T23:59:51.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.628+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626419fea0 con 0x7f62641006a0 2024-04-21T23:59:51.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.629+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6264107a90 con 0x7f62641006a0 2024-04-21T23:59:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.630+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f625401a020 con 0x7f62641006a0 2024-04-21T23:59:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.630+0000 7f6260ff9640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 0x7f624c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.631+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f62540ced90 con 0x7f62641006a0 2024-04-21T23:59:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.631+0000 7f6262ffd640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 0x7f624c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.631+0000 7f6262ffd640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 0x7f624c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6258036040 tx=0x7f6258005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.635+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6254021070 con 0x7f62641006a0 2024-04-21T23:59:51.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:51 smithi203 ceph-mon[31431]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 1 op/s 2024-04-21T23:59:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/980131782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2389213088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:51.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1677209682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:51.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.820+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f62640623e0 con 0x7f62641006a0 2024-04-21T23:59:51.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.820+0000 7f6260ff9640 1 -- 172.21.15.38:0/192143272 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f6254009de0 con 0x7f62641006a0 2024-04-21T23:59:51.813 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 msgr2=0x7f624c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 0x7f624c0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6258036040 tx=0x7f6258005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 msgr2=0x7f6264197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f625400d950 tx=0x7f625400de10 comp rx=0 tx=0).stop 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 shutdown_connections 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f624c0a2b90 0x7f624c0a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6264102250 0x7f626419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6264101890 0x7f6264197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 --2- 172.21.15.38:0/192143272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62641006a0 0x7f6264197a40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 >> 172.21.15.38:0/192143272 conn(0x7f62640fbc30 msgr2=0x7f6264104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 shutdown_connections 2024-04-21T23:59:51.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.822+0000 7f6269fa7640 1 -- 172.21.15.38:0/192143272 wait complete. 2024-04-21T23:59:51.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:59:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:51 smithi111 ceph-mon[31543]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 1 op/s 2024-04-21T23:59:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/980131782' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2389213088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1677209682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:59:51.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 -- 172.21.15.38:0/1501279840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10101680 msgr2=0x7f1b10101aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/1501279840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10101680 0x7f1b10101aa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009900 tx=0x7f1b0002bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:51.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 -- 172.21.15.38:0/1501279840 shutdown_connections 2024-04-21T23:59:51.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/1501279840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10103230 0x7f1b10109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/1501279840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10102870 0x7f1b10102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/1501279840 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10101680 0x7f1b10101aa0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 -- 172.21.15.38:0/1501279840 >> 172.21.15.38:0/1501279840 conn(0x7f1b100fcc10 msgr2=0x7f1b100ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 -- 172.21.15.38:0/1501279840 shutdown_connections 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.938+0000 7f1b14f9f640 1 -- 172.21.15.38:0/1501279840 wait complete. 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 Processor -- start 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 -- start start 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10101680 0x7f1b10199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10103230 0x7f1b101a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1010d300 con 0x7f1b10102870 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1010d180 con 0x7f1b10101680 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b14f9f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1010d480 con 0x7f1b10103230 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56556/0 (socket says 172.21.15.38:56556) 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 -- 172.21.15.38:0/2974777556 learned_addr learned my addr 172.21.15.38:0/2974777556 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0f7fe640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10101680 0x7f1b10199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0ffff640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10103230 0x7f1b101a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 -- 172.21.15.38:0/2974777556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10103230 msgr2=0x7f1b101a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10103230 0x7f1b101a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 -- 172.21.15.38:0/2974777556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10101680 msgr2=0x7f1b10199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10101680 0x7f1b10199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.939+0000 7f1b0effd640 1 -- 172.21.15.38:0/2974777556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b101a1850 con 0x7f1b10102870 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b0effd640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400d770 tx=0x7f1b0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04017070 con 0x7f1b10102870 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101a1b40 con 0x7f1b10102870 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b04005500 con 0x7f1b10102870 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04004e00 con 0x7f1b10102870 2024-04-21T23:59:51.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.940+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b101a2060 con 0x7f1b10102870 2024-04-21T23:59:51.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.941+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b101a2230 con 0x7f1b10102870 2024-04-21T23:59:51.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.942+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1b04004fa0 con 0x7f1b10102870 2024-04-21T23:59:51.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.942+0000 7f1b0cff9640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 0x7f1af80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:51.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.943+0000 7f1b0f7fe640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 0x7f1af80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:51.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.943+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1b040cecf0 con 0x7f1b10102870 2024-04-21T23:59:51.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.943+0000 7f1b0f7fe640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 0x7f1af80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b00005ab0 tx=0x7f1b00005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:51.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:51.947+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1b0409b360 con 0x7f1b10102870 2024-04-21T23:59:52.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.133+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1b101a2400 con 0x7f1b10102870 2024-04-21T23:59:52.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.134+0000 7f1b0cff9640 1 -- 172.21.15.38:0/2974777556 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f1b0409f660 con 0x7f1b10102870 2024-04-21T23:59:52.127 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-21T23:59:52.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 msgr2=0x7f1af80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 0x7f1af80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b00005ab0 tx=0x7f1b00005fd0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 msgr2=0x7f1b1019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400d770 tx=0x7f1b0400dc30 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 shutdown_connections 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f1af80a2b90 0x7f1af80a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b10103230 0x7f1b101a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b10102870 0x7f1b1019a180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 --2- 172.21.15.38:0/2974777556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b10101680 0x7f1b10199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 >> 172.21.15.38:0/2974777556 conn(0x7f1b100fcc10 msgr2=0x7f1b10073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 shutdown_connections 2024-04-21T23:59:52.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.136+0000 7f1b14f9f640 1 -- 172.21.15.38:0/2974777556 wait complete. 2024-04-21T23:59:52.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 -- 172.21.15.38:0/2041773087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80101360 msgr2=0x7f6d801017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 --2- 172.21.15.38:0/2041773087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80101360 0x7f6d801017e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009ca0 tx=0x7f6d7002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 -- 172.21.15.38:0/2041773087 shutdown_connections 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 --2- 172.21.15.38:0/2041773087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80101360 0x7f6d801017e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 --2- 172.21.15.38:0/2041773087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d801009c0 0x7f6d80100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 --2- 172.21.15.38:0/2041773087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d80100060 0x7f6d80100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.249+0000 7f6d873f5640 1 -- 172.21.15.38:0/2041773087 >> 172.21.15.38:0/2041773087 conn(0x7f6d800fbc00 msgr2=0x7f6d800fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- 172.21.15.38:0/2041773087 shutdown_connections 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- 172.21.15.38:0/2041773087 wait complete. 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 Processor -- start 2024-04-21T23:59:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- start start 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d801009c0 0x7f6d8019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 0x7f6d801a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010f5e0 con 0x7f6d80100060 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010f460 con 0x7f6d801009c0 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d873f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010f760 con 0x7f6d80101360 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d85191640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d85191640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56588/0 (socket says 172.21.15.38:56588) 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.250+0000 7f6d85191640 1 -- 172.21.15.38:0/4214534056 learned_addr learned my addr 172.21.15.38:0/4214534056 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d84990640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d801009c0 0x7f6d8019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85992640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 0x7f6d801a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 -- 172.21.15.38:0/4214534056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 msgr2=0x7f6d801a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 0x7f6d801a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 -- 172.21.15.38:0/4214534056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d801009c0 msgr2=0x7f6d8019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d801009c0 0x7f6d8019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 -- 172.21.15.38:0/4214534056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d801a1850 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85992640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 0x7f6d801a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d85191640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c920 tx=0x7f6d7800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78007c70 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d78007e10 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78018890 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d801a1b40 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.251+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d801a2090 con 0x7f6d80100060 2024-04-21T23:59:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.252+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d8010af10 con 0x7f6d80100060 2024-04-21T23:59:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.253+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d78018a50 con 0x7f6d80100060 2024-04-21T23:59:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.254+0000 7f6d7e7fc640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 0x7f6d680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.254+0000 7f6d84990640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 0x7f6d680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.254+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6d780cf750 con 0x7f6d80100060 2024-04-21T23:59:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.254+0000 7f6d84990640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 0x7f6d680a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400bbe0 tx=0x7f6d7400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.258+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6d78014030 con 0x7f6d80100060 2024-04-21T23:59:52.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.453+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6d8002d420 con 0x7f6d80100060 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.454+0000 7f6d7e7fc640 1 -- 172.21.15.38:0/4214534056 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7f6d7809c410 con 0x7f6d80100060 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4082720653 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1866190732 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1458203717 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/71250327 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/71250327 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:52.447 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/247200712 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:52.448 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4200150941 2024-04-22T23:51:57.369607+0000 2024-04-21T23:59:52.448 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2769068060 2024-04-22T23:52:25.561345+0000 2024-04-21T23:59:52.448 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 msgr2=0x7f6d680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 0x7f6d680a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400bbe0 tx=0x7f6d7400b390 comp rx=0 tx=0).stop 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 msgr2=0x7f6d80199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c920 tx=0x7f6d7800cde0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 shutdown_connections 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6d680a2b90 0x7f6d680a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d80101360 0x7f6d801a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d801009c0 0x7f6d8019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 --2- 172.21.15.38:0/4214534056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d80100060 0x7f6d80199c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 >> 172.21.15.38:0/4214534056 conn(0x7f6d800fbc00 msgr2=0x7f6d800fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 shutdown_connections 2024-04-21T23:59:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:52.456+0000 7f6d873f5640 1 -- 172.21.15.38:0/4214534056 wait complete. 2024-04-21T23:59:52.459 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi038.front.sepia.ceph.com... 2024-04-21T23:59:52.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:52.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:59:52.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[72099]: fuse finished with error 0 and tester_r 0 2024-04-21T23:59:52.484 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T23:59:52.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1631210014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/192143272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2974777556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1631210014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/192143272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2974777556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1631210014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/192143272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:59:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2974777556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:59:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:53 smithi038 ceph-mon[27972]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:59:53.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:53 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4214534056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:53 smithi203 ceph-mon[31431]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:59:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:53 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4214534056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:53 smithi111 ceph-mon[31543]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:59:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:53 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4214534056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:59:55.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:55 smithi038 ceph-mon[27972]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:55.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:55 smithi203 ceph-mon[31431]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:55 smithi111 ceph-mon[31543]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:57 smithi038 ceph-mon[27972]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:57.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:57 smithi203 ceph-mon[31431]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:57.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:57 smithi111 ceph-mon[31543]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T23:59:58.586 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2024-04-21T23:59:58.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:59:58.586 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T23:59:58.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:58.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- 172.21.15.38:0/1218233298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 msgr2=0x7fd630101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:58.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 --2- 172.21.15.38:0/1218233298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630101bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:58.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- 172.21.15.38:0/1218233298 shutdown_connections 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 --2- 172.21.15.38:0/1218233298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd630103360 0x7fd630109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 --2- 172.21.15.38:0/1218233298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6301029a0 0x7fd630102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 --2- 172.21.15.38:0/1218233298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630101bd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- 172.21.15.38:0/1218233298 >> 172.21.15.38:0/1218233298 conn(0x7fd6300fcd40 msgr2=0x7fd6300ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- 172.21.15.38:0/1218233298 shutdown_connections 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- 172.21.15.38:0/1218233298 wait complete. 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 Processor -- start 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.764+0000 7fd6355e5640 1 -- start start 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6301029a0 0x7fd630198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630103360 0x7fd63019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010d4a0 con 0x7fd6301017b0 2024-04-21T23:59:58.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010d320 con 0x7fd630103360 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010d620 con 0x7fd6301029a0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57630/0 (socket says 172.21.15.38:57630) 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 -- 172.21.15.38:0/2288215594 learned_addr learned my addr 172.21.15.38:0/2288215594 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62e7fc640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6301029a0 0x7fd630198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62f7fe640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630103360 0x7fd63019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 -- 172.21.15.38:0/2288215594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6301029a0 msgr2=0x7fd630198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6301029a0 0x7fd630198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 -- 172.21.15.38:0/2288215594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630103360 msgr2=0x7fd63019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630103360 0x7fd63019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 -- 172.21.15.38:0/2288215594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63019f770 con 0x7fd6301017b0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd62effd640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd620000c00 tx=0x7fd620004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620041070 con 0x7fd6301017b0 2024-04-21T23:59:58.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63019fa00 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6200054e0 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.765+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200346a0 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.766+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63019fef0 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.767+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd630108b50 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.768+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd620044070 con 0x7fd6301017b0 2024-04-21T23:59:58.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.768+0000 7fd617fff640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 0x7fd6180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:58.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.768+0000 7fd62e7fc640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 0x7fd6180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:58.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.768+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6200f1100 con 0x7fd6301017b0 2024-04-21T23:59:58.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.769+0000 7fd62e7fc640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 0x7fd6180a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd630199060 tx=0x7fd624008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:58.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.773+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd6200bde20 con 0x7fd6301017b0 2024-04-21T23:59:58.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.960+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd630062340 con 0x7fd6301017b0 2024-04-21T23:59:58.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.961+0000 7fd617fff640 1 -- 172.21.15.38:0/2288215594 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fd620003e20 con 0x7fd6301017b0 2024-04-21T23:59:58.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:58.955 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 msgr2=0x7fd6180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 0x7fd6180a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd630199060 tx=0x7fd624008040 comp rx=0 tx=0).stop 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 msgr2=0x7fd630197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd620000c00 tx=0x7fd620004040 comp rx=0 tx=0).stop 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 shutdown_connections 2024-04-21T23:59:58.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd6180a2b90 0x7fd6180a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd630103360 0x7fd63019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6301029a0 0x7fd630198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 --2- 172.21.15.38:0/2288215594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6301017b0 0x7fd630197b50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 >> 172.21.15.38:0/2288215594 conn(0x7fd6300fcd40 msgr2=0x7fd630105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.963+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 shutdown_connections 2024-04-21T23:59:58.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:58.964+0000 7fd6355e5640 1 -- 172.21.15.38:0/2288215594 wait complete. 2024-04-21T23:59:58.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:59:59.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- 172.21.15.38:0/905787882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504101870 msgr2=0x7fa504101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 --2- 172.21.15.38:0/905787882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504101870 0x7fa504101cf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009900 tx=0x7fa4f802bc30 comp rx=0 tx=0).stop 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- 172.21.15.38:0/905787882 shutdown_connections 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 --2- 172.21.15.38:0/905787882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504102230 0x7fa504108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 --2- 172.21.15.38:0/905787882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504101870 0x7fa504101cf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 --2- 172.21.15.38:0/905787882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504100680 0x7fa504100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- 172.21.15.38:0/905787882 >> 172.21.15.38:0/905787882 conn(0x7fa5040fbbf0 msgr2=0x7fa5040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- 172.21.15.38:0/905787882 shutdown_connections 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- 172.21.15.38:0/905787882 wait complete. 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 Processor -- start 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 -- start start 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.076+0000 7fa50a61b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504100680 0x7fa504069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 0x7fa50419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa50410c2e0 con 0x7fa504102230 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa50410c160 con 0x7fa504100680 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa50410c460 con 0x7fa504101870 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57646/0 (socket says 172.21.15.38:57646) 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 -- 172.21.15.38:0/1335300233 learned_addr learned my addr 172.21.15.38:0/1335300233 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa503fff640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504100680 0x7fa504069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa5037fe640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 0x7fa50419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 -- 172.21.15.38:0/1335300233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 msgr2=0x7fa50419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 0x7fa50419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 -- 172.21.15.38:0/1335300233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504100680 msgr2=0x7fa504069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504100680 0x7fa504069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 -- 172.21.15.38:0/1335300233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50419f700 con 0x7fa504102230 2024-04-21T23:59:59.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa5037fe640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 0x7fa50419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa508bb8640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00eaa0 tx=0x7fa4fc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00cdb0 con 0x7fa504102230 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4fc005020 con 0x7fa504102230 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc0116b0 con 0x7fa504102230 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50419f990 con 0x7fa504102230 2024-04-21T23:59:59.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.077+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50419fe90 con 0x7fa504102230 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.078+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5041079a0 con 0x7fa504102230 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.080+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa4fc005560 con 0x7fa504102230 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.080+0000 7fa5017fa640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 0x7fa4ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.081+0000 7fa503fff640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 0x7fa4ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.081+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4fc0cf940 con 0x7fa504102230 2024-04-21T23:59:59.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.081+0000 7fa503fff640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 0x7fa4ec0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa5041016d0 tx=0x7fa4f400a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:59.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.084+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa4fc016030 con 0x7fa504102230 2024-04-21T23:59:59.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.268+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa50402d330 con 0x7fa504102230 2024-04-21T23:59:59.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.270+0000 7fa5017fa640 1 -- 172.21.15.38:0/1335300233 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fa4fc09c600 con 0x7fa504102230 2024-04-21T23:59:59.263 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-21T23:59:59.263 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 msgr2=0x7fa4ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 0x7fa4ec0a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa5041016d0 tx=0x7fa4f400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 msgr2=0x7fa50419edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00eaa0 tx=0x7fa4fc00ef60 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 shutdown_connections 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa4ec0a2b90 0x7fa4ec0a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa504102230 0x7fa50419edf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa504101870 0x7fa50419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 --2- 172.21.15.38:0/1335300233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa504100680 0x7fa504069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 >> 172.21.15.38:0/1335300233 conn(0x7fa5040fbbf0 msgr2=0x7fa504104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:59.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 shutdown_connections 2024-04-21T23:59:59.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.272+0000 7fa50a61b640 1 -- 172.21.15.38:0/1335300233 wait complete. 2024-04-21T23:59:59.276 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T23:59:59.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/2080788372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d881022e0 msgr2=0x7f4d881046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/2080788372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d881022e0 0x7f4d881046d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009ca0 tx=0x7f4d7802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/2080788372 shutdown_connections 2024-04-21T23:59:59.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/2080788372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d88109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/2080788372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d88104c10 0x7f4d88107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/2080788372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d881022e0 0x7f4d881046d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/2080788372 >> 172.21.15.38:0/2080788372 conn(0x7f4d880fbc30 msgr2=0x7f4d880fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/2080788372 shutdown_connections 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/2080788372 wait complete. 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 Processor -- start 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.388+0000 7f4d8d3d6640 1 -- start start 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d881022e0 0x7f4d88101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d881007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810d450 con 0x7f4d88104c10 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810d2d0 con 0x7f4d88107540 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8810d5d0 con 0x7f4d881022e0 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57652/0 (socket says 172.21.15.38:57652) 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 -- 172.21.15.38:0/562925722 learned_addr learned my addr 172.21.15.38:0/562925722 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d86ffd640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d881022e0 0x7f4d88101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 -- 172.21.15.38:0/562925722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d881022e0 msgr2=0x7f4d88101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d881022e0 0x7f4d88101c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d877fe640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d881007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 -- 172.21.15.38:0/562925722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 msgr2=0x7f4d881007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d881007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:59:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 -- 172.21.15.38:0/562925722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88101060 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d877fe640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d881007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d867fc640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c970 tx=0x7f4d7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c007bb0 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d7c007d50 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d881a1c80 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.389+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0187a0 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.390+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881a2180 con 0x7f4d88104c10 2024-04-21T23:59:59.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.390+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d880fd720 con 0x7f4d88104c10 2024-04-21T23:59:59.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.392+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d7c00f450 con 0x7f4d88104c10 2024-04-21T23:59:59.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.392+0000 7f4d6ffff640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 0x7f4d700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:59:59.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.392+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4d7c0cf7e0 con 0x7f4d88104c10 2024-04-21T23:59:59.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.392+0000 7f4d86ffd640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 0x7f4d700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:59:59.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.393+0000 7f4d86ffd640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 0x7f4d700a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006010 tx=0x7f4d780093f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:59:59.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.396+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4d7c014030 con 0x7f4d88104c10 2024-04-21T23:59:59.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-21T23:59:59.623+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4d8802d370 con 0x7f4d88104c10 2024-04-21T23:59:59.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:59 smithi038 ceph-mon[27972]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:59:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:59 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2288215594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:59.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 21 23:59:59 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1335300233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:59.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:59 smithi203 ceph-mon[31431]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:59:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:59 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2288215594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:59.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 21 23:59:59 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1335300233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:59 smithi111 ceph-mon[31543]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:59:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:59 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2288215594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:59:59.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 21 23:59:59 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1335300233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.445+0000 7f4d6ffff640 1 -- 172.21.15.38:0/562925722 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f4d7c09c500 con 0x7f4d88104c10 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 msgr2=0x7f4d700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 0x7f4d700a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006010 tx=0x7f4d780093f0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 msgr2=0x7f4d88100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c970 tx=0x7f4d7c00ce30 comp rx=0 tx=0).stop 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 shutdown_connections 2024-04-22T00:00:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4d700a2b90 0x7f4d700a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d88107540 0x7f4d881007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88104c10 0x7f4d88100290 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 --2- 172.21.15.38:0/562925722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d881022e0 0x7f4d88101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 >> 172.21.15.38:0/562925722 conn(0x7f4d880fbc30 msgr2=0x7f4d88102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 shutdown_connections 2024-04-22T00:00:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.448+0000 7f4d8d3d6640 1 -- 172.21.15.38:0/562925722 wait complete. 2024-04-22T00:00:00.454 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-22T00:00:00.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-22T00:00:00.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.582+0000 7f30d8e40640 1 -- 172.21.15.38:0/3711814313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 msgr2=0x7f30d4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.582+0000 7f30d8e40640 1 --2- 172.21.15.38:0/3711814313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4106f20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009900 tx=0x7f30c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 -- 172.21.15.38:0/3711814313 shutdown_connections 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 --2- 172.21.15.38:0/3711814313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 --2- 172.21.15.38:0/3711814313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4106f20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 --2- 172.21.15.38:0/3711814313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 0x7f30d41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 -- 172.21.15.38:0/3711814313 >> 172.21.15.38:0/3711814313 conn(0x7f30d40fbbf0 msgr2=0x7f30d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 -- 172.21.15.38:0/3711814313 shutdown_connections 2024-04-22T00:00:00.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 -- 172.21.15.38:0/3711814313 wait complete. 2024-04-22T00:00:00.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 Processor -- start 2024-04-22T00:00:00.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 -- start start 2024-04-22T00:00:00.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.583+0000 7f30d8e40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 0x7f30d4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410d400 con 0x7f30d4104b30 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410d280 con 0x7f30d4102200 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f30d410d580 con 0x7f30d4107460 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57666/0 (socket says 172.21.15.38:57666) 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 -- 172.21.15.38:0/2843325053 learned_addr learned my addr 172.21.15.38:0/2843325053 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d259c640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 0x7f30d4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d2d9d640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 -- 172.21.15.38:0/2843325053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 msgr2=0x7f30d419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 -- 172.21.15.38:0/2843325053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 msgr2=0x7f30d4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 0x7f30d4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 -- 172.21.15.38:0/2843325053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d419f6f0 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d2d9d640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d1d9b640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8041070 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30c8004450 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d419f980 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.584+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8034440 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.585+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d419fe70 con 0x7f30d4104b30 2024-04-22T00:00:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.586+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d40fd6e0 con 0x7f30d4104b30 2024-04-22T00:00:00.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.587+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f30c8044070 con 0x7f30d4104b30 2024-04-22T00:00:00.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.587+0000 7f30c37fe640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 0x7f30bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:00.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.587+0000 7f30d259c640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 0x7f30bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:00.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.588+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30c80f0d80 con 0x7f30d4104b30 2024-04-22T00:00:00.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.588+0000 7f30d259c640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 0x7f30bc0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009bd0 tx=0x7f30c4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:00.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.592+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f30c80bdb30 con 0x7f30d4104b30 2024-04-22T00:00:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:00 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T00:00:00.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:00 smithi038 ceph-mon[27972]: overall HEALTH_OK 2024-04-22T00:00:00.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:00 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T00:00:00.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:00 smithi203 ceph-mon[31431]: overall HEALTH_OK 2024-04-22T00:00:00.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:00.819+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f30d4062340 con 0x7f30d4104b30 2024-04-22T00:00:00.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:00 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T00:00:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:00 smithi111 ceph-mon[31543]: overall HEALTH_OK 2024-04-22T00:00:01.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.235+0000 7f30c37fe640 1 -- 172.21.15.38:0/2843325053 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f30c80bd430 con 0x7f30d4104b30 2024-04-22T00:00:01.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 msgr2=0x7f30bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:01.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 0x7f30bc0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009bd0 tx=0x7f30c4005f90 comp rx=0 tx=0).stop 2024-04-22T00:00:01.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 msgr2=0x7f30d4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:01.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c802c640 comp rx=0 tx=0).stop 2024-04-22T00:00:01.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 shutdown_connections 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30bc0a2b90 0x7f30bc0a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30d4107460 0x7f30d419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30d4104b30 0x7f30d4198010 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 --2- 172.21.15.38:0/2843325053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30d4102200 0x7f30d4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 >> 172.21.15.38:0/2843325053 conn(0x7f30d40fbbf0 msgr2=0x7f30d4102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 shutdown_connections 2024-04-22T00:00:01.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.238+0000 7f30d8e40640 1 -- 172.21.15.38:0/2843325053 wait complete. 2024-04-22T00:00:01.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T00:00:01.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 -- 172.21.15.38:0/3175312936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd720102250 msgr2=0x7fd720108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3175312936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd720102250 0x7fd720108ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd718009ca0 tx=0x7fd71802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 -- 172.21.15.38:0/3175312936 shutdown_connections 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3175312936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd720102250 0x7fd720108ae0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3175312936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720101890 0x7fd720101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3175312936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7201006a0 0x7fd720100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 -- 172.21.15.38:0/3175312936 >> 172.21.15.38:0/3175312936 conn(0x7fd7200fbc30 msgr2=0x7fd7200fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 -- 172.21.15.38:0/3175312936 shutdown_connections 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.367+0000 7fd724bb1640 1 -- 172.21.15.38:0/3175312936 wait complete. 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 Processor -- start 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 -- start start 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd720101890 0x7fd720109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720102250 0x7fd720109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd72010d440 con 0x7fd7201006a0 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd72010d2c0 con 0x7fd720101890 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd724bb1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd72010d5c0 con 0x7fd720102250 2024-04-22T00:00:01.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57696/0 (socket says 172.21.15.38:57696) 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 -- 172.21.15.38:0/3741468622 learned_addr learned my addr 172.21.15.38:0/3741468622 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71dd9b640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd720101890 0x7fd720109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71ed9d640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720102250 0x7fd720109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 -- 172.21.15.38:0/3741468622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720102250 msgr2=0x7fd720109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720102250 0x7fd720109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 -- 172.21.15.38:0/3741468622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd720101890 msgr2=0x7fd720109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd720101890 0x7fd720109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.368+0000 7fd71e59c640 1 -- 172.21.15.38:0/3741468622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72010a430 con 0x7fd7201006a0 2024-04-22T00:00:01.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd71e59c640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd71000d770 tx=0x7fd71000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710017070 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd710009c30 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7201a1b60 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710003dd0 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.369+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7201a20b0 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.370+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7201079c0 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.371+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd710003fb0 con 0x7fd7201006a0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.371+0000 7fd70f7fe640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.372+0000 7fd71dd9b640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:01.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.372+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd7100cec30 con 0x7fd7201006a0 2024-04-22T00:00:01.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.372+0000 7fd71dd9b640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 0x7fd7080a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd72010ad90 tx=0x7fd71400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:01.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.376+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd71009b950 con 0x7fd7201006a0 2024-04-22T00:00:01.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:01.601+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fd72002d170 con 0x7fd7201006a0 2024-04-22T00:00:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T00:00:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T00:00:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T00:00:01.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T00:00:01.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T00:00:01.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T00:00:01.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T00:00:01.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:01 smithi038 ceph-mon[27972]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T00:00:01.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T00:00:01.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:01 smithi203 ceph-mon[31431]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T00:00:01.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/562925722' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2843325053' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T00:00:01.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:01 smithi111 ceph-mon[31543]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T00:00:02.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.238+0000 7fd70f7fe640 1 -- 172.21.15.38:0/3741468622 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7fd710009de0 con 0x7fd7201006a0 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 msgr2=0x7fd7080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 0x7fd7080a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd72010ad90 tx=0x7fd71400b390 comp rx=0 tx=0).stop 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 msgr2=0x7fd72010b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd71000d770 tx=0x7fd71000dc30 comp rx=0 tx=0).stop 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 shutdown_connections 2024-04-22T00:00:02.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd720102250 0x7fd720109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd720101890 0x7fd720109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 --2- 172.21.15.38:0/3741468622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7201006a0 0x7fd72010b020 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 >> 172.21.15.38:0/3741468622 conn(0x7fd7200fbc30 msgr2=0x7fd720104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 shutdown_connections 2024-04-22T00:00:02.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.241+0000 7fd724bb1640 1 -- 172.21.15.38:0/3741468622 wait complete. 2024-04-22T00:00:02.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.371+0000 7fd9cdcae640 1 -- 172.21.15.38:0/472901791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c81006a0 msgr2=0x7fd9c8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.371+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/472901791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c81006a0 0x7fd9c8100ac0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009900 tx=0x7fd9b802bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 -- 172.21.15.38:0/472901791 shutdown_connections 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/472901791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/472901791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c8101890 0x7fd9c8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/472901791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c81006a0 0x7fd9c8100ac0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 -- 172.21.15.38:0/472901791 >> 172.21.15.38:0/472901791 conn(0x7fd9c80fbc30 msgr2=0x7fd9c80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 -- 172.21.15.38:0/472901791 shutdown_connections 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 -- 172.21.15.38:0/472901791 wait complete. 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 Processor -- start 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.372+0000 7fd9cdcae640 1 -- start start 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 0x7fd9c806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810c320 con 0x7fd9c8101890 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810c1a0 con 0x7fd9c81006a0 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810c4a0 con 0x7fd9c8102250 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57706/0 (socket says 172.21.15.38:57706) 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 0x7fd9c806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:02.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 -- 172.21.15.38:0/3635265806 learned_addr learned my addr 172.21.15.38:0/3635265806 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c7fff640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 -- 172.21.15.38:0/3635265806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 msgr2=0x7fd9c819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 -- 172.21.15.38:0/3635265806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 msgr2=0x7fd9c806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 0x7fd9c806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 -- 172.21.15.38:0/3635265806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c819f740 con 0x7fd9c8101890 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c7fff640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c77fe640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 0x7fd9c806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c6ffd640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00ea60 tx=0x7fd9bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc00cde0 con 0x7fd9c8101890 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c819f9d0 con 0x7fd9c8101890 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc0035f0 con 0x7fd9c8101890 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc010670 con 0x7fd9c8101890 2024-04-22T00:00:02.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.373+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c819ff70 con 0x7fd9c8101890 2024-04-22T00:00:02.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.374+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c81079c0 con 0x7fd9c8101890 2024-04-22T00:00:02.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.375+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd9bc0037a0 con 0x7fd9c8101890 2024-04-22T00:00:02.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.376+0000 7fd9c4ff9640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 0x7fd9b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:02.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.376+0000 7fd9c77fe640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 0x7fd9b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:02.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.376+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd9bc0cf640 con 0x7fd9c8101890 2024-04-22T00:00:02.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.377+0000 7fd9c77fe640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 0x7fd9b00a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8000c00 tx=0x7fd9b8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:02.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.381+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd9bc09c300 con 0x7fd9c8101890 2024-04-22T00:00:02.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:02.607+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd9c8062340 con 0x7fd9c8101890 2024-04-22T00:00:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T00:00:02.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T00:00:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:active 2024-04-22T00:00:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T00:00:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T00:00:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T00:00:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:02 smithi038 ceph-mon[27972]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:active 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T00:00:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T00:00:02.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:02 smithi203 ceph-mon[31431]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T00:00:02.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T00:00:02.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3741468622' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T00:00:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] up:active 2024-04-22T00:00:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T00:00:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T00:00:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-22T00:00:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:02 smithi111 ceph-mon[31543]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-22T00:00:03.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.245+0000 7fd9c4ff9640 1 -- 172.21.15.38:0/3635265806 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fd9bc09bc00 con 0x7fd9c8101890 2024-04-22T00:00:03.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 msgr2=0x7fd9b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:03.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 0x7fd9b00a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8000c00 tx=0x7fd9b8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 msgr2=0x7fd9c819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:03.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00ea60 tx=0x7fd9bc00ef20 comp rx=0 tx=0).stop 2024-04-22T00:00:03.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 shutdown_connections 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9b00a2b90 0x7fd9b00a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9c8102250 0x7fd9c819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9c8101890 0x7fd9c819c810 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 --2- 172.21.15.38:0/3635265806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9c81006a0 0x7fd9c806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 >> 172.21.15.38:0/3635265806 conn(0x7fd9c80fbc30 msgr2=0x7fd9c8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 shutdown_connections 2024-04-22T00:00:03.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.248+0000 7fd9cdcae640 1 -- 172.21.15.38:0/3635265806 wait complete. 2024-04-22T00:00:03.258 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T00:00:03.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T00:00:03.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 -- 172.21.15.38:0/2116869497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 msgr2=0x7fbfb40ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/2116869497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb40ffe70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009930 tx=0x7fbfa402bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 -- 172.21.15.38:0/2116869497 shutdown_connections 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/2116869497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 0x7fbfb4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/2116869497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 0x7fbfb4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/2116869497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb40ffe70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 -- 172.21.15.38:0/2116869497 >> 172.21.15.38:0/2116869497 conn(0x7fbfb40fb380 msgr2=0x7fbfb40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 -- 172.21.15.38:0/2116869497 shutdown_connections 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.381+0000 7fbfb9e74640 1 -- 172.21.15.38:0/2116869497 wait complete. 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 Processor -- start 2024-04-22T00:00:03.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 -- start start 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 0x7fbfb41009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 0x7fbfb4100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb410d4d0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb410d350 con 0x7fbfb4102a20 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb9e74640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb410d650 con 0x7fbfb41033e0 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57714/0 (socket says 172.21.15.38:57714) 2024-04-22T00:00:03.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 -- 172.21.15.38:0/902022421 learned_addr learned my addr 172.21.15.38:0/902022421 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfaaffd640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 0x7fbfb41009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb3fff640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 0x7fbfb4100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 -- 172.21.15.38:0/902022421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 msgr2=0x7fbfb4100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 0x7fbfb4100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 -- 172.21.15.38:0/902022421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 msgr2=0x7fbfb41009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 0x7fbfb41009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.382+0000 7fbfb37fe640 1 -- 172.21.15.38:0/902022421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb41017a0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb37fe640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009900 tx=0x7fbfa402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4043020 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfa40054e0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb41a1c90 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4003cb0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.383+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb41a21b0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.384+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb4108ba0 con 0x7fbfb40ffa50 2024-04-22T00:00:03.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.385+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbfa4046020 con 0x7fbfb40ffa50 2024-04-22T00:00:03.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.386+0000 7fbfb17fa640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 0x7fbfa00a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:03.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.386+0000 7fbfaaffd640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 0x7fbfa00a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.386+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbfa40f1460 con 0x7fbfb40ffa50 2024-04-22T00:00:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.387+0000 7fbfaaffd640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 0x7fbfa00a5150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4102100 tx=0x7fbf9c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.391+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbfa404d020 con 0x7fbfb40ffa50 2024-04-22T00:00:03.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:03.602+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbfb4061b80 con 0x7fbfa00a2c90 2024-04-22T00:00:03.600 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:active 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-22T00:00:03.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-mon[27972]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T00:00:03.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:active 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-22T00:00:03.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:03 smithi203 ceph-mon[31431]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T00:00:03.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-22T00:00:03.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3635265806' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] up:active 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-22T00:00:03.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:03 smithi111 ceph-mon[31543]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T00:00:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:00:03.606+0000 7f1ad7f7d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:00:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:00:03.606+0000 7f1ad7f7d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:00:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:00:03.606+0000 7f1ad7f7d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:00:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:00:03.606+0000 7f1ad7f7d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:00:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:00:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:00:03.606+0000 7f1ad7f7d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:00:04.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.277+0000 7fbfb17fa640 1 -- 172.21.15.38:0/902022421 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbfb4061b80 con 0x7fbfa00a2c90 2024-04-22T00:00:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 msgr2=0x7fbfa00a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 0x7fbfa00a5150 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4102100 tx=0x7fbf9c005f90 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 msgr2=0x7fbfb4102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009900 tx=0x7fbfa402c6c0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 shutdown_connections 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfa00a2c90 0x7fbfa00a5150 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfb41033e0 0x7fbfb4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfb4102a20 0x7fbfb41009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 --2- 172.21.15.38:0/902022421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfb40ffa50 0x7fbfb4102390 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.279+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 >> 172.21.15.38:0/902022421 conn(0x7fbfb40fb380 msgr2=0x7fbfb4105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.280+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 shutdown_connections 2024-04-22T00:00:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.280+0000 7fbfb9e74640 1 -- 172.21.15.38:0/902022421 wait complete. 2024-04-22T00:00:04.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T00:00:04.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- 172.21.15.38:0/3891498991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 msgr2=0x7fd8580ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- 172.21.15.38:0/3891498991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd8580ffe90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd854009930 tx=0x7fd85402bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- 172.21.15.38:0/3891498991 shutdown_connections 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- 172.21.15.38:0/3891498991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8581034a0 0x7fd858109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- 172.21.15.38:0/3891498991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd858102ae0 0x7fd858102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- 172.21.15.38:0/3891498991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd8580ffe90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- 172.21.15.38:0/3891498991 >> 172.21.15.38:0/3891498991 conn(0x7fd8580fb380 msgr2=0x7fd8580fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- 172.21.15.38:0/3891498991 shutdown_connections 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- 172.21.15.38:0/3891498991 wait complete. 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 Processor -- start 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 -- start start 2024-04-22T00:00:04.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.406+0000 7fd86050c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd858102ae0 0x7fd858198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8581034a0 0x7fd85819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd85810d420 con 0x7fd8580ffa70 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd85810d2a0 con 0x7fd858102ae0 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd85810d5a0 con 0x7fd8581034a0 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40492/0 (socket says 172.21.15.38:40492) 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 -- 172.21.15.38:0/4018713144 learned_addr learned my addr 172.21.15.38:0/4018713144 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85daa7640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd858102ae0 0x7fd858198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85eaa9640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8581034a0 0x7fd85819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 -- 172.21.15.38:0/4018713144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8581034a0 msgr2=0x7fd85819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8581034a0 0x7fd85819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 -- 172.21.15.38:0/4018713144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd858102ae0 msgr2=0x7fd858198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd858102ae0 0x7fd858198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 -- 172.21.15.38:0/4018713144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85819f740 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd85e2a8640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd85402c7b0 tx=0x7fd85402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854041070 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd854004450 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85819f9d0 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854034440 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.407+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85819ff10 con 0x7fd8580ffa70 2024-04-22T00:00:04.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.409+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd858108c60 con 0x7fd8580ffa70 2024-04-22T00:00:04.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.410+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd854044070 con 0x7fd8580ffa70 2024-04-22T00:00:04.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.411+0000 7fd84f7fe640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 0x7fd8440a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.411+0000 7fd85daa7640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 0x7fd8440a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.411+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd8540f0de0 con 0x7fd8580ffa70 2024-04-22T00:00:04.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.411+0000 7fd85daa7640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 0x7fd8440a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd858199030 tx=0x7fd84800a250 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.416+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd8540bd4b0 con 0x7fd8580ffa70 2024-04-22T00:00:04.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.636+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd858061c10 con 0x7fd8440a2c10 2024-04-22T00:00:04.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.662+0000 7fd84f7fe640 1 -- 172.21.15.38:0/4018713144 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd858061c10 con 0x7fd8440a2c10 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 msgr2=0x7fd8440a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 0x7fd8440a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd858199030 tx=0x7fd84800a250 comp rx=0 tx=0).stop 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 msgr2=0x7fd858197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd85402c7b0 tx=0x7fd85402c6c0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 shutdown_connections 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd8440a2c10 0x7fd8440a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8581034a0 0x7fd85819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd858102ae0 0x7fd858198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 --2- 172.21.15.38:0/4018713144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8580ffa70 0x7fd858197b20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 >> 172.21.15.38:0/4018713144 conn(0x7fd8580fb380 msgr2=0x7fd858105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:04.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 shutdown_connections 2024-04-22T00:00:04.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.664+0000 7fd86050c640 1 -- 172.21.15.38:0/4018713144 wait complete. 2024-04-22T00:00:04.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T00:00:04.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 -- 172.21.15.38:0/3144758279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc102250 msgr2=0x7f92bc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 --2- 172.21.15.38:0/3144758279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc102250 0x7f92bc108ae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009ca0 tx=0x7f92b402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 -- 172.21.15.38:0/3144758279 shutdown_connections 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 --2- 172.21.15.38:0/3144758279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc102250 0x7f92bc108ae0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 --2- 172.21.15.38:0/3144758279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc101890 0x7f92bc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 --2- 172.21.15.38:0/3144758279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc1006a0 0x7f92bc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 -- 172.21.15.38:0/3144758279 >> 172.21.15.38:0/3144758279 conn(0x7f92bc0fbc30 msgr2=0x7f92bc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 -- 172.21.15.38:0/3144758279 shutdown_connections 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.782+0000 7f92c29de640 1 -- 172.21.15.38:0/3144758279 wait complete. 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 Processor -- start 2024-04-22T00:00:04.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 -- start start 2024-04-22T00:00:04.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc101890 0x7f92bc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 0x7f92bc19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc10c320 con 0x7f92bc1006a0 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc10c1a0 con 0x7f92bc101890 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92c29de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92bc10c4a0 con 0x7f92bc102250 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92bbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92bbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40518/0 (socket says 172.21.15.38:40518) 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.783+0000 7f92bbfff640 1 -- 172.21.15.38:0/2877748217 learned_addr learned my addr 172.21.15.38:0/2877748217 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bb7fe640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc101890 0x7f92bc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92c0f7b640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 0x7f92bc19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 -- 172.21.15.38:0/2877748217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 msgr2=0x7f92bc19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 0x7f92bc19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 -- 172.21.15.38:0/2877748217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc101890 msgr2=0x7f92bc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:04.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc101890 0x7f92bc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 -- 172.21.15.38:0/2877748217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc19f6a0 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92c0f7b640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 0x7f92bc19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92bbfff640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e9c0 tx=0x7f92ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00cd80 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac0035f0 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac010690 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92bc19f990 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.784+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92bc19ff30 con 0x7f92bc1006a0 2024-04-22T00:00:04.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.785+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92bc1a01c0 con 0x7f92bc1006a0 2024-04-22T00:00:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.786+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f92ac0037a0 con 0x7f92bc1006a0 2024-04-22T00:00:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.787+0000 7f92b97fa640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 0x7f92a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.787+0000 7f92bb7fe640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 0x7f92a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.787+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f92ac0cf720 con 0x7f92bc1006a0 2024-04-22T00:00:04.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.788+0000 7f92bb7fe640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 0x7f92a40a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f92bc198f90 tx=0x7f92b000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:04.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:04.791+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f92ac09c4d0 con 0x7f92bc1006a0 2024-04-22T00:00:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.007+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f92bc0fddb0 con 0x7f92a40a2c10 2024-04-22T00:00:05.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.013+0000 7f92b97fa640 1 -- 172.21.15.38:0/2877748217 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f92bc0fddb0 con 0x7f92a40a2c10 2024-04-22T00:00:05.006 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_0/01fab3da-e306-4a84-a649-c880500b3e4c 2024-04-22T00:00:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 msgr2=0x7f92a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 0x7f92a40a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f92bc198f90 tx=0x7f92b000b390 comp rx=0 tx=0).stop 2024-04-22T00:00:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 msgr2=0x7f92bc197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e9c0 tx=0x7f92ac00ee80 comp rx=0 tx=0).stop 2024-04-22T00:00:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 shutdown_connections 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f92a40a2c10 0x7f92a40a50d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92bc102250 0x7f92bc19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92bc101890 0x7f92bc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 --2- 172.21.15.38:0/2877748217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92bc1006a0 0x7f92bc197a80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.015+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 >> 172.21.15.38:0/2877748217 conn(0x7f92bc0fbc30 msgr2=0x7f92bc0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.016+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 shutdown_connections 2024-04-22T00:00:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.016+0000 7f92c29de640 1 -- 172.21.15.38:0/2877748217 wait complete. 2024-04-22T00:00:05.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 -- 172.21.15.38:0/241046781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12001074e0 msgr2=0x7f12001098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 --2- 172.21.15.38:0/241046781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12001074e0 0x7f12001098d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009ca0 tx=0x7f11f802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 -- 172.21.15.38:0/241046781 shutdown_connections 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 --2- 172.21.15.38:0/241046781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12001074e0 0x7f12001098d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 --2- 172.21.15.38:0/241046781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200104bb0 0x7f1200106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 --2- 172.21.15.38:0/241046781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1200102280 0x7f1200104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 -- 172.21.15.38:0/241046781 >> 172.21.15.38:0/241046781 conn(0x7f12000fbc10 msgr2=0x7f12000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 -- 172.21.15.38:0/241046781 shutdown_connections 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.133+0000 7f1205be5640 1 -- 172.21.15.38:0/241046781 wait complete. 2024-04-22T00:00:05.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 Processor -- start 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 -- start start 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200102280 0x7f1200197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 0x7f120019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d440 con 0x7f1200104bb0 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d2c0 con 0x7f1200102280 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f1205be5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f120010d5c0 con 0x7f12001074e0 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40538/0 (socket says 172.21.15.38:40538) 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 -- 172.21.15.38:0/1763369082 learned_addr learned my addr 172.21.15.38:0/1763369082 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11ff7fe640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200102280 0x7f1200197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11fffff640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 0x7f120019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 -- 172.21.15.38:0/1763369082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 msgr2=0x7f120019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 0x7f120019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 -- 172.21.15.38:0/1763369082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200102280 msgr2=0x7f1200197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200102280 0x7f1200197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11feffd640 1 -- 172.21.15.38:0/1763369082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120019f6a0 con 0x7f1200104bb0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.134+0000 7f11fffff640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 0x7f120019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f11feffd640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9a0 tx=0x7f11f400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f400cd80 con 0x7f1200104bb0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11f40035f0 con 0x7f1200104bb0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120019f990 con 0x7f1200104bb0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4010670 con 0x7f1200104bb0 2024-04-22T00:00:05.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.135+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120019fee0 con 0x7f1200104bb0 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.136+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12000fd700 con 0x7f1200104bb0 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.137+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f11f40037a0 con 0x7f1200104bb0 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.138+0000 7f11fcff9640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 0x7f11e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.138+0000 7f11ff7fe640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 0x7f11e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.138+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11f40cf5b0 con 0x7f1200104bb0 2024-04-22T00:00:05.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.138+0000 7f11ff7fe640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 0x7f11e80a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005fd0 tx=0x7f11f000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.142+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f11f409c2d0 con 0x7f1200104bb0 2024-04-22T00:00:05.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.359+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1200061b80 con 0x7f11e80a2b90 2024-04-22T00:00:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.388+0000 7f11fcff9640 1 -- 172.21.15.38:0/1763369082 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1200061b80 con 0x7f11e80a2b90 2024-04-22T00:00:05.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 msgr2=0x7f11e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 0x7f11e80a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005fd0 tx=0x7f11f000b390 comp rx=0 tx=0).stop 2024-04-22T00:00:05.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 msgr2=0x7f1200197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9a0 tx=0x7f11f400ee60 comp rx=0 tx=0).stop 2024-04-22T00:00:05.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 shutdown_connections 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f11e80a2b90 0x7f11e80a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12001074e0 0x7f120019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1200104bb0 0x7f1200197fc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 --2- 172.21.15.38:0/1763369082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1200102280 0x7f1200197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 >> 172.21.15.38:0/1763369082 conn(0x7f12000fbc10 msgr2=0x7f1200102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 shutdown_connections 2024-04-22T00:00:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.390+0000 7f1205be5640 1 -- 172.21.15.38:0/1763369082 wait complete. 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:active 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: Cluster is now healthy 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T00:00:05.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:05 smithi038 ceph-mon[27972]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-22T00:00:05.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T00:00:05.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.508+0000 7f8a80f72640 1 -- 172.21.15.38:0/2252236633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c100680 msgr2=0x7f8a7c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.508+0000 7f8a80f72640 1 --2- 172.21.15.38:0/2252236633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c100680 0x7f8a7c100aa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009ca0 tx=0x7f8a6c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 -- 172.21.15.38:0/2252236633 shutdown_connections 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 --2- 172.21.15.38:0/2252236633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c102230 0x7f8a7c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 --2- 172.21.15.38:0/2252236633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 --2- 172.21.15.38:0/2252236633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c100680 0x7f8a7c100aa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 -- 172.21.15.38:0/2252236633 >> 172.21.15.38:0/2252236633 conn(0x7f8a7c0fbbf0 msgr2=0x7f8a7c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 -- 172.21.15.38:0/2252236633 shutdown_connections 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 -- 172.21.15.38:0/2252236633 wait complete. 2024-04-22T00:00:05.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 Processor -- start 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.509+0000 7f8a80f72640 1 -- start start 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c100680 0x7f8a7c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c10d420 con 0x7f8a7c102230 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c10d2a0 con 0x7f8a7c101870 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c10d5a0 con 0x7f8a7c100680 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a7ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a7ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40546/0 (socket says 172.21.15.38:40546) 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a7ad9d640 1 -- 172.21.15.38:0/389486067 learned_addr learned my addr 172.21.15.38:0/389486067 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a7a59c640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c100680 0x7f8a7c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 -- 172.21.15.38:0/389486067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c100680 msgr2=0x7f8a7c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c100680 0x7f8a7c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 -- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 msgr2=0x7f8a7c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 -- 172.21.15.38:0/389486067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c19f730 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a7ad9d640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a79d9b640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c198050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9f0 tx=0x7f8a6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400cd80 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7c19fa20 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a640035f0 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64010670 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.510+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7c19ff70 con 0x7f8a7c101870 2024-04-22T00:00:05.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.511+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a7c107ac0 con 0x7f8a7c101870 2024-04-22T00:00:05.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.513+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8a640037a0 con 0x7f8a7c101870 2024-04-22T00:00:05.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.513+0000 7f8a737fe640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 0x7f8a680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.513+0000 7f8a7a59c640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 0x7f8a680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.513+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8a640cf4c0 con 0x7f8a7c101870 2024-04-22T00:00:05.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.514+0000 7f8a7a59c640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 0x7f8a680a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c02c700 tx=0x7f8a6c032590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.516+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8a6409c1e0 con 0x7f8a7c101870 2024-04-22T00:00:05.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:active 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: Cluster is now healthy 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T00:00:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:05 smithi203 ceph-mon[31431]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-22T00:00:05.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:05.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] up:active 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: Cluster is now healthy 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-22T00:00:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:05 smithi111 ceph-mon[31543]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-22T00:00:05.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.730+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8a7c061b80 con 0x7f8a680a2b90 2024-04-22T00:00:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.737+0000 7f8a737fe640 1 -- 172.21.15.38:0/389486067 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8a7c061b80 con 0x7f8a680a2b90 2024-04-22T00:00:05.731 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d 2024-04-22T00:00:05.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 msgr2=0x7f8a680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 0x7f8a680a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c02c700 tx=0x7f8a6c032590 comp rx=0 tx=0).stop 2024-04-22T00:00:05.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 msgr2=0x7f8a7c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c198050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9f0 tx=0x7f8a6400eeb0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 shutdown_connections 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f8a680a2b90 0x7f8a680a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a7c102230 0x7f8a7c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a7c101870 0x7f8a7c198050 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 --2- 172.21.15.38:0/389486067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a7c100680 0x7f8a7c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 >> 172.21.15.38:0/389486067 conn(0x7f8a7c0fbbf0 msgr2=0x7f8a7c104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 shutdown_connections 2024-04-22T00:00:05.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.739+0000 7f8a80f72640 1 -- 172.21.15.38:0/389486067 wait complete. 2024-04-22T00:00:05.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 -- 172.21.15.38:0/1830025848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720101880 msgr2=0x7f8720101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 --2- 172.21.15.38:0/1830025848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720101880 0x7f8720101d00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8708009900 tx=0x7f870802bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 -- 172.21.15.38:0/1830025848 shutdown_connections 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 --2- 172.21.15.38:0/1830025848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720102240 0x7f8720108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 --2- 172.21.15.38:0/1830025848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720101880 0x7f8720101d00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 --2- 172.21.15.38:0/1830025848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720100690 0x7f8720100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 -- 172.21.15.38:0/1830025848 >> 172.21.15.38:0/1830025848 conn(0x7f87200fbc20 msgr2=0x7f87200fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 -- 172.21.15.38:0/1830025848 shutdown_connections 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.852+0000 7f87250d4640 1 -- 172.21.15.38:0/1830025848 wait complete. 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 Processor -- start 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 -- start start 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720101880 0x7f8720198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720102240 0x7f872019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f872010c360 con 0x7f8720100690 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f872010c1e0 con 0x7f8720101880 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f87250d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f872010c4e0 con 0x7f8720102240 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f871ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f871ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40552/0 (socket says 172.21.15.38:40552) 2024-04-22T00:00:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f871ed9d640 1 -- 172.21.15.38:0/3655987076 learned_addr learned my addr 172.21.15.38:0/3655987076 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f8717fff640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720101880 0x7f8720198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.853+0000 7f871f59e640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720102240 0x7f872019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 -- 172.21.15.38:0/3655987076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720102240 msgr2=0x7f872019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720102240 0x7f872019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 -- 172.21.15.38:0/3655987076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720101880 msgr2=0x7f8720198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720101880 0x7f8720198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 -- 172.21.15.38:0/3655987076 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872019f710 con 0x7f8720100690 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871ed9d640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f871000d950 tx=0x7f871000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710017070 con 0x7f8720100690 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8710009c30 con 0x7f8720100690 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872019fa00 con 0x7f8720100690 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871000f600 con 0x7f8720100690 2024-04-22T00:00:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.854+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872019fed0 con 0x7f8720100690 2024-04-22T00:00:05.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.855+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87201a0100 con 0x7f8720100690 2024-04-22T00:00:05.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.856+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f871001a070 con 0x7f8720100690 2024-04-22T00:00:05.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.857+0000 7f871cd99640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 0x7f870c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.857+0000 7f8717fff640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 0x7f870c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.857+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87100cecc0 con 0x7f8720100690 2024-04-22T00:00:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.857+0000 7f8717fff640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 0x7f870c0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f870802c240 tx=0x7f870802df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:05.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:05.861+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f87100d5080 con 0x7f8720100690 2024-04-22T00:00:06.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.084+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87201a0490 con 0x7f8720100690 2024-04-22T00:00:06.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.085+0000 7f871cd99640 1 -- 172.21.15.38:0/3655987076 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f8710009de0 con 0x7f8720100690 2024-04-22T00:00:06.078 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:06.078 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:00:06.078 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.086+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 msgr2=0x7f870c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.086+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 0x7f870c0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f870802c240 tx=0x7f870802df90 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.086+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 msgr2=0x7f8720197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.086+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f871000d950 tx=0x7f871000de10 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 shutdown_connections 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f870c0a2b90 0x7f870c0a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8720102240 0x7f872019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8720101880 0x7f8720198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 --2- 172.21.15.38:0/3655987076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8720100690 0x7f8720197af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 >> 172.21.15.38:0/3655987076 conn(0x7f87200fbc20 msgr2=0x7f87200fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 shutdown_connections 2024-04-22T00:00:06.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.087+0000 7f87250d4640 1 -- 172.21.15.38:0/3655987076 wait complete. 2024-04-22T00:00:06.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:00:06.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- 172.21.15.38:0/3408781667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964102230 msgr2=0x7fd964108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- 172.21.15.38:0/3408781667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964102230 0x7fd964108ac0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009ca0 tx=0x7fd95c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- 172.21.15.38:0/3408781667 shutdown_connections 2024-04-22T00:00:06.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- 172.21.15.38:0/3408781667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964102230 0x7fd964108ac0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- 172.21.15.38:0/3408781667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964101870 0x7fd964101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- 172.21.15.38:0/3408781667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964100680 0x7fd964100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- 172.21.15.38:0/3408781667 >> 172.21.15.38:0/3408781667 conn(0x7fd9640fbbf0 msgr2=0x7fd9640fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- 172.21.15.38:0/3408781667 shutdown_connections 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- 172.21.15.38:0/3408781667 wait complete. 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 Processor -- start 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- start start 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964100680 0x7fd964197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 0x7fd96419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410c300 con 0x7fd964101870 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410c180 con 0x7fd964102230 2024-04-22T00:00:06.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.200+0000 7fd96b15c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410c480 con 0x7fd964100680 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40564/0 (socket says 172.21.15.38:40564) 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 -- 172.21.15.38:0/1639896207 learned_addr learned my addr 172.21.15.38:0/1639896207 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd9696f9640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 0x7fd96419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 -- 172.21.15.38:0/1639896207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964100680 msgr2=0x7fd964197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd968ef8640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964100680 0x7fd964197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964100680 0x7fd964197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 -- 172.21.15.38:0/1639896207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 msgr2=0x7fd96419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 0x7fd96419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 -- 172.21.15.38:0/1639896207 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96419f6b0 con 0x7fd964101870 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd9696f9640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 0x7fd96419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd963fff640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd95000c970 tx=0x7fd95000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950007bb0 con 0x7fd964101870 2024-04-22T00:00:06.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd950007d50 con 0x7fd964101870 2024-04-22T00:00:06.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96419f9a0 con 0x7fd964101870 2024-04-22T00:00:06.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95000f450 con 0x7fd964101870 2024-04-22T00:00:06.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.201+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96419fea0 con 0x7fd964101870 2024-04-22T00:00:06.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.202+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9641a0130 con 0x7fd964101870 2024-04-22T00:00:06.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.203+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd950005020 con 0x7fd964101870 2024-04-22T00:00:06.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.204+0000 7fd961ffb640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 0x7fd9540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.204+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd9500cf070 con 0x7fd964101870 2024-04-22T00:00:06.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.204+0000 7fd968ef8640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 0x7fd9540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.204+0000 7fd968ef8640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 0x7fd9540a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9641016d0 tx=0x7fd958005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:06.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.208+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd950014030 con 0x7fd964101870 2024-04-22T00:00:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.293+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd950097d00 con 0x7fd964101870 2024-04-22T00:00:06.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.402+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9641a0300 con 0x7fd964101870 2024-04-22T00:00:06.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.404+0000 7fd961ffb640 1 -- 172.21.15.38:0/1639896207 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fd95009bd90 con 0x7fd964101870 2024-04-22T00:00:06.397 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:06.397 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:00:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 msgr2=0x7fd9540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 0x7fd9540a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9641016d0 tx=0x7fd958005f90 comp rx=0 tx=0).stop 2024-04-22T00:00:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 msgr2=0x7fd964197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd95000c970 tx=0x7fd95000ce30 comp rx=0 tx=0).stop 2024-04-22T00:00:06.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 shutdown_connections 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd9540a2b90 0x7fd9540a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd964102230 0x7fd96419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd964101870 0x7fd964197fd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 --2- 172.21.15.38:0/1639896207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd964100680 0x7fd964197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 >> 172.21.15.38:0/1639896207 conn(0x7fd9640fbbf0 msgr2=0x7fd9640fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 shutdown_connections 2024-04-22T00:00:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.406+0000 7fd96b15c640 1 -- 172.21.15.38:0/1639896207 wait complete. 2024-04-22T00:00:06.409 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.409 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.409 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:active 2024-04-22T00:00:06.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-22T00:00:06.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:00:06.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:06 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3655987076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:00:06.411 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T00:00:06.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T00:00:06.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 -- 172.21.15.38:0/1513150640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8102980 msgr2=0x7f16d8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 --2- 172.21.15.38:0/1513150640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8102980 0x7f16d8102e00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009900 tx=0x7f16c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 -- 172.21.15.38:0/1513150640 shutdown_connections 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 --2- 172.21.15.38:0/1513150640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 --2- 172.21.15.38:0/1513150640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8102980 0x7f16d8102e00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 --2- 172.21.15.38:0/1513150640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8101790 0x7f16d8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 -- 172.21.15.38:0/1513150640 >> 172.21.15.38:0/1513150640 conn(0x7f16d80fcd20 msgr2=0x7f16d80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 -- 172.21.15.38:0/1513150640 shutdown_connections 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.522+0000 7f16df678640 1 -- 172.21.15.38:0/1513150640 wait complete. 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 Processor -- start 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 -- start start 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 0x7f16d8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16d810d3f0 con 0x7f16d8101790 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16d810d270 con 0x7f16d8102980 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16df678640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16d810d570 con 0x7f16d8103340 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40590/0 (socket says 172.21.15.38:40590) 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 -- 172.21.15.38:0/1282948851 learned_addr learned my addr 172.21.15.38:0/1282948851 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dcc13640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 0x7f16d8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16ddc15640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 -- 172.21.15.38:0/1282948851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 msgr2=0x7f16d819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 -- 172.21.15.38:0/1282948851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 msgr2=0x7f16d8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 0x7f16d8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 -- 172.21.15.38:0/1282948851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d819f660 con 0x7f16d8101790 2024-04-22T00:00:06.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dcc13640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 0x7f16d8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16ddc15640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.523+0000 7f16dd414640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d950 tx=0x7f16d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.524+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4017020 con 0x7f16d8101790 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.524+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d819f950 con 0x7f16d8101790 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.524+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16d4009c30 con 0x7f16d8101790 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.524+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d400f6b0 con 0x7f16d8101790 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.524+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d819fef0 con 0x7f16d8101790 2024-04-22T00:00:06.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.525+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d80689b0 con 0x7f16d8101790 2024-04-22T00:00:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.526+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f16d401a020 con 0x7f16d8101790 2024-04-22T00:00:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.526+0000 7f16ce7fc640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 0x7f16c40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.526+0000 7f16dcc13640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 0x7f16c40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.526+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f16d40cf530 con 0x7f16d8101790 2024-04-22T00:00:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.527+0000 7f16dcc13640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 0x7f16c40a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f16c8036040 tx=0x7f16c8000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:06.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.531+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f16d4021020 con 0x7f16d8101790 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:active 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:06 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3655987076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:00:06.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] up:active 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:06.538 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3655987076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:00:06.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.713+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f16d8061b80 con 0x7f16c40a2c20 2024-04-22T00:00:06.919 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:00:06.545+0000 7f8b4e317700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T00:00:06.919 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:00:06.545+0000 7f8b4e317700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-22T00:00:06.919 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:00:06 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:00:06.545+0000 7f8b4e317700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-22T00:00:06.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.724+0000 7f16ce7fc640 1 -- 172.21.15.38:0/1282948851 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340315 (secure 0 0 0) 0x7f16d8061b80 con 0x7f16c40a2c20 2024-04-22T00:00:06.923 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 msgr2=0x7f16c40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 0x7f16c40a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f16c8036040 tx=0x7f16c8000bc0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 msgr2=0x7f16d8197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d950 tx=0x7f16d400de10 comp rx=0 tx=0).stop 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 shutdown_connections 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f16c40a2c20 0x7f16c40a50e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16d8103340 0x7f16d819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16d8102980 0x7f16d8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 --2- 172.21.15.38:0/1282948851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16d8101790 0x7f16d8197a40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 >> 172.21.15.38:0/1282948851 conn(0x7f16d80fcd20 msgr2=0x7f16d8105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 shutdown_connections 2024-04-22T00:00:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:06.727+0000 7f16df678640 1 -- 172.21.15.38:0/1282948851 wait complete. 2024-04-22T00:00:06.928 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-22T00:00:05.654491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598492,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":138,"num_read_kb":121,"num_write":315,"num_write_kb":2984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354372,"kb_used_data":13524,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587452,"statfs":{"total":1151940427776,"available":1151577550848,"internally_reserved":0,"allocated":13848576,"data_stored":7531578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":348894524},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3625,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":87,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896421+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-21T23:59:41.896421+0000","last_peered":"2024-04-21T23:59:41.896421+0000","last_clean":"2024-04-21T23:59:41.896421+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-21T23:59:41.896421+0000","last_undegraded":"2024-04-21T23:59:41.896421+0000","last_fullsized":"2024-04-21T23:59:41.896421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895638+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-21T23:59:41.895638+0000","last_peered":"2024-04-21T23:59:41.895638+0000","last_clean":"2024-04-21T23:59:41.895638+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-21T23:59:41.895638+0000","last_undegraded":"2024-04-21T23:59:41.895638+0000","last_fullsized":"2024-04-21T23:59:41.895638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895160+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-21T23:59:41.895160+0000","last_peered":"2024-04-21T23:59:41.895160+0000","last_clean":"2024-04-21T23:59:41.895160+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-21T23:59:41.895160+0000","last_undegraded":"2024-04-21T23:59:41.895160+0000","last_fullsized":"2024-04-21T23:59:41.895160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896475+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-21T23:59:41.896475+0000","last_peered":"2024-04-21T23:59:41.896475+0000","last_clean":"2024-04-21T23:59:41.896475+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-21T23:59:41.896475+0000","last_undegraded":"2024-04-21T23:59:41.896475+0000","last_fullsized":"2024-04-21T23:59:41.896475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895577+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-21T23:59:41.895577+0000","last_peered":"2024-04-21T23:59:41.895577+0000","last_clean":"2024-04-21T23:59:41.895577+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-21T23:59:41.895577+0000","last_undegraded":"2024-04-21T23:59:41.895577+0000","last_fullsized":"2024-04-21T23:59:41.895577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397644+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-21T23:59:41.397644+0000","last_peered":"2024-04-21T23:59:41.397644+0000","last_clean":"2024-04-21T23:59:41.397644+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-21T23:59:41.397644+0000","last_undegraded":"2024-04-21T23:59:41.397644+0000","last_fullsized":"2024-04-21T23:59:41.397644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896636+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-21T23:59:41.896636+0000","last_peered":"2024-04-21T23:59:41.896636+0000","last_clean":"2024-04-21T23:59:41.896636+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-21T23:59:41.896636+0000","last_undegraded":"2024-04-21T23:59:41.896636+0000","last_fullsized":"2024-04-21T23:59:41.896636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896479+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-21T23:59:41.896479+0000","last_peered":"2024-04-21T23:59:41.896479+0000","last_clean":"2024-04-21T23:59:41.896479+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-21T23:59:41.896479+0000","last_undegraded":"2024-04-21T23:59:41.896479+0000","last_fullsized":"2024-04-21T23:59:41.896479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896245+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-21T23:59:41.896245+0000","last_peered":"2024-04-21T23:59:41.896245+0000","last_clean":"2024-04-21T23:59:41.896245+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-21T23:59:41.896245+0000","last_undegraded":"2024-04-21T23:59:41.896245+0000","last_fullsized":"2024-04-21T23:59:41.896245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.398121+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-21T23:59:41.398121+0000","last_peered":"2024-04-21T23:59:41.398121+0000","last_clean":"2024-04-21T23:59:41.398121+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-21T23:59:41.398121+0000","last_undegraded":"2024-04-21T23:59:41.398121+0000","last_fullsized":"2024-04-21T23:59:41.398121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896336+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-21T23:59:41.896336+0000","last_peered":"2024-04-21T23:59:41.896336+0000","last_clean":"2024-04-21T23:59:41.896336+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-21T23:59:41.896336+0000","last_undegraded":"2024-04-21T23:59:41.896336+0000","last_fullsized":"2024-04-21T23:59:41.896336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896378+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-21T23:59:41.896378+0000","last_peered":"2024-04-21T23:59:41.896378+0000","last_clean":"2024-04-21T23:59:41.896378+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-21T23:59:41.896378+0000","last_undegraded":"2024-04-21T23:59:41.896378+0000","last_fullsized":"2024-04-21T23:59:41.896378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896379+0000","last_change":"2024-04-21T23:59:36.248992+0000","last_active":"2024-04-21T23:59:41.896379+0000","last_peered":"2024-04-21T23:59:41.896379+0000","last_clean":"2024-04-21T23:59:41.896379+0000","last_became_active":"2024-04-21T23:59:36.248652+0000","last_became_peered":"2024-04-21T23:59:36.248652+0000","last_unstale":"2024-04-21T23:59:41.896379+0000","last_undegraded":"2024-04-21T23:59:41.896379+0000","last_fullsized":"2024-04-21T23:59:41.896379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:30.301519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895152+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-21T23:59:41.895152+0000","last_peered":"2024-04-21T23:59:41.895152+0000","last_clean":"2024-04-21T23:59:41.895152+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-21T23:59:41.895152+0000","last_undegraded":"2024-04-21T23:59:41.895152+0000","last_fullsized":"2024-04-21T23:59:41.895152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896016+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-21T23:59:41.896016+0000","last_peered":"2024-04-21T23:59:41.896016+0000","last_clean":"2024-04-21T23:59:41.896016+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-21T23:59:41.896016+0000","last_undegraded":"2024-04-21T23:59:41.896016+0000","last_fullsized":"2024-04-21T23:59:41.896016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.894921+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-21T23:59:41.894921+0000","last_peered":"2024-04-21T23:59:41.894921+0000","last_clean":"2024-04-21T23:59:41.894921+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-21T23:59:41.894921+0000","last_undegraded":"2024-04-21T23:59:41.894921+0000","last_fullsized":"2024-04-21T23:59:41.894921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896254+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-21T23:59:41.896254+0000","last_peered":"2024-04-21T23:59:41.896254+0000","last_clean":"2024-04-21T23:59:41.896254+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-21T23:59:41.896254+0000","last_undegraded":"2024-04-21T23:59:41.896254+0000","last_fullsized":"2024-04-21T23:59:41.896254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.395153+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-21T23:59:41.395153+0000","last_peered":"2024-04-21T23:59:41.395153+0000","last_clean":"2024-04-21T23:59:41.395153+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-21T23:59:41.395153+0000","last_undegraded":"2024-04-21T23:59:41.395153+0000","last_fullsized":"2024-04-21T23:59:41.395153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394037+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-21T23:59:41.394037+0000","last_peered":"2024-04-21T23:59:41.394037+0000","last_clean":"2024-04-21T23:59:41.394037+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-21T23:59:41.394037+0000","last_undegraded":"2024-04-21T23:59:41.394037+0000","last_fullsized":"2024-04-21T23:59:41.394037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393688+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-21T23:59:41.393688+0000","last_peered":"2024-04-21T23:59:41.393688+0000","last_clean":"2024-04-21T23:59:41.393688+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-21T23:59:41.393688+0000","last_undegraded":"2024-04-21T23:59:41.393688+0000","last_fullsized":"2024-04-21T23:59:41.393688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:05.279528+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:00:05.279528+0000","last_peered":"2024-04-22T00:00:05.279528+0000","last_clean":"2024-04-22T00:00:05.279528+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:00:05.279528+0000","last_undegraded":"2024-04-22T00:00:05.279528+0000","last_fullsized":"2024-04-22T00:00:05.279528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895112+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-21T23:59:41.895112+0000","last_peered":"2024-04-21T23:59:41.895112+0000","last_clean":"2024-04-21T23:59:41.895112+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-21T23:59:41.895112+0000","last_undegraded":"2024-04-21T23:59:41.895112+0000","last_fullsized":"2024-04-21T23:59:41.895112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895043+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-21T23:59:41.895043+0000","last_peered":"2024-04-21T23:59:41.895043+0000","last_clean":"2024-04-21T23:59:41.895043+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-21T23:59:41.895043+0000","last_undegraded":"2024-04-21T23:59:41.895043+0000","last_fullsized":"2024-04-21T23:59:41.895043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896191+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-21T23:59:41.896191+0000","last_peered":"2024-04-21T23:59:41.896191+0000","last_clean":"2024-04-21T23:59:41.896191+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-21T23:59:41.896191+0000","last_undegraded":"2024-04-21T23:59:41.896191+0000","last_fullsized":"2024-04-21T23:59:41.896191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897309+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-21T23:59:41.897309+0000","last_peered":"2024-04-21T23:59:41.897309+0000","last_clean":"2024-04-21T23:59:41.897309+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-21T23:59:41.897309+0000","last_undegraded":"2024-04-21T23:59:41.897309+0000","last_fullsized":"2024-04-21T23:59:41.897309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895329+0000","last_change":"2024-04-21T23:59:40.414367+0000","last_active":"2024-04-21T23:59:41.895329+0000","last_peered":"2024-04-21T23:59:41.895329+0000","last_clean":"2024-04-21T23:59:41.895329+0000","last_became_active":"2024-04-21T23:59:40.413580+0000","last_became_peered":"2024-04-21T23:59:40.413580+0000","last_unstale":"2024-04-21T23:59:41.895329+0000","last_undegraded":"2024-04-21T23:59:41.895329+0000","last_fullsized":"2024-04-21T23:59:41.895329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:13.489706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.255235+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:00:02.255235+0000","last_peered":"2024-04-22T00:00:02.255235+0000","last_clean":"2024-04-22T00:00:02.255235+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:00:02.255235+0000","last_undegraded":"2024-04-22T00:00:02.255235+0000","last_fullsized":"2024-04-22T00:00:02.255235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896792+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-21T23:59:41.896792+0000","last_peered":"2024-04-21T23:59:41.896792+0000","last_clean":"2024-04-21T23:59:41.896792+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-21T23:59:41.896792+0000","last_undegraded":"2024-04-21T23:59:41.896792+0000","last_fullsized":"2024-04-21T23:59:41.896792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895148+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-21T23:59:41.895148+0000","last_peered":"2024-04-21T23:59:41.895148+0000","last_clean":"2024-04-21T23:59:41.895148+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-21T23:59:41.895148+0000","last_undegraded":"2024-04-21T23:59:41.895148+0000","last_fullsized":"2024-04-21T23:59:41.895148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895805+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-21T23:59:41.895805+0000","last_peered":"2024-04-21T23:59:41.895805+0000","last_clean":"2024-04-21T23:59:41.895805+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-21T23:59:41.895805+0000","last_undegraded":"2024-04-21T23:59:41.895805+0000","last_fullsized":"2024-04-21T23:59:41.895805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818772+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-21T23:59:41.818772+0000","last_peered":"2024-04-21T23:59:41.818772+0000","last_clean":"2024-04-21T23:59:41.818772+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-21T23:59:41.818772+0000","last_undegraded":"2024-04-21T23:59:41.818772+0000","last_fullsized":"2024-04-21T23:59:41.818772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393937+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-21T23:59:41.393937+0000","last_peered":"2024-04-21T23:59:41.393937+0000","last_clean":"2024-04-21T23:59:41.393937+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-21T23:59:41.393937+0000","last_undegraded":"2024-04-21T23:59:41.393937+0000","last_fullsized":"2024-04-21T23:59:41.393937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.248122+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:00:01.248122+0000","last_peered":"2024-04-22T00:00:01.248122+0000","last_clean":"2024-04-22T00:00:01.248122+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:00:01.248122+0000","last_undegraded":"2024-04-22T00:00:01.248122+0000","last_fullsized":"2024-04-22T00:00:01.248122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895671+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-21T23:59:41.895671+0000","last_peered":"2024-04-21T23:59:41.895671+0000","last_clean":"2024-04-21T23:59:41.895671+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-21T23:59:41.895671+0000","last_undegraded":"2024-04-21T23:59:41.895671+0000","last_fullsized":"2024-04-21T23:59:41.895671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393867+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-21T23:59:41.393867+0000","last_peered":"2024-04-21T23:59:41.393867+0000","last_clean":"2024-04-21T23:59:41.393867+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-21T23:59:41.393867+0000","last_undegraded":"2024-04-21T23:59:41.393867+0000","last_fullsized":"2024-04-21T23:59:41.393867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397631+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-21T23:59:41.397631+0000","last_peered":"2024-04-21T23:59:41.397631+0000","last_clean":"2024-04-21T23:59:41.397631+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-21T23:59:41.397631+0000","last_undegraded":"2024-04-21T23:59:41.397631+0000","last_fullsized":"2024-04-21T23:59:41.397631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818585+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-21T23:59:41.818585+0000","last_peered":"2024-04-21T23:59:41.818585+0000","last_clean":"2024-04-21T23:59:41.818585+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-21T23:59:41.818585+0000","last_undegraded":"2024-04-21T23:59:41.818585+0000","last_fullsized":"2024-04-21T23:59:41.818585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896343+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-21T23:59:41.896343+0000","last_peered":"2024-04-21T23:59:41.896343+0000","last_clean":"2024-04-21T23:59:41.896343+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-21T23:59:41.896343+0000","last_undegraded":"2024-04-21T23:59:41.896343+0000","last_fullsized":"2024-04-21T23:59:41.896343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895536+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-21T23:59:41.895536+0000","last_peered":"2024-04-21T23:59:41.895536+0000","last_clean":"2024-04-21T23:59:41.895536+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-21T23:59:41.895536+0000","last_undegraded":"2024-04-21T23:59:41.895536+0000","last_fullsized":"2024-04-21T23:59:41.895536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896376+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-21T23:59:41.896376+0000","last_peered":"2024-04-21T23:59:41.896376+0000","last_clean":"2024-04-21T23:59:41.896376+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-21T23:59:41.896376+0000","last_undegraded":"2024-04-21T23:59:41.896376+0000","last_fullsized":"2024-04-21T23:59:41.896376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895523+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-21T23:59:41.895523+0000","last_peered":"2024-04-21T23:59:41.895523+0000","last_clean":"2024-04-21T23:59:41.895523+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-21T23:59:41.895523+0000","last_undegraded":"2024-04-21T23:59:41.895523+0000","last_fullsized":"2024-04-21T23:59:41.895523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.261058+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:00:03.261058+0000","last_peered":"2024-04-22T00:00:03.261058+0000","last_clean":"2024-04-22T00:00:03.261058+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:00:03.261058+0000","last_undegraded":"2024-04-22T00:00:03.261058+0000","last_fullsized":"2024-04-22T00:00:03.261058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394550+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-21T23:59:41.394550+0000","last_peered":"2024-04-21T23:59:41.394550+0000","last_clean":"2024-04-21T23:59:41.394550+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-21T23:59:41.394550+0000","last_undegraded":"2024-04-21T23:59:41.394550+0000","last_fullsized":"2024-04-21T23:59:41.394550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896492+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-21T23:59:41.896492+0000","last_peered":"2024-04-21T23:59:41.896492+0000","last_clean":"2024-04-21T23:59:41.896492+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-21T23:59:41.896492+0000","last_undegraded":"2024-04-21T23:59:41.896492+0000","last_fullsized":"2024-04-21T23:59:41.896492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.246696+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:00:01.246696+0000","last_peered":"2024-04-22T00:00:01.246696+0000","last_clean":"2024-04-22T00:00:01.246696+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:00:01.246696+0000","last_undegraded":"2024-04-22T00:00:01.246696+0000","last_fullsized":"2024-04-22T00:00:01.246696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896574+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-21T23:59:41.896574+0000","last_peered":"2024-04-21T23:59:41.896574+0000","last_clean":"2024-04-21T23:59:41.896574+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-21T23:59:41.896574+0000","last_undegraded":"2024-04-21T23:59:41.896574+0000","last_fullsized":"2024-04-21T23:59:41.896574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394365+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-21T23:59:41.394365+0000","last_peered":"2024-04-21T23:59:41.394365+0000","last_clean":"2024-04-21T23:59:41.394365+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-21T23:59:41.394365+0000","last_undegraded":"2024-04-21T23:59:41.394365+0000","last_fullsized":"2024-04-21T23:59:41.394365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247136+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:00:01.247136+0000","last_peered":"2024-04-22T00:00:01.247136+0000","last_clean":"2024-04-22T00:00:01.247136+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:00:01.247136+0000","last_undegraded":"2024-04-22T00:00:01.247136+0000","last_fullsized":"2024-04-22T00:00:01.247136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897190+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-21T23:59:41.897190+0000","last_peered":"2024-04-21T23:59:41.897190+0000","last_clean":"2024-04-21T23:59:41.897190+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-21T23:59:41.897190+0000","last_undegraded":"2024-04-21T23:59:41.897190+0000","last_fullsized":"2024-04-21T23:59:41.897190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895825+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-21T23:59:41.895825+0000","last_peered":"2024-04-21T23:59:41.895825+0000","last_clean":"2024-04-21T23:59:41.895825+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-21T23:59:41.895825+0000","last_undegraded":"2024-04-21T23:59:41.895825+0000","last_fullsized":"2024-04-21T23:59:41.895825+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247311+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:00:01.247311+0000","last_peered":"2024-04-22T00:00:01.247311+0000","last_clean":"2024-04-22T00:00:01.247311+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:00:01.247311+0000","last_undegraded":"2024-04-22T00:00:01.247311+0000","last_fullsized":"2024-04-22T00:00:01.247311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896879+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-21T23:59:41.896879+0000","last_peered":"2024-04-21T23:59:41.896879+0000","last_clean":"2024-04-21T23:59:41.896879+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-21T23:59:41.896879+0000","last_undegraded":"2024-04-21T23:59:41.896879+0000","last_fullsized":"2024-04-21T23:59:41.896879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394542+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-21T23:59:41.394542+0000","last_peered":"2024-04-21T23:59:41.394542+0000","last_clean":"2024-04-21T23:59:41.394542+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-21T23:59:41.394542+0000","last_undegraded":"2024-04-21T23:59:41.394542+0000","last_fullsized":"2024-04-21T23:59:41.394542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.246903+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:00:01.246903+0000","last_peered":"2024-04-22T00:00:01.246903+0000","last_clean":"2024-04-22T00:00:01.246903+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:00:01.246903+0000","last_undegraded":"2024-04-22T00:00:01.246903+0000","last_fullsized":"2024-04-22T00:00:01.246903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896098+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-21T23:59:41.896098+0000","last_peered":"2024-04-21T23:59:41.896098+0000","last_clean":"2024-04-21T23:59:41.896098+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-21T23:59:41.896098+0000","last_undegraded":"2024-04-21T23:59:41.896098+0000","last_fullsized":"2024-04-21T23:59:41.896098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.819084+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-21T23:59:41.819084+0000","last_peered":"2024-04-21T23:59:41.819084+0000","last_clean":"2024-04-21T23:59:41.819084+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-21T23:59:41.819084+0000","last_undegraded":"2024-04-21T23:59:41.819084+0000","last_fullsized":"2024-04-21T23:59:41.819084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.254143+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:00:02.254143+0000","last_peered":"2024-04-22T00:00:02.254143+0000","last_clean":"2024-04-22T00:00:02.254143+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:00:02.254143+0000","last_undegraded":"2024-04-22T00:00:02.254143+0000","last_fullsized":"2024-04-22T00:00:02.254143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895993+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-21T23:59:41.895993+0000","last_peered":"2024-04-21T23:59:41.895993+0000","last_clean":"2024-04-21T23:59:41.895993+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-21T23:59:41.895993+0000","last_undegraded":"2024-04-21T23:59:41.895993+0000","last_fullsized":"2024-04-21T23:59:41.895993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896076+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-21T23:59:41.896076+0000","last_peered":"2024-04-21T23:59:41.896076+0000","last_clean":"2024-04-21T23:59:41.896076+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-21T23:59:41.896076+0000","last_undegraded":"2024-04-21T23:59:41.896076+0000","last_fullsized":"2024-04-21T23:59:41.896076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.260652+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:00:03.260652+0000","last_peered":"2024-04-22T00:00:03.260652+0000","last_clean":"2024-04-22T00:00:03.260652+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:00:03.260652+0000","last_undegraded":"2024-04-22T00:00:03.260652+0000","last_fullsized":"2024-04-22T00:00:03.260652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896250+0000","last_change":"2024-04-21T23:59:36.254670+0000","last_active":"2024-04-21T23:59:41.896250+0000","last_peered":"2024-04-21T23:59:41.896250+0000","last_clean":"2024-04-21T23:59:41.896250+0000","last_became_active":"2024-04-21T23:59:36.254368+0000","last_became_peered":"2024-04-21T23:59:36.254368+0000","last_unstale":"2024-04-21T23:59:41.896250+0000","last_undegraded":"2024-04-21T23:59:41.896250+0000","last_fullsized":"2024-04-21T23:59:41.896250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:20.985922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.395145+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-21T23:59:41.395145+0000","last_peered":"2024-04-21T23:59:41.395145+0000","last_clean":"2024-04-21T23:59:41.395145+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-21T23:59:41.395145+0000","last_undegraded":"2024-04-21T23:59:41.395145+0000","last_fullsized":"2024-04-21T23:59:41.395145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895256+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-21T23:59:41.895256+0000","last_peered":"2024-04-21T23:59:41.895256+0000","last_clean":"2024-04-21T23:59:41.895256+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-21T23:59:41.895256+0000","last_undegraded":"2024-04-21T23:59:41.895256+0000","last_fullsized":"2024-04-21T23:59:41.895256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895686+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-21T23:59:41.895686+0000","last_peered":"2024-04-21T23:59:41.895686+0000","last_clean":"2024-04-21T23:59:41.895686+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-21T23:59:41.895686+0000","last_undegraded":"2024-04-21T23:59:41.895686+0000","last_fullsized":"2024-04-21T23:59:41.895686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896983+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-21T23:59:41.896983+0000","last_peered":"2024-04-21T23:59:41.896983+0000","last_clean":"2024-04-21T23:59:41.896983+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-21T23:59:41.896983+0000","last_undegraded":"2024-04-21T23:59:41.896983+0000","last_fullsized":"2024-04-21T23:59:41.896983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.256345+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:00:02.256345+0000","last_peered":"2024-04-22T00:00:02.256345+0000","last_clean":"2024-04-22T00:00:02.256345+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:00:02.256345+0000","last_undegraded":"2024-04-22T00:00:02.256345+0000","last_fullsized":"2024-04-22T00:00:02.256345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818293+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-21T23:59:41.818293+0000","last_peered":"2024-04-21T23:59:41.818293+0000","last_clean":"2024-04-21T23:59:41.818293+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-21T23:59:41.818293+0000","last_undegraded":"2024-04-21T23:59:41.818293+0000","last_fullsized":"2024-04-21T23:59:41.818293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818419+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-21T23:59:41.818419+0000","last_peered":"2024-04-21T23:59:41.818419+0000","last_clean":"2024-04-21T23:59:41.818419+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-21T23:59:41.818419+0000","last_undegraded":"2024-04-21T23:59:41.818419+0000","last_fullsized":"2024-04-21T23:59:41.818419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896336+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-21T23:59:41.896336+0000","last_peered":"2024-04-21T23:59:41.896336+0000","last_clean":"2024-04-21T23:59:41.896336+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-21T23:59:41.896336+0000","last_undegraded":"2024-04-21T23:59:41.896336+0000","last_fullsized":"2024-04-21T23:59:41.896336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896047+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-21T23:59:41.896047+0000","last_peered":"2024-04-21T23:59:41.896047+0000","last_clean":"2024-04-21T23:59:41.896047+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-21T23:59:41.896047+0000","last_undegraded":"2024-04-21T23:59:41.896047+0000","last_fullsized":"2024-04-21T23:59:41.896047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895960+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-21T23:59:41.895960+0000","last_peered":"2024-04-21T23:59:41.895960+0000","last_clean":"2024-04-21T23:59:41.895960+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-21T23:59:41.895960+0000","last_undegraded":"2024-04-21T23:59:41.895960+0000","last_fullsized":"2024-04-21T23:59:41.895960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394866+0000","last_change":"2024-04-21T23:59:35.616463+0000","last_active":"2024-04-21T23:59:41.394866+0000","last_peered":"2024-04-21T23:59:41.394866+0000","last_clean":"2024-04-21T23:59:41.394866+0000","last_became_active":"2024-04-21T23:59:35.616158+0000","last_became_peered":"2024-04-21T23:59:35.616158+0000","last_unstale":"2024-04-21T23:59:41.394866+0000","last_undegraded":"2024-04-21T23:59:41.394866+0000","last_fullsized":"2024-04-21T23:59:41.394866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:30.736734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.398071+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-21T23:59:41.398071+0000","last_peered":"2024-04-21T23:59:41.398071+0000","last_clean":"2024-04-21T23:59:41.398071+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-21T23:59:41.398071+0000","last_undegraded":"2024-04-21T23:59:41.398071+0000","last_fullsized":"2024-04-21T23:59:41.398071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896802+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-21T23:59:41.896802+0000","last_peered":"2024-04-21T23:59:41.896802+0000","last_clean":"2024-04-21T23:59:41.896802+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-21T23:59:41.896802+0000","last_undegraded":"2024-04-21T23:59:41.896802+0000","last_fullsized":"2024-04-21T23:59:41.896802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896766+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-21T23:59:41.896766+0000","last_peered":"2024-04-21T23:59:41.896766+0000","last_clean":"2024-04-21T23:59:41.896766+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-21T23:59:41.896766+0000","last_undegraded":"2024-04-21T23:59:41.896766+0000","last_fullsized":"2024-04-21T23:59:41.896766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397687+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-21T23:59:41.397687+0000","last_peered":"2024-04-21T23:59:41.397687+0000","last_clean":"2024-04-21T23:59:41.397687+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-21T23:59:41.397687+0000","last_undegraded":"2024-04-21T23:59:41.397687+0000","last_fullsized":"2024-04-21T23:59:41.397687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896455+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-21T23:59:41.896455+0000","last_peered":"2024-04-21T23:59:41.896455+0000","last_clean":"2024-04-21T23:59:41.896455+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-21T23:59:41.896455+0000","last_undegraded":"2024-04-21T23:59:41.896455+0000","last_fullsized":"2024-04-21T23:59:41.896455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818338+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-21T23:59:41.818338+0000","last_peered":"2024-04-21T23:59:41.818338+0000","last_clean":"2024-04-21T23:59:41.818338+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-21T23:59:41.818338+0000","last_undegraded":"2024-04-21T23:59:41.818338+0000","last_fullsized":"2024-04-21T23:59:41.818338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895484+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-21T23:59:41.895484+0000","last_peered":"2024-04-21T23:59:41.895484+0000","last_clean":"2024-04-21T23:59:41.895484+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-21T23:59:41.895484+0000","last_undegraded":"2024-04-21T23:59:41.895484+0000","last_fullsized":"2024-04-21T23:59:41.895484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896210+0000","last_change":"2024-04-21T23:59:40.412963+0000","last_active":"2024-04-21T23:59:41.896210+0000","last_peered":"2024-04-21T23:59:41.896210+0000","last_clean":"2024-04-21T23:59:41.896210+0000","last_became_active":"2024-04-21T23:59:40.412568+0000","last_became_peered":"2024-04-21T23:59:40.412568+0000","last_unstale":"2024-04-21T23:59:41.896210+0000","last_undegraded":"2024-04-21T23:59:41.896210+0000","last_fullsized":"2024-04-21T23:59:41.896210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:45.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895180+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-21T23:59:41.895180+0000","last_peered":"2024-04-21T23:59:41.895180+0000","last_clean":"2024-04-21T23:59:41.895180+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-21T23:59:41.895180+0000","last_undegraded":"2024-04-21T23:59:41.895180+0000","last_fullsized":"2024-04-21T23:59:41.895180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895662+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-21T23:59:41.895662+0000","last_peered":"2024-04-21T23:59:41.895662+0000","last_clean":"2024-04-21T23:59:41.895662+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-21T23:59:41.895662+0000","last_undegraded":"2024-04-21T23:59:41.895662+0000","last_fullsized":"2024-04-21T23:59:41.895662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895495+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-21T23:59:41.895495+0000","last_peered":"2024-04-21T23:59:41.895495+0000","last_clean":"2024-04-21T23:59:41.895495+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-21T23:59:41.895495+0000","last_undegraded":"2024-04-21T23:59:41.895495+0000","last_fullsized":"2024-04-21T23:59:41.895495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818854+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-21T23:59:41.818854+0000","last_peered":"2024-04-21T23:59:41.818854+0000","last_clean":"2024-04-21T23:59:41.818854+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-21T23:59:41.818854+0000","last_undegraded":"2024-04-21T23:59:41.818854+0000","last_fullsized":"2024-04-21T23:59:41.818854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895341+0000","last_change":"2024-04-21T23:59:36.255179+0000","last_active":"2024-04-21T23:59:41.895341+0000","last_peered":"2024-04-21T23:59:41.895341+0000","last_clean":"2024-04-21T23:59:41.895341+0000","last_became_active":"2024-04-21T23:59:36.254858+0000","last_became_peered":"2024-04-21T23:59:36.254858+0000","last_unstale":"2024-04-21T23:59:41.895341+0000","last_undegraded":"2024-04-21T23:59:41.895341+0000","last_fullsized":"2024-04-21T23:59:41.895341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:36:37.308326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.253562+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:00:02.253562+0000","last_peered":"2024-04-22T00:00:02.253562+0000","last_clean":"2024-04-22T00:00:02.253562+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:00:02.253562+0000","last_undegraded":"2024-04-22T00:00:02.253562+0000","last_fullsized":"2024-04-22T00:00:02.253562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895883+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-21T23:59:41.895883+0000","last_peered":"2024-04-21T23:59:41.895883+0000","last_clean":"2024-04-21T23:59:41.895883+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-21T23:59:41.895883+0000","last_undegraded":"2024-04-21T23:59:41.895883+0000","last_fullsized":"2024-04-21T23:59:41.895883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896197+0000","last_change":"2024-04-21T23:59:35.613025+0000","last_active":"2024-04-21T23:59:41.896197+0000","last_peered":"2024-04-21T23:59:41.896197+0000","last_clean":"2024-04-21T23:59:41.896197+0000","last_became_active":"2024-04-21T23:59:35.612725+0000","last_became_peered":"2024-04-21T23:59:35.612725+0000","last_unstale":"2024-04-21T23:59:41.896197+0000","last_undegraded":"2024-04-21T23:59:41.896197+0000","last_fullsized":"2024-04-21T23:59:41.896197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:31:17.866391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896972+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-21T23:59:41.896972+0000","last_peered":"2024-04-21T23:59:41.896972+0000","last_clean":"2024-04-21T23:59:41.896972+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-21T23:59:41.896972+0000","last_undegraded":"2024-04-21T23:59:41.896972+0000","last_fullsized":"2024-04-21T23:59:41.896972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896020+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-21T23:59:41.896020+0000","last_peered":"2024-04-21T23:59:41.896020+0000","last_clean":"2024-04-21T23:59:41.896020+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-21T23:59:41.896020+0000","last_undegraded":"2024-04-21T23:59:41.896020+0000","last_fullsized":"2024-04-21T23:59:41.896020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.263190+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:00:03.263190+0000","last_peered":"2024-04-22T00:00:03.263190+0000","last_clean":"2024-04-22T00:00:03.263190+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:00:03.263190+0000","last_undegraded":"2024-04-22T00:00:03.263190+0000","last_fullsized":"2024-04-22T00:00:03.263190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896086+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-21T23:59:41.896086+0000","last_peered":"2024-04-21T23:59:41.896086+0000","last_clean":"2024-04-21T23:59:41.896086+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-21T23:59:41.896086+0000","last_undegraded":"2024-04-21T23:59:41.896086+0000","last_fullsized":"2024-04-21T23:59:41.896086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393432+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-21T23:59:41.393432+0000","last_peered":"2024-04-21T23:59:41.393432+0000","last_clean":"2024-04-21T23:59:41.393432+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-21T23:59:41.393432+0000","last_undegraded":"2024-04-21T23:59:41.393432+0000","last_fullsized":"2024-04-21T23:59:41.393432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895471+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-21T23:59:41.895471+0000","last_peered":"2024-04-21T23:59:41.895471+0000","last_clean":"2024-04-21T23:59:41.895471+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-21T23:59:41.895471+0000","last_undegraded":"2024-04-21T23:59:41.895471+0000","last_fullsized":"2024-04-21T23:59:41.895471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895735+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-21T23:59:41.895735+0000","last_peered":"2024-04-21T23:59:41.895735+0000","last_clean":"2024-04-21T23:59:41.895735+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-21T23:59:41.895735+0000","last_undegraded":"2024-04-21T23:59:41.895735+0000","last_fullsized":"2024-04-21T23:59:41.895735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895404+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-21T23:59:41.895404+0000","last_peered":"2024-04-21T23:59:41.895404+0000","last_clean":"2024-04-21T23:59:41.895404+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-21T23:59:41.895404+0000","last_undegraded":"2024-04-21T23:59:41.895404+0000","last_fullsized":"2024-04-21T23:59:41.895404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247976+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:00:01.247976+0000","last_peered":"2024-04-22T00:00:01.247976+0000","last_clean":"2024-04-22T00:00:01.247976+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:00:01.247976+0000","last_undegraded":"2024-04-22T00:00:01.247976+0000","last_fullsized":"2024-04-22T00:00:01.247976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895372+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-21T23:59:41.895372+0000","last_peered":"2024-04-21T23:59:41.895372+0000","last_clean":"2024-04-21T23:59:41.895372+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-21T23:59:41.895372+0000","last_undegraded":"2024-04-21T23:59:41.895372+0000","last_fullsized":"2024-04-21T23:59:41.895372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394992+0000","last_change":"2024-04-21T23:59:40.419040+0000","last_active":"2024-04-21T23:59:41.394992+0000","last_peered":"2024-04-21T23:59:41.394992+0000","last_clean":"2024-04-21T23:59:41.394992+0000","last_became_active":"2024-04-21T23:59:40.418112+0000","last_became_peered":"2024-04-21T23:59:40.418112+0000","last_unstale":"2024-04-21T23:59:41.394992+0000","last_undegraded":"2024-04-21T23:59:41.394992+0000","last_fullsized":"2024-04-21T23:59:41.394992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:31:18.156358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:04.273972+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:00:04.273972+0000","last_peered":"2024-04-22T00:00:04.273972+0000","last_clean":"2024-04-22T00:00:04.273972+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:00:04.273972+0000","last_undegraded":"2024-04-22T00:00:04.273972+0000","last_fullsized":"2024-04-22T00:00:04.273972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895409+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-21T23:59:41.895409+0000","last_peered":"2024-04-21T23:59:41.895409+0000","last_clean":"2024-04-21T23:59:41.895409+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-21T23:59:41.895409+0000","last_undegraded":"2024-04-21T23:59:41.895409+0000","last_fullsized":"2024-04-21T23:59:41.895409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896145+0000","last_change":"2024-04-21T23:59:40.409609+0000","last_active":"2024-04-21T23:59:41.896145+0000","last_peered":"2024-04-21T23:59:41.896145+0000","last_clean":"2024-04-21T23:59:41.896145+0000","last_became_active":"2024-04-21T23:59:40.408951+0000","last_became_peered":"2024-04-21T23:59:40.408951+0000","last_unstale":"2024-04-21T23:59:41.896145+0000","last_undegraded":"2024-04-21T23:59:41.896145+0000","last_fullsized":"2024-04-21T23:59:41.896145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:51:53.973363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.242426+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:00:02.242426+0000","last_peered":"2024-04-22T00:00:02.242426+0000","last_clean":"2024-04-22T00:00:02.242426+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:00:02.242426+0000","last_undegraded":"2024-04-22T00:00:02.242426+0000","last_fullsized":"2024-04-22T00:00:02.242426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895822+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-21T23:59:41.895822+0000","last_peered":"2024-04-21T23:59:41.895822+0000","last_clean":"2024-04-21T23:59:41.895822+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-21T23:59:41.895822+0000","last_undegraded":"2024-04-21T23:59:41.895822+0000","last_fullsized":"2024-04-21T23:59:41.895822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895581+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-21T23:59:41.895581+0000","last_peered":"2024-04-21T23:59:41.895581+0000","last_clean":"2024-04-21T23:59:41.895581+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-21T23:59:41.895581+0000","last_undegraded":"2024-04-21T23:59:41.895581+0000","last_fullsized":"2024-04-21T23:59:41.895581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247815+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:00:01.247815+0000","last_peered":"2024-04-22T00:00:01.247815+0000","last_clean":"2024-04-22T00:00:01.247815+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:00:01.247815+0000","last_undegraded":"2024-04-22T00:00:01.247815+0000","last_fullsized":"2024-04-22T00:00:01.247815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896751+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-21T23:59:41.896751+0000","last_peered":"2024-04-21T23:59:41.896751+0000","last_clean":"2024-04-21T23:59:41.896751+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-21T23:59:41.896751+0000","last_undegraded":"2024-04-21T23:59:41.896751+0000","last_fullsized":"2024-04-21T23:59:41.896751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896283+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-21T23:59:41.896283+0000","last_peered":"2024-04-21T23:59:41.896283+0000","last_clean":"2024-04-21T23:59:41.896283+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-21T23:59:41.896283+0000","last_undegraded":"2024-04-21T23:59:41.896283+0000","last_fullsized":"2024-04-21T23:59:41.896283+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.259395+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:00:03.259395+0000","last_peered":"2024-04-22T00:00:03.259395+0000","last_clean":"2024-04-22T00:00:03.259395+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:00:03.259395+0000","last_undegraded":"2024-04-22T00:00:03.259395+0000","last_fullsized":"2024-04-22T00:00:03.259395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896498+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-21T23:59:41.896498+0000","last_peered":"2024-04-21T23:59:41.896498+0000","last_clean":"2024-04-21T23:59:41.896498+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-21T23:59:41.896498+0000","last_undegraded":"2024-04-21T23:59:41.896498+0000","last_fullsized":"2024-04-21T23:59:41.896498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895624+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-21T23:59:41.895624+0000","last_peered":"2024-04-21T23:59:41.895624+0000","last_clean":"2024-04-21T23:59:41.895624+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-21T23:59:41.895624+0000","last_undegraded":"2024-04-21T23:59:41.895624+0000","last_fullsized":"2024-04-21T23:59:41.895624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.260716+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:00:03.260716+0000","last_peered":"2024-04-22T00:00:03.260716+0000","last_clean":"2024-04-22T00:00:03.260716+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:00:03.260716+0000","last_undegraded":"2024-04-22T00:00:03.260716+0000","last_fullsized":"2024-04-22T00:00:03.260716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896587+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-21T23:59:41.896587+0000","last_peered":"2024-04-21T23:59:41.896587+0000","last_clean":"2024-04-21T23:59:41.896587+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-21T23:59:41.896587+0000","last_undegraded":"2024-04-21T23:59:41.896587+0000","last_fullsized":"2024-04-21T23:59:41.896587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818155+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-21T23:59:41.818155+0000","last_peered":"2024-04-21T23:59:41.818155+0000","last_clean":"2024-04-21T23:59:41.818155+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-21T23:59:41.818155+0000","last_undegraded":"2024-04-21T23:59:41.818155+0000","last_fullsized":"2024-04-21T23:59:41.818155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896811+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-21T23:59:41.896811+0000","last_peered":"2024-04-21T23:59:41.896811+0000","last_clean":"2024-04-21T23:59:41.896811+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-21T23:59:41.896811+0000","last_undegraded":"2024-04-21T23:59:41.896811+0000","last_fullsized":"2024-04-21T23:59:41.896811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896536+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-21T23:59:41.896536+0000","last_peered":"2024-04-21T23:59:41.896536+0000","last_clean":"2024-04-21T23:59:41.896536+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-21T23:59:41.896536+0000","last_undegraded":"2024-04-21T23:59:41.896536+0000","last_fullsized":"2024-04-21T23:59:41.896536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895498+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-21T23:59:41.895498+0000","last_peered":"2024-04-21T23:59:41.895498+0000","last_clean":"2024-04-21T23:59:41.895498+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-21T23:59:41.895498+0000","last_undegraded":"2024-04-21T23:59:41.895498+0000","last_fullsized":"2024-04-21T23:59:41.895498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393669+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-21T23:59:41.393669+0000","last_peered":"2024-04-21T23:59:41.393669+0000","last_clean":"2024-04-21T23:59:41.393669+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-21T23:59:41.393669+0000","last_undegraded":"2024-04-21T23:59:41.393669+0000","last_fullsized":"2024-04-21T23:59:41.393669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393303+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-21T23:59:41.393303+0000","last_peered":"2024-04-21T23:59:41.393303+0000","last_clean":"2024-04-21T23:59:41.393303+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-21T23:59:41.393303+0000","last_undegraded":"2024-04-21T23:59:41.393303+0000","last_fullsized":"2024-04-21T23:59:41.393303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895919+0000","last_change":"2024-04-21T23:59:36.249178+0000","last_active":"2024-04-21T23:59:41.895919+0000","last_peered":"2024-04-21T23:59:41.895919+0000","last_clean":"2024-04-21T23:59:41.895919+0000","last_became_active":"2024-04-21T23:59:36.248728+0000","last_became_peered":"2024-04-21T23:59:36.248728+0000","last_unstale":"2024-04-21T23:59:41.895919+0000","last_undegraded":"2024-04-21T23:59:41.895919+0000","last_fullsized":"2024-04-21T23:59:41.895919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:37:15.983221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897066+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-21T23:59:41.897066+0000","last_peered":"2024-04-21T23:59:41.897066+0000","last_clean":"2024-04-21T23:59:41.897066+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-21T23:59:41.897066+0000","last_undegraded":"2024-04-21T23:59:41.897066+0000","last_fullsized":"2024-04-21T23:59:41.897066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896097+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-21T23:59:41.896097+0000","last_peered":"2024-04-21T23:59:41.896097+0000","last_clean":"2024-04-21T23:59:41.896097+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-21T23:59:41.896097+0000","last_undegraded":"2024-04-21T23:59:41.896097+0000","last_fullsized":"2024-04-21T23:59:41.896097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394083+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-21T23:59:41.394083+0000","last_peered":"2024-04-21T23:59:41.394083+0000","last_clean":"2024-04-21T23:59:41.394083+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-21T23:59:41.394083+0000","last_undegraded":"2024-04-21T23:59:41.394083+0000","last_fullsized":"2024-04-21T23:59:41.394083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896359+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-21T23:59:41.896359+0000","last_peered":"2024-04-21T23:59:41.896359+0000","last_clean":"2024-04-21T23:59:41.896359+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-21T23:59:41.896359+0000","last_undegraded":"2024-04-21T23:59:41.896359+0000","last_fullsized":"2024-04-21T23:59:41.896359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818285+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-21T23:59:41.818285+0000","last_peered":"2024-04-21T23:59:41.818285+0000","last_clean":"2024-04-21T23:59:41.818285+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-21T23:59:41.818285+0000","last_undegraded":"2024-04-21T23:59:41.818285+0000","last_fullsized":"2024-04-21T23:59:41.818285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896247+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-21T23:59:41.896247+0000","last_peered":"2024-04-21T23:59:41.896247+0000","last_clean":"2024-04-21T23:59:41.896247+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-21T23:59:41.896247+0000","last_undegraded":"2024-04-21T23:59:41.896247+0000","last_fullsized":"2024-04-21T23:59:41.896247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896525+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-21T23:59:41.896525+0000","last_peered":"2024-04-21T23:59:41.896525+0000","last_clean":"2024-04-21T23:59:41.896525+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-21T23:59:41.896525+0000","last_undegraded":"2024-04-21T23:59:41.896525+0000","last_fullsized":"2024-04-21T23:59:41.896525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896477+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-21T23:59:41.896477+0000","last_peered":"2024-04-21T23:59:41.896477+0000","last_clean":"2024-04-21T23:59:41.896477+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-21T23:59:41.896477+0000","last_undegraded":"2024-04-21T23:59:41.896477+0000","last_fullsized":"2024-04-21T23:59:41.896477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896049+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-21T23:59:41.896049+0000","last_peered":"2024-04-21T23:59:41.896049+0000","last_clean":"2024-04-21T23:59:41.896049+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-21T23:59:41.896049+0000","last_undegraded":"2024-04-21T23:59:41.896049+0000","last_fullsized":"2024-04-21T23:59:41.896049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8124,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":134,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":43711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1069375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":483029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32756,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712396,"statfs":{"total":95995035648,"available":95961493504,"internally_reserved":0,"allocated":1617920,"data_stored":1069821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:00:06.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T00:00:07.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 -- 172.21.15.38:0/4096641368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 msgr2=0x7f5bf8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 --2- 172.21.15.38:0/4096641368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf8108ae0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009900 tx=0x7f5be002bc30 comp rx=0 tx=0).stop 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 -- 172.21.15.38:0/4096641368 shutdown_connections 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 --2- 172.21.15.38:0/4096641368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf8108ae0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 --2- 172.21.15.38:0/4096641368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf8101890 0x7f5bf8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 --2- 172.21.15.38:0/4096641368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf81006a0 0x7f5bf8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.047+0000 7f5bff13a640 1 -- 172.21.15.38:0/4096641368 >> 172.21.15.38:0/4096641368 conn(0x7f5bf80fbc30 msgr2=0x7f5bf80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- 172.21.15.38:0/4096641368 shutdown_connections 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- 172.21.15.38:0/4096641368 wait complete. 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 Processor -- start 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- start start 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 0x7f5bf806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:07.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf8101890 0x7f5bf819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810c350 con 0x7f5bf8102250 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810c1d0 con 0x7f5bf8101890 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bff13a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810c4d0 con 0x7f5bf81006a0 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bfd6d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bfd6d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40608/0 (socket says 172.21.15.38:40608) 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bfd6d7640 1 -- 172.21.15.38:0/3689114532 learned_addr learned my addr 172.21.15.38:0/3689114532 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.048+0000 7f5bf7fff640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf8101890 0x7f5bf819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfced6640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 0x7f5bf806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 -- 172.21.15.38:0/3689114532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 msgr2=0x7f5bf806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 0x7f5bf806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 -- 172.21.15.38:0/3689114532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf8101890 msgr2=0x7f5bf819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf8101890 0x7f5bf819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 -- 172.21.15.38:0/3689114532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf819f700 con 0x7f5bf8102250 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfced6640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 0x7f5bf806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bfd6d7640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5be0005ab0 tx=0x7f5be0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0043020 con 0x7f5bf8102250 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf819f930 con 0x7f5bf8102250 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be00054e0 con 0x7f5bf8102250 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0003e20 con 0x7f5bf8102250 2024-04-22T00:00:07.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.049+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf819fed0 con 0x7f5bf8102250 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.050+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf81078f0 con 0x7f5bf8102250 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.051+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5be0046020 con 0x7f5bf8102250 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.052+0000 7f5bf5ffb640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 0x7f5be40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.052+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5be00f1990 con 0x7f5bf8102250 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.052+0000 7f5bfced6640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 0x7f5be40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:07.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.053+0000 7f5bfced6640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 0x7f5be40a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5bf81016f0 tx=0x7f5bf0005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:07.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.056+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5be004d020 con 0x7f5bf8102250 2024-04-22T00:00:07.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.237+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5bf8061c40 con 0x7f5be40a2c20 2024-04-22T00:00:07.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.246+0000 7f5bf5ffb640 1 -- 172.21.15.38:0/3689114532 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340315 (secure 0 0 0) 0x7f5bf8061c40 con 0x7f5be40a2c20 2024-04-22T00:00:07.243 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:07.244 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 msgr2=0x7f5be40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 0x7f5be40a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5bf81016f0 tx=0x7f5bf0005e70 comp rx=0 tx=0).stop 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 msgr2=0x7f5bf819edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5be0005ab0 tx=0x7f5be0005dc0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 shutdown_connections 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5be40a2c20 0x7f5be40a50e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bf8102250 0x7f5bf819edf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bf8101890 0x7f5bf819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 --2- 172.21.15.38:0/3689114532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bf81006a0 0x7f5bf806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:07.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 >> 172.21.15.38:0/3689114532 conn(0x7f5bf80fbc30 msgr2=0x7f5bf8104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:07.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 shutdown_connections 2024-04-22T00:00:07.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:07.250+0000 7f5bff13a640 1 -- 172.21.15.38:0/3689114532 wait complete. 2024-04-22T00:00:07.254 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-22T00:00:05.654491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598492,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":138,"num_read_kb":121,"num_write":315,"num_write_kb":2984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354372,"kb_used_data":13524,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587452,"statfs":{"total":1151940427776,"available":1151577550848,"internally_reserved":0,"allocated":13848576,"data_stored":7531578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":348894524},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3625,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":87,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896421+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-21T23:59:41.896421+0000","last_peered":"2024-04-21T23:59:41.896421+0000","last_clean":"2024-04-21T23:59:41.896421+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-21T23:59:41.896421+0000","last_undegraded":"2024-04-21T23:59:41.896421+0000","last_fullsized":"2024-04-21T23:59:41.896421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895638+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-21T23:59:41.895638+0000","last_peered":"2024-04-21T23:59:41.895638+0000","last_clean":"2024-04-21T23:59:41.895638+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-21T23:59:41.895638+0000","last_undegraded":"2024-04-21T23:59:41.895638+0000","last_fullsized":"2024-04-21T23:59:41.895638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895160+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-21T23:59:41.895160+0000","last_peered":"2024-04-21T23:59:41.895160+0000","last_clean":"2024-04-21T23:59:41.895160+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-21T23:59:41.895160+0000","last_undegraded":"2024-04-21T23:59:41.895160+0000","last_fullsized":"2024-04-21T23:59:41.895160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896475+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-21T23:59:41.896475+0000","last_peered":"2024-04-21T23:59:41.896475+0000","last_clean":"2024-04-21T23:59:41.896475+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-21T23:59:41.896475+0000","last_undegraded":"2024-04-21T23:59:41.896475+0000","last_fullsized":"2024-04-21T23:59:41.896475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895577+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-21T23:59:41.895577+0000","last_peered":"2024-04-21T23:59:41.895577+0000","last_clean":"2024-04-21T23:59:41.895577+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-21T23:59:41.895577+0000","last_undegraded":"2024-04-21T23:59:41.895577+0000","last_fullsized":"2024-04-21T23:59:41.895577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397644+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-21T23:59:41.397644+0000","last_peered":"2024-04-21T23:59:41.397644+0000","last_clean":"2024-04-21T23:59:41.397644+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-21T23:59:41.397644+0000","last_undegraded":"2024-04-21T23:59:41.397644+0000","last_fullsized":"2024-04-21T23:59:41.397644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896636+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-21T23:59:41.896636+0000","last_peered":"2024-04-21T23:59:41.896636+0000","last_clean":"2024-04-21T23:59:41.896636+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-21T23:59:41.896636+0000","last_undegraded":"2024-04-21T23:59:41.896636+0000","last_fullsized":"2024-04-21T23:59:41.896636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896479+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-21T23:59:41.896479+0000","last_peered":"2024-04-21T23:59:41.896479+0000","last_clean":"2024-04-21T23:59:41.896479+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-21T23:59:41.896479+0000","last_undegraded":"2024-04-21T23:59:41.896479+0000","last_fullsized":"2024-04-21T23:59:41.896479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896245+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-21T23:59:41.896245+0000","last_peered":"2024-04-21T23:59:41.896245+0000","last_clean":"2024-04-21T23:59:41.896245+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-21T23:59:41.896245+0000","last_undegraded":"2024-04-21T23:59:41.896245+0000","last_fullsized":"2024-04-21T23:59:41.896245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.398121+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-21T23:59:41.398121+0000","last_peered":"2024-04-21T23:59:41.398121+0000","last_clean":"2024-04-21T23:59:41.398121+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-21T23:59:41.398121+0000","last_undegraded":"2024-04-21T23:59:41.398121+0000","last_fullsized":"2024-04-21T23:59:41.398121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896336+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-21T23:59:41.896336+0000","last_peered":"2024-04-21T23:59:41.896336+0000","last_clean":"2024-04-21T23:59:41.896336+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-21T23:59:41.896336+0000","last_undegraded":"2024-04-21T23:59:41.896336+0000","last_fullsized":"2024-04-21T23:59:41.896336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896378+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-21T23:59:41.896378+0000","last_peered":"2024-04-21T23:59:41.896378+0000","last_clean":"2024-04-21T23:59:41.896378+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-21T23:59:41.896378+0000","last_undegraded":"2024-04-21T23:59:41.896378+0000","last_fullsized":"2024-04-21T23:59:41.896378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896379+0000","last_change":"2024-04-21T23:59:36.248992+0000","last_active":"2024-04-21T23:59:41.896379+0000","last_peered":"2024-04-21T23:59:41.896379+0000","last_clean":"2024-04-21T23:59:41.896379+0000","last_became_active":"2024-04-21T23:59:36.248652+0000","last_became_peered":"2024-04-21T23:59:36.248652+0000","last_unstale":"2024-04-21T23:59:41.896379+0000","last_undegraded":"2024-04-21T23:59:41.896379+0000","last_fullsized":"2024-04-21T23:59:41.896379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:30.301519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895152+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-21T23:59:41.895152+0000","last_peered":"2024-04-21T23:59:41.895152+0000","last_clean":"2024-04-21T23:59:41.895152+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-21T23:59:41.895152+0000","last_undegraded":"2024-04-21T23:59:41.895152+0000","last_fullsized":"2024-04-21T23:59:41.895152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896016+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-21T23:59:41.896016+0000","last_peered":"2024-04-21T23:59:41.896016+0000","last_clean":"2024-04-21T23:59:41.896016+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-21T23:59:41.896016+0000","last_undegraded":"2024-04-21T23:59:41.896016+0000","last_fullsized":"2024-04-21T23:59:41.896016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.894921+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-21T23:59:41.894921+0000","last_peered":"2024-04-21T23:59:41.894921+0000","last_clean":"2024-04-21T23:59:41.894921+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-21T23:59:41.894921+0000","last_undegraded":"2024-04-21T23:59:41.894921+0000","last_fullsized":"2024-04-21T23:59:41.894921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896254+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-21T23:59:41.896254+0000","last_peered":"2024-04-21T23:59:41.896254+0000","last_clean":"2024-04-21T23:59:41.896254+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-21T23:59:41.896254+0000","last_undegraded":"2024-04-21T23:59:41.896254+0000","last_fullsized":"2024-04-21T23:59:41.896254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.395153+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-21T23:59:41.395153+0000","last_peered":"2024-04-21T23:59:41.395153+0000","last_clean":"2024-04-21T23:59:41.395153+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-21T23:59:41.395153+0000","last_undegraded":"2024-04-21T23:59:41.395153+0000","last_fullsized":"2024-04-21T23:59:41.395153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394037+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-21T23:59:41.394037+0000","last_peered":"2024-04-21T23:59:41.394037+0000","last_clean":"2024-04-21T23:59:41.394037+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-21T23:59:41.394037+0000","last_undegraded":"2024-04-21T23:59:41.394037+0000","last_fullsized":"2024-04-21T23:59:41.394037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393688+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-21T23:59:41.393688+0000","last_peered":"2024-04-21T23:59:41.393688+0000","last_clean":"2024-04-21T23:59:41.393688+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-21T23:59:41.393688+0000","last_undegraded":"2024-04-21T23:59:41.393688+0000","last_fullsized":"2024-04-21T23:59:41.393688+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:05.279528+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:00:05.279528+0000","last_peered":"2024-04-22T00:00:05.279528+0000","last_clean":"2024-04-22T00:00:05.279528+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:00:05.279528+0000","last_undegraded":"2024-04-22T00:00:05.279528+0000","last_fullsized":"2024-04-22T00:00:05.279528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895112+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-21T23:59:41.895112+0000","last_peered":"2024-04-21T23:59:41.895112+0000","last_clean":"2024-04-21T23:59:41.895112+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-21T23:59:41.895112+0000","last_undegraded":"2024-04-21T23:59:41.895112+0000","last_fullsized":"2024-04-21T23:59:41.895112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895043+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-21T23:59:41.895043+0000","last_peered":"2024-04-21T23:59:41.895043+0000","last_clean":"2024-04-21T23:59:41.895043+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-21T23:59:41.895043+0000","last_undegraded":"2024-04-21T23:59:41.895043+0000","last_fullsized":"2024-04-21T23:59:41.895043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896191+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-21T23:59:41.896191+0000","last_peered":"2024-04-21T23:59:41.896191+0000","last_clean":"2024-04-21T23:59:41.896191+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-21T23:59:41.896191+0000","last_undegraded":"2024-04-21T23:59:41.896191+0000","last_fullsized":"2024-04-21T23:59:41.896191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897309+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-21T23:59:41.897309+0000","last_peered":"2024-04-21T23:59:41.897309+0000","last_clean":"2024-04-21T23:59:41.897309+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-21T23:59:41.897309+0000","last_undegraded":"2024-04-21T23:59:41.897309+0000","last_fullsized":"2024-04-21T23:59:41.897309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895329+0000","last_change":"2024-04-21T23:59:40.414367+0000","last_active":"2024-04-21T23:59:41.895329+0000","last_peered":"2024-04-21T23:59:41.895329+0000","last_clean":"2024-04-21T23:59:41.895329+0000","last_became_active":"2024-04-21T23:59:40.413580+0000","last_became_peered":"2024-04-21T23:59:40.413580+0000","last_unstale":"2024-04-21T23:59:41.895329+0000","last_undegraded":"2024-04-21T23:59:41.895329+0000","last_fullsized":"2024-04-21T23:59:41.895329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:13.489706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.255235+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:00:02.255235+0000","last_peered":"2024-04-22T00:00:02.255235+0000","last_clean":"2024-04-22T00:00:02.255235+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:00:02.255235+0000","last_undegraded":"2024-04-22T00:00:02.255235+0000","last_fullsized":"2024-04-22T00:00:02.255235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896792+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-21T23:59:41.896792+0000","last_peered":"2024-04-21T23:59:41.896792+0000","last_clean":"2024-04-21T23:59:41.896792+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-21T23:59:41.896792+0000","last_undegraded":"2024-04-21T23:59:41.896792+0000","last_fullsized":"2024-04-21T23:59:41.896792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895148+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-21T23:59:41.895148+0000","last_peered":"2024-04-21T23:59:41.895148+0000","last_clean":"2024-04-21T23:59:41.895148+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-21T23:59:41.895148+0000","last_undegraded":"2024-04-21T23:59:41.895148+0000","last_fullsized":"2024-04-21T23:59:41.895148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895805+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-21T23:59:41.895805+0000","last_peered":"2024-04-21T23:59:41.895805+0000","last_clean":"2024-04-21T23:59:41.895805+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-21T23:59:41.895805+0000","last_undegraded":"2024-04-21T23:59:41.895805+0000","last_fullsized":"2024-04-21T23:59:41.895805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818772+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-21T23:59:41.818772+0000","last_peered":"2024-04-21T23:59:41.818772+0000","last_clean":"2024-04-21T23:59:41.818772+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-21T23:59:41.818772+0000","last_undegraded":"2024-04-21T23:59:41.818772+0000","last_fullsized":"2024-04-21T23:59:41.818772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393937+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-21T23:59:41.393937+0000","last_peered":"2024-04-21T23:59:41.393937+0000","last_clean":"2024-04-21T23:59:41.393937+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-21T23:59:41.393937+0000","last_undegraded":"2024-04-21T23:59:41.393937+0000","last_fullsized":"2024-04-21T23:59:41.393937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.248122+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:00:01.248122+0000","last_peered":"2024-04-22T00:00:01.248122+0000","last_clean":"2024-04-22T00:00:01.248122+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:00:01.248122+0000","last_undegraded":"2024-04-22T00:00:01.248122+0000","last_fullsized":"2024-04-22T00:00:01.248122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895671+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-21T23:59:41.895671+0000","last_peered":"2024-04-21T23:59:41.895671+0000","last_clean":"2024-04-21T23:59:41.895671+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-21T23:59:41.895671+0000","last_undegraded":"2024-04-21T23:59:41.895671+0000","last_fullsized":"2024-04-21T23:59:41.895671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393867+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-21T23:59:41.393867+0000","last_peered":"2024-04-21T23:59:41.393867+0000","last_clean":"2024-04-21T23:59:41.393867+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-21T23:59:41.393867+0000","last_undegraded":"2024-04-21T23:59:41.393867+0000","last_fullsized":"2024-04-21T23:59:41.393867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397631+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-21T23:59:41.397631+0000","last_peered":"2024-04-21T23:59:41.397631+0000","last_clean":"2024-04-21T23:59:41.397631+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-21T23:59:41.397631+0000","last_undegraded":"2024-04-21T23:59:41.397631+0000","last_fullsized":"2024-04-21T23:59:41.397631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818585+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-21T23:59:41.818585+0000","last_peered":"2024-04-21T23:59:41.818585+0000","last_clean":"2024-04-21T23:59:41.818585+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-21T23:59:41.818585+0000","last_undegraded":"2024-04-21T23:59:41.818585+0000","last_fullsized":"2024-04-21T23:59:41.818585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896343+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-21T23:59:41.896343+0000","last_peered":"2024-04-21T23:59:41.896343+0000","last_clean":"2024-04-21T23:59:41.896343+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-21T23:59:41.896343+0000","last_undegraded":"2024-04-21T23:59:41.896343+0000","last_fullsized":"2024-04-21T23:59:41.896343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895536+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-21T23:59:41.895536+0000","last_peered":"2024-04-21T23:59:41.895536+0000","last_clean":"2024-04-21T23:59:41.895536+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-21T23:59:41.895536+0000","last_undegraded":"2024-04-21T23:59:41.895536+0000","last_fullsized":"2024-04-21T23:59:41.895536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896376+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-21T23:59:41.896376+0000","last_peered":"2024-04-21T23:59:41.896376+0000","last_clean":"2024-04-21T23:59:41.896376+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-21T23:59:41.896376+0000","last_undegraded":"2024-04-21T23:59:41.896376+0000","last_fullsized":"2024-04-21T23:59:41.896376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895523+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-21T23:59:41.895523+0000","last_peered":"2024-04-21T23:59:41.895523+0000","last_clean":"2024-04-21T23:59:41.895523+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-21T23:59:41.895523+0000","last_undegraded":"2024-04-21T23:59:41.895523+0000","last_fullsized":"2024-04-21T23:59:41.895523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.261058+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:00:03.261058+0000","last_peered":"2024-04-22T00:00:03.261058+0000","last_clean":"2024-04-22T00:00:03.261058+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:00:03.261058+0000","last_undegraded":"2024-04-22T00:00:03.261058+0000","last_fullsized":"2024-04-22T00:00:03.261058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394550+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-21T23:59:41.394550+0000","last_peered":"2024-04-21T23:59:41.394550+0000","last_clean":"2024-04-21T23:59:41.394550+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-21T23:59:41.394550+0000","last_undegraded":"2024-04-21T23:59:41.394550+0000","last_fullsized":"2024-04-21T23:59:41.394550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896492+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-21T23:59:41.896492+0000","last_peered":"2024-04-21T23:59:41.896492+0000","last_clean":"2024-04-21T23:59:41.896492+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-21T23:59:41.896492+0000","last_undegraded":"2024-04-21T23:59:41.896492+0000","last_fullsized":"2024-04-21T23:59:41.896492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.246696+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:00:01.246696+0000","last_peered":"2024-04-22T00:00:01.246696+0000","last_clean":"2024-04-22T00:00:01.246696+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:00:01.246696+0000","last_undegraded":"2024-04-22T00:00:01.246696+0000","last_fullsized":"2024-04-22T00:00:01.246696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896574+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-21T23:59:41.896574+0000","last_peered":"2024-04-21T23:59:41.896574+0000","last_clean":"2024-04-21T23:59:41.896574+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-21T23:59:41.896574+0000","last_undegraded":"2024-04-21T23:59:41.896574+0000","last_fullsized":"2024-04-21T23:59:41.896574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394365+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-21T23:59:41.394365+0000","last_peered":"2024-04-21T23:59:41.394365+0000","last_clean":"2024-04-21T23:59:41.394365+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-21T23:59:41.394365+0000","last_undegraded":"2024-04-21T23:59:41.394365+0000","last_fullsized":"2024-04-21T23:59:41.394365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247136+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:00:01.247136+0000","last_peered":"2024-04-22T00:00:01.247136+0000","last_clean":"2024-04-22T00:00:01.247136+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:00:01.247136+0000","last_undegraded":"2024-04-22T00:00:01.247136+0000","last_fullsized":"2024-04-22T00:00:01.247136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897190+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-21T23:59:41.897190+0000","last_peered":"2024-04-21T23:59:41.897190+0000","last_clean":"2024-04-21T23:59:41.897190+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-21T23:59:41.897190+0000","last_undegraded":"2024-04-21T23:59:41.897190+0000","last_fullsized":"2024-04-21T23:59:41.897190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895825+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-21T23:59:41.895825+0000","last_peered":"2024-04-21T23:59:41.895825+0000","last_clean":"2024-04-21T23:59:41.895825+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-21T23:59:41.895825+0000","last_undegraded":"2024-04-21T23:59:41.895825+0000","last_fullsized":"2024-04-21T23:59:41.895825+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247311+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:00:01.247311+0000","last_peered":"2024-04-22T00:00:01.247311+0000","last_clean":"2024-04-22T00:00:01.247311+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:00:01.247311+0000","last_undegraded":"2024-04-22T00:00:01.247311+0000","last_fullsized":"2024-04-22T00:00:01.247311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896879+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-21T23:59:41.896879+0000","last_peered":"2024-04-21T23:59:41.896879+0000","last_clean":"2024-04-21T23:59:41.896879+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-21T23:59:41.896879+0000","last_undegraded":"2024-04-21T23:59:41.896879+0000","last_fullsized":"2024-04-21T23:59:41.896879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394542+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-21T23:59:41.394542+0000","last_peered":"2024-04-21T23:59:41.394542+0000","last_clean":"2024-04-21T23:59:41.394542+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-21T23:59:41.394542+0000","last_undegraded":"2024-04-21T23:59:41.394542+0000","last_fullsized":"2024-04-21T23:59:41.394542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.246903+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:00:01.246903+0000","last_peered":"2024-04-22T00:00:01.246903+0000","last_clean":"2024-04-22T00:00:01.246903+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:00:01.246903+0000","last_undegraded":"2024-04-22T00:00:01.246903+0000","last_fullsized":"2024-04-22T00:00:01.246903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896098+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-21T23:59:41.896098+0000","last_peered":"2024-04-21T23:59:41.896098+0000","last_clean":"2024-04-21T23:59:41.896098+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-21T23:59:41.896098+0000","last_undegraded":"2024-04-21T23:59:41.896098+0000","last_fullsized":"2024-04-21T23:59:41.896098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.819084+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-21T23:59:41.819084+0000","last_peered":"2024-04-21T23:59:41.819084+0000","last_clean":"2024-04-21T23:59:41.819084+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-21T23:59:41.819084+0000","last_undegraded":"2024-04-21T23:59:41.819084+0000","last_fullsized":"2024-04-21T23:59:41.819084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.254143+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:00:02.254143+0000","last_peered":"2024-04-22T00:00:02.254143+0000","last_clean":"2024-04-22T00:00:02.254143+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:00:02.254143+0000","last_undegraded":"2024-04-22T00:00:02.254143+0000","last_fullsized":"2024-04-22T00:00:02.254143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895993+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-21T23:59:41.895993+0000","last_peered":"2024-04-21T23:59:41.895993+0000","last_clean":"2024-04-21T23:59:41.895993+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-21T23:59:41.895993+0000","last_undegraded":"2024-04-21T23:59:41.895993+0000","last_fullsized":"2024-04-21T23:59:41.895993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896076+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-21T23:59:41.896076+0000","last_peered":"2024-04-21T23:59:41.896076+0000","last_clean":"2024-04-21T23:59:41.896076+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-21T23:59:41.896076+0000","last_undegraded":"2024-04-21T23:59:41.896076+0000","last_fullsized":"2024-04-21T23:59:41.896076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.260652+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:00:03.260652+0000","last_peered":"2024-04-22T00:00:03.260652+0000","last_clean":"2024-04-22T00:00:03.260652+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:00:03.260652+0000","last_undegraded":"2024-04-22T00:00:03.260652+0000","last_fullsized":"2024-04-22T00:00:03.260652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896250+0000","last_change":"2024-04-21T23:59:36.254670+0000","last_active":"2024-04-21T23:59:41.896250+0000","last_peered":"2024-04-21T23:59:41.896250+0000","last_clean":"2024-04-21T23:59:41.896250+0000","last_became_active":"2024-04-21T23:59:36.254368+0000","last_became_peered":"2024-04-21T23:59:36.254368+0000","last_unstale":"2024-04-21T23:59:41.896250+0000","last_undegraded":"2024-04-21T23:59:41.896250+0000","last_fullsized":"2024-04-21T23:59:41.896250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:48:20.985922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.395145+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-21T23:59:41.395145+0000","last_peered":"2024-04-21T23:59:41.395145+0000","last_clean":"2024-04-21T23:59:41.395145+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-21T23:59:41.395145+0000","last_undegraded":"2024-04-21T23:59:41.395145+0000","last_fullsized":"2024-04-21T23:59:41.395145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895256+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-21T23:59:41.895256+0000","last_peered":"2024-04-21T23:59:41.895256+0000","last_clean":"2024-04-21T23:59:41.895256+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-21T23:59:41.895256+0000","last_undegraded":"2024-04-21T23:59:41.895256+0000","last_fullsized":"2024-04-21T23:59:41.895256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895686+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-21T23:59:41.895686+0000","last_peered":"2024-04-21T23:59:41.895686+0000","last_clean":"2024-04-21T23:59:41.895686+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-21T23:59:41.895686+0000","last_undegraded":"2024-04-21T23:59:41.895686+0000","last_fullsized":"2024-04-21T23:59:41.895686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896983+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-21T23:59:41.896983+0000","last_peered":"2024-04-21T23:59:41.896983+0000","last_clean":"2024-04-21T23:59:41.896983+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-21T23:59:41.896983+0000","last_undegraded":"2024-04-21T23:59:41.896983+0000","last_fullsized":"2024-04-21T23:59:41.896983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.256345+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:00:02.256345+0000","last_peered":"2024-04-22T00:00:02.256345+0000","last_clean":"2024-04-22T00:00:02.256345+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:00:02.256345+0000","last_undegraded":"2024-04-22T00:00:02.256345+0000","last_fullsized":"2024-04-22T00:00:02.256345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818293+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-21T23:59:41.818293+0000","last_peered":"2024-04-21T23:59:41.818293+0000","last_clean":"2024-04-21T23:59:41.818293+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-21T23:59:41.818293+0000","last_undegraded":"2024-04-21T23:59:41.818293+0000","last_fullsized":"2024-04-21T23:59:41.818293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818419+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-21T23:59:41.818419+0000","last_peered":"2024-04-21T23:59:41.818419+0000","last_clean":"2024-04-21T23:59:41.818419+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-21T23:59:41.818419+0000","last_undegraded":"2024-04-21T23:59:41.818419+0000","last_fullsized":"2024-04-21T23:59:41.818419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896336+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-21T23:59:41.896336+0000","last_peered":"2024-04-21T23:59:41.896336+0000","last_clean":"2024-04-21T23:59:41.896336+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-21T23:59:41.896336+0000","last_undegraded":"2024-04-21T23:59:41.896336+0000","last_fullsized":"2024-04-21T23:59:41.896336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896047+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-21T23:59:41.896047+0000","last_peered":"2024-04-21T23:59:41.896047+0000","last_clean":"2024-04-21T23:59:41.896047+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-21T23:59:41.896047+0000","last_undegraded":"2024-04-21T23:59:41.896047+0000","last_fullsized":"2024-04-21T23:59:41.896047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895960+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-21T23:59:41.895960+0000","last_peered":"2024-04-21T23:59:41.895960+0000","last_clean":"2024-04-21T23:59:41.895960+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-21T23:59:41.895960+0000","last_undegraded":"2024-04-21T23:59:41.895960+0000","last_fullsized":"2024-04-21T23:59:41.895960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394866+0000","last_change":"2024-04-21T23:59:35.616463+0000","last_active":"2024-04-21T23:59:41.394866+0000","last_peered":"2024-04-21T23:59:41.394866+0000","last_clean":"2024-04-21T23:59:41.394866+0000","last_became_active":"2024-04-21T23:59:35.616158+0000","last_became_peered":"2024-04-21T23:59:35.616158+0000","last_unstale":"2024-04-21T23:59:41.394866+0000","last_undegraded":"2024-04-21T23:59:41.394866+0000","last_fullsized":"2024-04-21T23:59:41.394866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:17:30.736734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.398071+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-21T23:59:41.398071+0000","last_peered":"2024-04-21T23:59:41.398071+0000","last_clean":"2024-04-21T23:59:41.398071+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-21T23:59:41.398071+0000","last_undegraded":"2024-04-21T23:59:41.398071+0000","last_fullsized":"2024-04-21T23:59:41.398071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896802+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-21T23:59:41.896802+0000","last_peered":"2024-04-21T23:59:41.896802+0000","last_clean":"2024-04-21T23:59:41.896802+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-21T23:59:41.896802+0000","last_undegraded":"2024-04-21T23:59:41.896802+0000","last_fullsized":"2024-04-21T23:59:41.896802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896766+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-21T23:59:41.896766+0000","last_peered":"2024-04-21T23:59:41.896766+0000","last_clean":"2024-04-21T23:59:41.896766+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-21T23:59:41.896766+0000","last_undegraded":"2024-04-21T23:59:41.896766+0000","last_fullsized":"2024-04-21T23:59:41.896766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.397687+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-21T23:59:41.397687+0000","last_peered":"2024-04-21T23:59:41.397687+0000","last_clean":"2024-04-21T23:59:41.397687+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-21T23:59:41.397687+0000","last_undegraded":"2024-04-21T23:59:41.397687+0000","last_fullsized":"2024-04-21T23:59:41.397687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896455+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-21T23:59:41.896455+0000","last_peered":"2024-04-21T23:59:41.896455+0000","last_clean":"2024-04-21T23:59:41.896455+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-21T23:59:41.896455+0000","last_undegraded":"2024-04-21T23:59:41.896455+0000","last_fullsized":"2024-04-21T23:59:41.896455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818338+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-21T23:59:41.818338+0000","last_peered":"2024-04-21T23:59:41.818338+0000","last_clean":"2024-04-21T23:59:41.818338+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-21T23:59:41.818338+0000","last_undegraded":"2024-04-21T23:59:41.818338+0000","last_fullsized":"2024-04-21T23:59:41.818338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895484+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-21T23:59:41.895484+0000","last_peered":"2024-04-21T23:59:41.895484+0000","last_clean":"2024-04-21T23:59:41.895484+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-21T23:59:41.895484+0000","last_undegraded":"2024-04-21T23:59:41.895484+0000","last_fullsized":"2024-04-21T23:59:41.895484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896210+0000","last_change":"2024-04-21T23:59:40.412963+0000","last_active":"2024-04-21T23:59:41.896210+0000","last_peered":"2024-04-21T23:59:41.896210+0000","last_clean":"2024-04-21T23:59:41.896210+0000","last_became_active":"2024-04-21T23:59:40.412568+0000","last_became_peered":"2024-04-21T23:59:40.412568+0000","last_unstale":"2024-04-21T23:59:41.896210+0000","last_undegraded":"2024-04-21T23:59:41.896210+0000","last_fullsized":"2024-04-21T23:59:41.896210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:45.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895180+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-21T23:59:41.895180+0000","last_peered":"2024-04-21T23:59:41.895180+0000","last_clean":"2024-04-21T23:59:41.895180+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-21T23:59:41.895180+0000","last_undegraded":"2024-04-21T23:59:41.895180+0000","last_fullsized":"2024-04-21T23:59:41.895180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895662+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-21T23:59:41.895662+0000","last_peered":"2024-04-21T23:59:41.895662+0000","last_clean":"2024-04-21T23:59:41.895662+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-21T23:59:41.895662+0000","last_undegraded":"2024-04-21T23:59:41.895662+0000","last_fullsized":"2024-04-21T23:59:41.895662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895495+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-21T23:59:41.895495+0000","last_peered":"2024-04-21T23:59:41.895495+0000","last_clean":"2024-04-21T23:59:41.895495+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-21T23:59:41.895495+0000","last_undegraded":"2024-04-21T23:59:41.895495+0000","last_fullsized":"2024-04-21T23:59:41.895495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818854+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-21T23:59:41.818854+0000","last_peered":"2024-04-21T23:59:41.818854+0000","last_clean":"2024-04-21T23:59:41.818854+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-21T23:59:41.818854+0000","last_undegraded":"2024-04-21T23:59:41.818854+0000","last_fullsized":"2024-04-21T23:59:41.818854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895341+0000","last_change":"2024-04-21T23:59:36.255179+0000","last_active":"2024-04-21T23:59:41.895341+0000","last_peered":"2024-04-21T23:59:41.895341+0000","last_clean":"2024-04-21T23:59:41.895341+0000","last_became_active":"2024-04-21T23:59:36.254858+0000","last_became_peered":"2024-04-21T23:59:36.254858+0000","last_unstale":"2024-04-21T23:59:41.895341+0000","last_undegraded":"2024-04-21T23:59:41.895341+0000","last_fullsized":"2024-04-21T23:59:41.895341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:36:37.308326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.253562+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:00:02.253562+0000","last_peered":"2024-04-22T00:00:02.253562+0000","last_clean":"2024-04-22T00:00:02.253562+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:00:02.253562+0000","last_undegraded":"2024-04-22T00:00:02.253562+0000","last_fullsized":"2024-04-22T00:00:02.253562+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895883+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-21T23:59:41.895883+0000","last_peered":"2024-04-21T23:59:41.895883+0000","last_clean":"2024-04-21T23:59:41.895883+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-21T23:59:41.895883+0000","last_undegraded":"2024-04-21T23:59:41.895883+0000","last_fullsized":"2024-04-21T23:59:41.895883+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896197+0000","last_change":"2024-04-21T23:59:35.613025+0000","last_active":"2024-04-21T23:59:41.896197+0000","last_peered":"2024-04-21T23:59:41.896197+0000","last_clean":"2024-04-21T23:59:41.896197+0000","last_became_active":"2024-04-21T23:59:35.612725+0000","last_became_peered":"2024-04-21T23:59:35.612725+0000","last_unstale":"2024-04-21T23:59:41.896197+0000","last_undegraded":"2024-04-21T23:59:41.896197+0000","last_fullsized":"2024-04-21T23:59:41.896197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:31:17.866391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896972+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-21T23:59:41.896972+0000","last_peered":"2024-04-21T23:59:41.896972+0000","last_clean":"2024-04-21T23:59:41.896972+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-21T23:59:41.896972+0000","last_undegraded":"2024-04-21T23:59:41.896972+0000","last_fullsized":"2024-04-21T23:59:41.896972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896020+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-21T23:59:41.896020+0000","last_peered":"2024-04-21T23:59:41.896020+0000","last_clean":"2024-04-21T23:59:41.896020+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-21T23:59:41.896020+0000","last_undegraded":"2024-04-21T23:59:41.896020+0000","last_fullsized":"2024-04-21T23:59:41.896020+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.263190+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:00:03.263190+0000","last_peered":"2024-04-22T00:00:03.263190+0000","last_clean":"2024-04-22T00:00:03.263190+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:00:03.263190+0000","last_undegraded":"2024-04-22T00:00:03.263190+0000","last_fullsized":"2024-04-22T00:00:03.263190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896086+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-21T23:59:41.896086+0000","last_peered":"2024-04-21T23:59:41.896086+0000","last_clean":"2024-04-21T23:59:41.896086+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-21T23:59:41.896086+0000","last_undegraded":"2024-04-21T23:59:41.896086+0000","last_fullsized":"2024-04-21T23:59:41.896086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393432+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-21T23:59:41.393432+0000","last_peered":"2024-04-21T23:59:41.393432+0000","last_clean":"2024-04-21T23:59:41.393432+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-21T23:59:41.393432+0000","last_undegraded":"2024-04-21T23:59:41.393432+0000","last_fullsized":"2024-04-21T23:59:41.393432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895471+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-21T23:59:41.895471+0000","last_peered":"2024-04-21T23:59:41.895471+0000","last_clean":"2024-04-21T23:59:41.895471+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-21T23:59:41.895471+0000","last_undegraded":"2024-04-21T23:59:41.895471+0000","last_fullsized":"2024-04-21T23:59:41.895471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895735+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-21T23:59:41.895735+0000","last_peered":"2024-04-21T23:59:41.895735+0000","last_clean":"2024-04-21T23:59:41.895735+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-21T23:59:41.895735+0000","last_undegraded":"2024-04-21T23:59:41.895735+0000","last_fullsized":"2024-04-21T23:59:41.895735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895404+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-21T23:59:41.895404+0000","last_peered":"2024-04-21T23:59:41.895404+0000","last_clean":"2024-04-21T23:59:41.895404+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-21T23:59:41.895404+0000","last_undegraded":"2024-04-21T23:59:41.895404+0000","last_fullsized":"2024-04-21T23:59:41.895404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247976+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:00:01.247976+0000","last_peered":"2024-04-22T00:00:01.247976+0000","last_clean":"2024-04-22T00:00:01.247976+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:00:01.247976+0000","last_undegraded":"2024-04-22T00:00:01.247976+0000","last_fullsized":"2024-04-22T00:00:01.247976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895372+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-21T23:59:41.895372+0000","last_peered":"2024-04-21T23:59:41.895372+0000","last_clean":"2024-04-21T23:59:41.895372+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-21T23:59:41.895372+0000","last_undegraded":"2024-04-21T23:59:41.895372+0000","last_fullsized":"2024-04-21T23:59:41.895372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394992+0000","last_change":"2024-04-21T23:59:40.419040+0000","last_active":"2024-04-21T23:59:41.394992+0000","last_peered":"2024-04-21T23:59:41.394992+0000","last_clean":"2024-04-21T23:59:41.394992+0000","last_became_active":"2024-04-21T23:59:40.418112+0000","last_became_peered":"2024-04-21T23:59:40.418112+0000","last_unstale":"2024-04-21T23:59:41.394992+0000","last_undegraded":"2024-04-21T23:59:41.394992+0000","last_fullsized":"2024-04-21T23:59:41.394992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:31:18.156358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:04.273972+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:00:04.273972+0000","last_peered":"2024-04-22T00:00:04.273972+0000","last_clean":"2024-04-22T00:00:04.273972+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:00:04.273972+0000","last_undegraded":"2024-04-22T00:00:04.273972+0000","last_fullsized":"2024-04-22T00:00:04.273972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895409+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-21T23:59:41.895409+0000","last_peered":"2024-04-21T23:59:41.895409+0000","last_clean":"2024-04-21T23:59:41.895409+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-21T23:59:41.895409+0000","last_undegraded":"2024-04-21T23:59:41.895409+0000","last_fullsized":"2024-04-21T23:59:41.895409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896145+0000","last_change":"2024-04-21T23:59:40.409609+0000","last_active":"2024-04-21T23:59:41.896145+0000","last_peered":"2024-04-21T23:59:41.896145+0000","last_clean":"2024-04-21T23:59:41.896145+0000","last_became_active":"2024-04-21T23:59:40.408951+0000","last_became_peered":"2024-04-21T23:59:40.408951+0000","last_unstale":"2024-04-21T23:59:41.896145+0000","last_undegraded":"2024-04-21T23:59:41.896145+0000","last_fullsized":"2024-04-21T23:59:41.896145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:51:53.973363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:02.242426+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:00:02.242426+0000","last_peered":"2024-04-22T00:00:02.242426+0000","last_clean":"2024-04-22T00:00:02.242426+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:00:02.242426+0000","last_undegraded":"2024-04-22T00:00:02.242426+0000","last_fullsized":"2024-04-22T00:00:02.242426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895822+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-21T23:59:41.895822+0000","last_peered":"2024-04-21T23:59:41.895822+0000","last_clean":"2024-04-21T23:59:41.895822+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-21T23:59:41.895822+0000","last_undegraded":"2024-04-21T23:59:41.895822+0000","last_fullsized":"2024-04-21T23:59:41.895822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895581+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-21T23:59:41.895581+0000","last_peered":"2024-04-21T23:59:41.895581+0000","last_clean":"2024-04-21T23:59:41.895581+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-21T23:59:41.895581+0000","last_undegraded":"2024-04-21T23:59:41.895581+0000","last_fullsized":"2024-04-21T23:59:41.895581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:01.247815+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:00:01.247815+0000","last_peered":"2024-04-22T00:00:01.247815+0000","last_clean":"2024-04-22T00:00:01.247815+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:00:01.247815+0000","last_undegraded":"2024-04-22T00:00:01.247815+0000","last_fullsized":"2024-04-22T00:00:01.247815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896751+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-21T23:59:41.896751+0000","last_peered":"2024-04-21T23:59:41.896751+0000","last_clean":"2024-04-21T23:59:41.896751+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-21T23:59:41.896751+0000","last_undegraded":"2024-04-21T23:59:41.896751+0000","last_fullsized":"2024-04-21T23:59:41.896751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896283+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-21T23:59:41.896283+0000","last_peered":"2024-04-21T23:59:41.896283+0000","last_clean":"2024-04-21T23:59:41.896283+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-21T23:59:41.896283+0000","last_undegraded":"2024-04-21T23:59:41.896283+0000","last_fullsized":"2024-04-21T23:59:41.896283+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.259395+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:00:03.259395+0000","last_peered":"2024-04-22T00:00:03.259395+0000","last_clean":"2024-04-22T00:00:03.259395+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:00:03.259395+0000","last_undegraded":"2024-04-22T00:00:03.259395+0000","last_fullsized":"2024-04-22T00:00:03.259395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896498+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-21T23:59:41.896498+0000","last_peered":"2024-04-21T23:59:41.896498+0000","last_clean":"2024-04-21T23:59:41.896498+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-21T23:59:41.896498+0000","last_undegraded":"2024-04-21T23:59:41.896498+0000","last_fullsized":"2024-04-21T23:59:41.896498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895624+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-21T23:59:41.895624+0000","last_peered":"2024-04-21T23:59:41.895624+0000","last_clean":"2024-04-21T23:59:41.895624+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-21T23:59:41.895624+0000","last_undegraded":"2024-04-21T23:59:41.895624+0000","last_fullsized":"2024-04-21T23:59:41.895624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T00:00:03.260716+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:00:03.260716+0000","last_peered":"2024-04-22T00:00:03.260716+0000","last_clean":"2024-04-22T00:00:03.260716+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:00:03.260716+0000","last_undegraded":"2024-04-22T00:00:03.260716+0000","last_fullsized":"2024-04-22T00:00:03.260716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896587+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-21T23:59:41.896587+0000","last_peered":"2024-04-21T23:59:41.896587+0000","last_clean":"2024-04-21T23:59:41.896587+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-21T23:59:41.896587+0000","last_undegraded":"2024-04-21T23:59:41.896587+0000","last_fullsized":"2024-04-21T23:59:41.896587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818155+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-21T23:59:41.818155+0000","last_peered":"2024-04-21T23:59:41.818155+0000","last_clean":"2024-04-21T23:59:41.818155+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-21T23:59:41.818155+0000","last_undegraded":"2024-04-21T23:59:41.818155+0000","last_fullsized":"2024-04-21T23:59:41.818155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896811+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-21T23:59:41.896811+0000","last_peered":"2024-04-21T23:59:41.896811+0000","last_clean":"2024-04-21T23:59:41.896811+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-21T23:59:41.896811+0000","last_undegraded":"2024-04-21T23:59:41.896811+0000","last_fullsized":"2024-04-21T23:59:41.896811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896536+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-21T23:59:41.896536+0000","last_peered":"2024-04-21T23:59:41.896536+0000","last_clean":"2024-04-21T23:59:41.896536+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-21T23:59:41.896536+0000","last_undegraded":"2024-04-21T23:59:41.896536+0000","last_fullsized":"2024-04-21T23:59:41.896536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895498+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-21T23:59:41.895498+0000","last_peered":"2024-04-21T23:59:41.895498+0000","last_clean":"2024-04-21T23:59:41.895498+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-21T23:59:41.895498+0000","last_undegraded":"2024-04-21T23:59:41.895498+0000","last_fullsized":"2024-04-21T23:59:41.895498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393669+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-21T23:59:41.393669+0000","last_peered":"2024-04-21T23:59:41.393669+0000","last_clean":"2024-04-21T23:59:41.393669+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-21T23:59:41.393669+0000","last_undegraded":"2024-04-21T23:59:41.393669+0000","last_fullsized":"2024-04-21T23:59:41.393669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.393303+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-21T23:59:41.393303+0000","last_peered":"2024-04-21T23:59:41.393303+0000","last_clean":"2024-04-21T23:59:41.393303+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-21T23:59:41.393303+0000","last_undegraded":"2024-04-21T23:59:41.393303+0000","last_fullsized":"2024-04-21T23:59:41.393303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.895919+0000","last_change":"2024-04-21T23:59:36.249178+0000","last_active":"2024-04-21T23:59:41.895919+0000","last_peered":"2024-04-21T23:59:41.895919+0000","last_clean":"2024-04-21T23:59:41.895919+0000","last_became_active":"2024-04-21T23:59:36.248728+0000","last_became_peered":"2024-04-21T23:59:36.248728+0000","last_unstale":"2024-04-21T23:59:41.895919+0000","last_undegraded":"2024-04-21T23:59:41.895919+0000","last_fullsized":"2024-04-21T23:59:41.895919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:37:15.983221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.897066+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-21T23:59:41.897066+0000","last_peered":"2024-04-21T23:59:41.897066+0000","last_clean":"2024-04-21T23:59:41.897066+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-21T23:59:41.897066+0000","last_undegraded":"2024-04-21T23:59:41.897066+0000","last_fullsized":"2024-04-21T23:59:41.897066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896097+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-21T23:59:41.896097+0000","last_peered":"2024-04-21T23:59:41.896097+0000","last_clean":"2024-04-21T23:59:41.896097+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-21T23:59:41.896097+0000","last_undegraded":"2024-04-21T23:59:41.896097+0000","last_fullsized":"2024-04-21T23:59:41.896097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.394083+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-21T23:59:41.394083+0000","last_peered":"2024-04-21T23:59:41.394083+0000","last_clean":"2024-04-21T23:59:41.394083+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-21T23:59:41.394083+0000","last_undegraded":"2024-04-21T23:59:41.394083+0000","last_fullsized":"2024-04-21T23:59:41.394083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896359+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-21T23:59:41.896359+0000","last_peered":"2024-04-21T23:59:41.896359+0000","last_clean":"2024-04-21T23:59:41.896359+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-21T23:59:41.896359+0000","last_undegraded":"2024-04-21T23:59:41.896359+0000","last_fullsized":"2024-04-21T23:59:41.896359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.818285+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-21T23:59:41.818285+0000","last_peered":"2024-04-21T23:59:41.818285+0000","last_clean":"2024-04-21T23:59:41.818285+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-21T23:59:41.818285+0000","last_undegraded":"2024-04-21T23:59:41.818285+0000","last_fullsized":"2024-04-21T23:59:41.818285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896247+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-21T23:59:41.896247+0000","last_peered":"2024-04-21T23:59:41.896247+0000","last_clean":"2024-04-21T23:59:41.896247+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-21T23:59:41.896247+0000","last_undegraded":"2024-04-21T23:59:41.896247+0000","last_fullsized":"2024-04-21T23:59:41.896247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896525+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-21T23:59:41.896525+0000","last_peered":"2024-04-21T23:59:41.896525+0000","last_clean":"2024-04-21T23:59:41.896525+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-21T23:59:41.896525+0000","last_undegraded":"2024-04-21T23:59:41.896525+0000","last_fullsized":"2024-04-21T23:59:41.896525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896477+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-21T23:59:41.896477+0000","last_peered":"2024-04-21T23:59:41.896477+0000","last_clean":"2024-04-21T23:59:41.896477+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-21T23:59:41.896477+0000","last_undegraded":"2024-04-21T23:59:41.896477+0000","last_fullsized":"2024-04-21T23:59:41.896477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T23:59:41.896049+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-21T23:59:41.896049+0000","last_peered":"2024-04-21T23:59:41.896049+0000","last_clean":"2024-04-21T23:59:41.896049+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-21T23:59:41.896049+0000","last_undegraded":"2024-04-21T23:59:41.896049+0000","last_fullsized":"2024-04-21T23:59:41.896049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8124,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":134,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":43711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1069375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":483029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32756,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712396,"statfs":{"total":95995035648,"available":95961493504,"internally_reserved":0,"allocated":1617920,"data_stored":1069821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:00:07.259 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T00:00:07.259 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T00:00:07.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T00:00:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:07 smithi203 ceph-mon[31431]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-22T00:00:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:07 smithi203 ceph-mon[31431]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:07 smithi203 ceph-mon[31431]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T00:00:07.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1639896207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:07.541 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:07.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:07 smithi111 ceph-mon[31543]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-22T00:00:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:07 smithi111 ceph-mon[31543]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:07 smithi111 ceph-mon[31543]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T00:00:07.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1639896207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:07.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:07 smithi038 ceph-mon[27972]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-22T00:00:07.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:07 smithi038 ceph-mon[27972]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:00:07.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:07 smithi038 ceph-mon[27972]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-04-22T00:00:07.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1639896207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:08.550 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:08 smithi038 ceph-mon[27972]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:08.550 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:08 smithi038 ceph-mon[27972]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:08.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:08 smithi111 ceph-mon[31543]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:08.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:08 smithi111 ceph-mon[31543]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:08.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:08 smithi203 ceph-mon[31431]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:08.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:08 smithi203 ceph-mon[31431]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:00:09.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 -- 172.21.15.38:0/1890892232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1023b0 msgr2=0x7f103c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:09.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 --2- 172.21.15.38:0/1890892232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1023b0 0x7f103c102810 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f10300099c0 tx=0x7f1030009cd0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 -- 172.21.15.38:0/1890892232 shutdown_connections 2024-04-22T00:00:09.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 --2- 172.21.15.38:0/1890892232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f103c102d50 0x7f103c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 --2- 172.21.15.38:0/1890892232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c1011b0 0x7f103c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 --2- 172.21.15.38:0/1890892232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1023b0 0x7f103c102810 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 -- 172.21.15.38:0/1890892232 >> 172.21.15.38:0/1890892232 conn(0x7f103c0fc750 msgr2=0x7f103c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 -- 172.21.15.38:0/1890892232 shutdown_connections 2024-04-22T00:00:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.447+0000 7f1042104700 1 -- 172.21.15.38:0/1890892232 wait complete. 2024-04-22T00:00:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 Processor -- start 2024-04-22T00:00:09.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:09 smithi038 ceph-mon[27972]: pgmap v310: 129 pgs: 129 active+clean; 627 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 -- start start 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f103c1023b0 0x7f103c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 0x7f103c19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f103c10cd80 con 0x7f103c1011b0 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f103c10cc00 con 0x7f103c102d50 2024-04-22T00:00:09.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.448+0000 7f1042104700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f103c10cf00 con 0x7f103c1023b0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40622/0 (socket says 172.21.15.38:40622) 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 -- 172.21.15.38:0/4047549208 learned_addr learned my addr 172.21.15.38:0/4047549208 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103bfff700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 0x7f103c19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 -- 172.21.15.38:0/4047549208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f103c1023b0 msgr2=0x7f103c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f103c1023b0 0x7f103c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 -- 172.21.15.38:0/4047549208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 msgr2=0x7f103c19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 0x7f103c19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 -- 172.21.15.38:0/4047549208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c1a00e0 con 0x7f103c1011b0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103bfff700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 0x7f103c19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f103b7fe700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d9b0 tx=0x7f102c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c017030 con 0x7f103c1011b0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.449+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f102c004500 con 0x7f103c1011b0 2024-04-22T00:00:09.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.450+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103c1a03d0 con 0x7f103c1011b0 2024-04-22T00:00:09.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.450+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f102c003aa0 con 0x7f103c1011b0 2024-04-22T00:00:09.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.450+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f103c1a0960 con 0x7f103c1011b0 2024-04-22T00:00:09.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.451+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f103c0507a0 con 0x7f103c1011b0 2024-04-22T00:00:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.452+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f102c01a070 con 0x7f103c1011b0 2024-04-22T00:00:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.452+0000 7f1038ff9700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 0x7f10240a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.453+0000 7f103affd700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 0x7f10240a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.453+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f102c0cedb0 con 0x7f103c1011b0 2024-04-22T00:00:09.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.453+0000 7f103affd700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 0x7f10240a52a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10300098e0 tx=0x7f1030000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:09.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.457+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f102c09b970 con 0x7f103c1011b0 2024-04-22T00:00:09.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:09 smithi111 ceph-mon[31543]: pgmap v310: 129 pgs: 129 active+clean; 627 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-22T00:00:09.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.749+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f103c1a0e30 con 0x7f103c1011b0 2024-04-22T00:00:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.752+0000 7f1038ff9700 1 -- 172.21.15.38:0/4047549208 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f102c09b270 con 0x7f103c1011b0 2024-04-22T00:00:09.745 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-04-22T00:00:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCJqCVmaJ64LBAA/1fPVZhfdb6Puk3BBom0nw== 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 msgr2=0x7f10240a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 0x7f10240a52a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10300098e0 tx=0x7f1030000bc0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 msgr2=0x7f103c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d9b0 tx=0x7f102c00dd70 comp rx=0 tx=0).stop 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 shutdown_connections 2024-04-22T00:00:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f10240a2de0 0x7f10240a52a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f103c102d50 0x7f103c19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f103c1011b0 0x7f103c198480 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 --2- 172.21.15.38:0/4047549208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f103c1023b0 0x7f103c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.754+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 >> 172.21.15.38:0/4047549208 conn(0x7f103c0fc750 msgr2=0x7f103c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.755+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 shutdown_connections 2024-04-22T00:00:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:09.755+0000 7f1042104700 1 -- 172.21.15.38:0/4047549208 wait complete. 2024-04-22T00:00:09.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:09 smithi203 ceph-mon[31431]: pgmap v310: 129 pgs: 129 active+clean; 627 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-22T00:00:10.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:10 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T00:00:10.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:10 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T00:00:10.535 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:00:10.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T00:00:10.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T00:00:10.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:10 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T00:00:10.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:10 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T00:00:10.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T00:00:10.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T00:00:10.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T00:00:10.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:10 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T00:00:10.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:10 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4047549208' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T00:00:10.873 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:11.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:11 smithi038 ceph-mon[27972]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T00:00:11.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:11 smithi111 ceph-mon[31543]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T00:00:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:11 smithi203 ceph-mon[31431]: pgmap v311: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-04-22T00:00:12.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 -- 172.21.15.38:0/3654348771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 msgr2=0x7f20e41028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/3654348771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e41028c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009990 tx=0x7f20d4009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 -- 172.21.15.38:0/3654348771 shutdown_connections 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/3654348771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4107570 0x7f20e410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/3654348771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e41028c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/3654348771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4102e90 0x7f20e4106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 -- 172.21.15.38:0/3654348771 >> 172.21.15.38:0/3654348771 conn(0x7f20e4079390 msgr2=0x7f20e40797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 -- 172.21.15.38:0/3654348771 shutdown_connections 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.969+0000 7f20ea4a6700 1 -- 172.21.15.38:0/3654348771 wait complete. 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 Processor -- start 2024-04-22T00:00:12.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 -- start start 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4102e90 0x7f20e419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4107570 0x7f20e4198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f20e41125b0 con 0x7f20e41024e0 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20e4112430 con 0x7f20e4102e90 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.970+0000 7f20ea4a6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20e4112730 con 0x7f20e4107570 2024-04-22T00:00:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40632/0 (socket says 172.21.15.38:40632) 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 -- 172.21.15.38:0/933712026 learned_addr learned my addr 172.21.15.38:0/933712026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e37fe700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4102e90 0x7f20e419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e8a43700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4107570 0x7f20e4198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 -- 172.21.15.38:0/933712026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4107570 msgr2=0x7f20e4198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4107570 0x7f20e4198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 -- 172.21.15.38:0/933712026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4102e90 msgr2=0x7f20e419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4102e90 0x7f20e419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 -- 172.21.15.38:0/933712026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e4198ff0 con 0x7f20e41024e0 2024-04-22T00:00:12.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e3fff700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f20d40038b0 tx=0x7f20d4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4027070 con 0x7f20e41024e0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e4199280 con 0x7f20e41024e0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d4004050 con 0x7f20e41024e0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.971+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e41a55a0 con 0x7f20e41024e0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.972+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4022ba0 con 0x7f20e41024e0 2024-04-22T00:00:12.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.973+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e410a790 con 0x7f20e41024e0 2024-04-22T00:00:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.974+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f20d402a070 con 0x7f20e41024e0 2024-04-22T00:00:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.975+0000 7f20e17fa700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 0x7f20cc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.975+0000 7f20e37fe700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 0x7f20cc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.975+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f20d40d04f0 con 0x7f20e41024e0 2024-04-22T00:00:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.975+0000 7f20e37fe700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 0x7f20cc0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20e4101e50 tx=0x7f20d8009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:12.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:12.978+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f20d4022e20 con 0x7f20e41024e0 2024-04-22T00:00:13.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.243+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f20e4199dc0 con 0x7f20e41024e0 2024-04-22T00:00:13.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.248+0000 7f20e17fa700 1 -- 172.21.15.38:0/933712026 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7f20d4013070 con 0x7f20e41024e0 2024-04-22T00:00:13.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.251+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 msgr2=0x7f20cc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.251+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 0x7f20cc0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20e4101e50 tx=0x7f20d8009190 comp rx=0 tx=0).stop 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.251+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 msgr2=0x7f20e419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.251+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f20d40038b0 tx=0x7f20d4009f50 comp rx=0 tx=0).stop 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 shutdown_connections 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20cc0a2cb0 0x7f20cc0a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20e4107570 0x7f20e4198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e41024e0 0x7f20e419e6d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 --2- 172.21.15.38:0/933712026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20e4102e90 0x7f20e419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:13.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 >> 172.21.15.38:0/933712026 conn(0x7f20e4079390 msgr2=0x7f20e41086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:13.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 shutdown_connections 2024-04-22T00:00:13.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:13.252+0000 7f20ea4a6700 1 -- 172.21.15.38:0/933712026 wait complete. 2024-04-22T00:00:13.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:13 smithi111 ceph-mon[31543]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-04-22T00:00:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:13 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/933712026' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T00:00:13.621 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:13 smithi038 ceph-mon[27972]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-04-22T00:00:13.621 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:13 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/933712026' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T00:00:13.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:13 smithi203 ceph-mon[31431]: pgmap v312: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.4 KiB/s wr, 21 op/s 2024-04-22T00:00:13.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:13 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/933712026' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T00:00:14.048 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2340980806},{"type":"v1","addr":"172.21.15.38:6801","nonce":2340980806}]},"active_addr":"172.21.15.38:6801/2340980806","active_change":"2024-04-21T23:52:25.561452+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24118,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2899039909}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":914846331}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1468209311}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2416463439}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3700725283}]}]} 2024-04-22T00:00:14.054 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T00:00:14.054 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T00:00:14.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T00:00:14.313 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:15.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:15 smithi038 ceph-mon[27972]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.4 KiB/s wr, 23 op/s 2024-04-22T00:00:15.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:15 smithi111 ceph-mon[31543]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.4 KiB/s wr, 23 op/s 2024-04-22T00:00:15.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:15 smithi203 ceph-mon[31431]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 9.4 KiB/s wr, 23 op/s 2024-04-22T00:00:16.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 -- 172.21.15.38:0/1354631266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0107570 msgr2=0x7f84d010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/1354631266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0107570 0x7f84d010f140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f84c800b0a0 tx=0x7f84c800b3b0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 -- 172.21.15.38:0/1354631266 shutdown_connections 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/1354631266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0107570 0x7f84d010f140 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/1354631266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 0x7f84d01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/1354631266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0102e90 0x7f84d0106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 -- 172.21.15.38:0/1354631266 >> 172.21.15.38:0/1354631266 conn(0x7f84d0079390 msgr2=0x7f84d00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:16.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 -- 172.21.15.38:0/1354631266 shutdown_connections 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.119+0000 7f84d5fc6700 1 -- 172.21.15.38:0/1354631266 wait complete. 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.120+0000 7f84d5fc6700 1 Processor -- start 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.120+0000 7f84d5fc6700 1 -- start start 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.120+0000 7f84d5fc6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 0x7f84d019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84d5fc6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84d5fc6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 0x7f84d0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84d5fc6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f84d0112600 con 0x7f84d0102e90 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84d5fc6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f84d0112480 con 0x7f84d0107570 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84d5fc6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84d0112780 con 0x7f84d01024e0 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45678/0 (socket says 172.21.15.38:45678) 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 -- 172.21.15.38:0/3581495546 learned_addr learned my addr 172.21.15.38:0/3581495546 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:16.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84cffff700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 0x7f84d0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84cf7fe700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 0x7f84d019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 -- 172.21.15.38:0/3581495546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 msgr2=0x7f84d019e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 0x7f84d019e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 -- 172.21.15.38:0/3581495546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 msgr2=0x7f84d0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 0x7f84d0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 -- 172.21.15.38:0/3581495546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d0199080 con 0x7f84d0102e90 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84cffff700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 0x7f84d0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.121+0000 7f84ceffd700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f84c400d980 tx=0x7f84c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.122+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4017070 con 0x7f84d0102e90 2024-04-22T00:00:16.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.122+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c400fd70 con 0x7f84d0102e90 2024-04-22T00:00:16.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.122+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c400e3b0 con 0x7f84d0102e90 2024-04-22T00:00:16.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.122+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d0199370 con 0x7f84d0102e90 2024-04-22T00:00:16.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.122+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d01a55e0 con 0x7f84d0102e90 2024-04-22T00:00:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.123+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d0103970 con 0x7f84d0102e90 2024-04-22T00:00:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.124+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f84c400f890 con 0x7f84d0102e90 2024-04-22T00:00:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.125+0000 7f84ccff9700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 0x7f84b80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.125+0000 7f84cf7fe700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 0x7f84b80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.125+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84c40ceea0 con 0x7f84d0102e90 2024-04-22T00:00:16.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.125+0000 7f84cf7fe700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 0x7f84b80a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f84d0107140 tx=0x7f84c0009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:16.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.129+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f84c409ba60 con 0x7f84d0102e90 2024-04-22T00:00:16.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.336+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84d002db30 con 0x7f84d0102e90 2024-04-22T00:00:16.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.338+0000 7f84ccff9700 1 -- 172.21.15.38:0/3581495546 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7f84c409b360 con 0x7f84d0102e90 2024-04-22T00:00:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:16.331 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:00:16.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 msgr2=0x7f84b80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 0x7f84b80a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f84d0107140 tx=0x7f84c0009190 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 msgr2=0x7f84d019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f84c400d980 tx=0x7f84c400dc90 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 shutdown_connections 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f84b80a2cb0 0x7f84b80a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84d0107570 0x7f84d0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d01024e0 0x7f84d019e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 --2- 172.21.15.38:0/3581495546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f84d0102e90 0x7f84d019ec50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 >> 172.21.15.38:0/3581495546 conn(0x7f84d0079390 msgr2=0x7f84d0100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 shutdown_connections 2024-04-22T00:00:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:16.340+0000 7f84d5fc6700 1 -- 172.21.15.38:0/3581495546 wait complete. 2024-04-22T00:00:17.131 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T00:00:17.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T00:00:17.385 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:17 smithi038 ceph-mon[27972]: pgmap v314: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-04-22T00:00:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:17 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3581495546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:17.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:17 smithi111 ceph-mon[31543]: pgmap v314: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-04-22T00:00:17.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:17 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3581495546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:17.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:17 smithi203 ceph-mon[31431]: pgmap v314: 129 pgs: 129 active+clean; 636 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-04-22T00:00:17.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:17 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3581495546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:19.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:19 smithi038 ceph-mon[27972]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 -- 172.21.15.38:0/1505061059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9341082b0 msgr2=0x7fb934108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 --2- 172.21.15.38:0/1505061059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9341082b0 0x7fb934108690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb924009990 tx=0x7fb924009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 -- 172.21.15.38:0/1505061059 shutdown_connections 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 --2- 172.21.15.38:0/1505061059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb93406a8e0 0x7fb93410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 --2- 172.21.15.38:0/1505061059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9341082b0 0x7fb934108690 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 --2- 172.21.15.38:0/1505061059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934069f40 0x7fb93406a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 -- 172.21.15.38:0/1505061059 >> 172.21.15.38:0/1505061059 conn(0x7fb9340780e0 msgr2=0x7fb9340784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 -- 172.21.15.38:0/1505061059 shutdown_connections 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.371+0000 7fb938f3e700 1 -- 172.21.15.38:0/1505061059 wait complete. 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 Processor -- start 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 -- start start 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 0x7fb93419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9341082b0 0x7fb9341987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb9341124f0 con 0x7fb93406a8e0 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb934112370 con 0x7fb9341082b0 2024-04-22T00:00:19.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.372+0000 7fb938f3e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb934112670 con 0x7fb934069f40 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb931d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 0x7fb93419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:36834/0 (socket says 172.21.15.38:36834) 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 -- 172.21.15.38:0/160659872 learned_addr learned my addr 172.21.15.38:0/160659872 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb932d9d700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9341082b0 0x7fb9341987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 -- 172.21.15.38:0/160659872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9341082b0 msgr2=0x7fb9341987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:19.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9341082b0 0x7fb9341987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 -- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 msgr2=0x7fb93419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 0x7fb93419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 -- 172.21.15.38:0/160659872 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb934198fd0 con 0x7fb934069f40 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb931d9b700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 0x7fb93419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb93259c700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb92400bff0 tx=0x7fb924005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924027070 con 0x7fb934069f40 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb924004050 con 0x7fb934069f40 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.373+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb934199260 con 0x7fb934069f40 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.374+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924022d00 con 0x7fb934069f40 2024-04-22T00:00:19.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.374+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9341a5480 con 0x7fb934069f40 2024-04-22T00:00:19.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.374+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb934103960 con 0x7fb934069f40 2024-04-22T00:00:19.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.376+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb92402a070 con 0x7fb934069f40 2024-04-22T00:00:19.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.376+0000 7fb9237fe700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 0x7fb91c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:19.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.376+0000 7fb931d9b700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 0x7fb91c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:19.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.376+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb9240d0b10 con 0x7fb934069f40 2024-04-22T00:00:19.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.377+0000 7fb931d9b700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 0x7fb91c0a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb934078720 tx=0x7fb92800a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:19.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.380+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb924013070 con 0x7fb934069f40 2024-04-22T00:00:19.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:19 smithi111 ceph-mon[31543]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2024-04-22T00:00:19.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.587+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb93402db30 con 0x7fb934069f40 2024-04-22T00:00:19.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.590+0000 7fb9237fe700 1 -- 172.21.15.38:0/160659872 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24113 (secure 0 0 0) 0x7fb92409d700 con 0x7fb934069f40 2024-04-22T00:00:19.583 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:00:19.583 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-21T23:59:40.763796+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 msgr2=0x7fb91c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 0x7fb91c0a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb934078720 tx=0x7fb92800a380 comp rx=0 tx=0).stop 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 msgr2=0x7fb93419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb92400bff0 tx=0x7fb924005590 comp rx=0 tx=0).stop 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 shutdown_connections 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb91c0a2cb0 0x7fb91c0a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9341082b0 0x7fb9341987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb934069f40 0x7fb93419e620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 --2- 172.21.15.38:0/160659872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb93406a8e0 0x7fb93419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:19.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 >> 172.21.15.38:0/160659872 conn(0x7fb9340780e0 msgr2=0x7fb9340ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:19.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 shutdown_connections 2024-04-22T00:00:19.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:19.591+0000 7fb938f3e700 1 -- 172.21.15.38:0/160659872 wait complete. 2024-04-22T00:00:19.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:19 smithi203 ceph-mon[31431]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2024-04-22T00:00:20.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T00:00:20.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T00:00:20.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T00:00:20.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T00:00:20.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T00:00:20.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T00:00:20.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T00:00:20.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T00:00:20.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T00:00:20.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T00:00:20.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T00:00:20.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T00:00:20.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:20 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/160659872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:20.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/160659872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:20.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:20 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/160659872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:00:20.970 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.972 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.973 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.975 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.988 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.989 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.989 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.989 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:20.993 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:21.000 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:21.001 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:21.399 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:21 smithi038 ceph-mon[27972]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-22T00:00:21.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:21 smithi111 ceph-mon[31543]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-22T00:00:21.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:21 smithi203 ceph-mon[31431]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-22T00:00:23.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:23 smithi038 ceph-mon[27972]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T00:00:23.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:23 smithi203 ceph-mon[31431]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T00:00:23.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:23 smithi111 ceph-mon[31543]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-22T00:00:25.682 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:25 smithi038 ceph-mon[27972]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:00:25.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:25 smithi203 ceph-mon[31431]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:00:25.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:25 smithi111 ceph-mon[31543]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-22T00:00:26.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-22T00:00:26.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-22T00:00:26.377 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:26 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-22T00:00:26.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-22T00:00:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:26 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-22T00:00:26.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-22T00:00:26.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:26 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-22T00:00:27.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:00:27.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T00:00:27.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-22T00:00:27.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T00:00:27.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T00:00:27.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-22T00:00:27.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:27 smithi038 ceph-mon[27972]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T00:00:27.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:00:27.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-22T00:00:27.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:27 smithi203 ceph-mon[31431]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T00:00:27.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-22T00:00:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-22T00:00:27.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-22T00:00:27.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:27 smithi111 ceph-mon[31543]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 -- 172.21.15.38:0/3162481727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08541011c0 msgr2=0x7f085410f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 --2- 172.21.15.38:0/3162481727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08541011c0 0x7f085410f100 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f084c00b0a0 tx=0x7f084c00b3b0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 -- 172.21.15.38:0/3162481727 shutdown_connections 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 --2- 172.21.15.38:0/3162481727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08541011c0 0x7f085410f100 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 --2- 172.21.15.38:0/3162481727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 0x7f08541086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 --2- 172.21.15.38:0/3162481727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0854100820 0x7f0854100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.857+0000 7f0859ca1700 1 -- 172.21.15.38:0/3162481727 >> 172.21.15.38:0/3162481727 conn(0x7f08540777a0 msgr2=0x7f08540feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:27.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.858+0000 7f0859ca1700 1 -- 172.21.15.38:0/3162481727 shutdown_connections 2024-04-22T00:00:27.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.858+0000 7f0859ca1700 1 -- 172.21.15.38:0/3162481727 wait complete. 2024-04-22T00:00:27.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.858+0000 7f0859ca1700 1 Processor -- start 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.858+0000 7f0859ca1700 1 -- start start 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 0x7f08541a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08541011c0 0x7f08541a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 0x7f085419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08541125c0 con 0x7f0854100820 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0854112440 con 0x7f08541082c0 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0859ca1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0854112740 con 0x7f08541011c0 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 0x7f08541a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 0x7f08541a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52742/0 (socket says 172.21.15.38:52742) 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 -- 172.21.15.38:0/2525558801 learned_addr learned my addr 172.21.15.38:0/2525558801 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0853fff700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 0x7f085419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:27.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0852ffd700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08541011c0 0x7f08541a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 -- 172.21.15.38:0/2525558801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08541011c0 msgr2=0x7f08541a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08541011c0 0x7f08541a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 -- 172.21.15.38:0/2525558801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 msgr2=0x7f085419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 0x7f085419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085419b160 con 0x7f0854100820 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0852ffd700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08541011c0 0x7f08541a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f0853fff700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08541082c0 0x7f085419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.859+0000 7f08537fe700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 0x7f08541a0840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f084400c920 tx=0x7f084400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.860+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0844007b00 con 0x7f0854100820 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.860+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0844004d10 con 0x7f0854100820 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.860+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085419b450 con 0x7f0854100820 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.860+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084400f450 con 0x7f0854100820 2024-04-22T00:00:27.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.860+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08541a7710 con 0x7f0854100820 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.861+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f085410e700 con 0x7f0854100820 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.862+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0844004750 con 0x7f0854100820 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.862+0000 7f0850ff9700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f083c0a2cb0 0x7f083c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0852ffd700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f083c0a2cb0 0x7f083c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f08440cf5b0 con 0x7f0854100820 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0850ff9700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] conn(0x7f083c0ade00 0x7f083c0b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0844021e50 con 0x7f083c0ade00 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f08440d6050 con 0x7f0854100820 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0853fff700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] conn(0x7f083c0ade00 0x7f083c0b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0852ffd700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f083c0a2cb0 0x7f083c0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f085410ecf0 tx=0x7f0848008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.863+0000 7f0853fff700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] conn(0x7f083c0ade00 0x7f083c0b02a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:27.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.864+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== osd.0 v2:172.21.15.38:6802/1038840841 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0844021e50 con 0x7f083c0ade00 2024-04-22T00:00:27.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.896+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 --> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0854103b40 con 0x7f083c0ade00 2024-04-22T00:00:27.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0850ff9700 1 -- 172.21.15.38:0/2525558801 <== osd.0 v2:172.21.15.38:6802/1038840841 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0854103b40 con 0x7f083c0ade00 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] conn(0x7f083c0ade00 msgr2=0x7f083c0b02a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] conn(0x7f083c0ade00 0x7f083c0b02a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f083c0a2cb0 msgr2=0x7f083c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f083c0a2cb0 0x7f083c0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f085410ecf0 tx=0x7f0848008040 comp rx=0 tx=0).stop 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 msgr2=0x7f08541a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.897+0000 7f0859ca1700 1 --2- 172.21.15.38:0/2525558801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0854100820 0x7f08541a0840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f084400c920 tx=0x7f084400cce0 comp rx=0 tx=0).stop 2024-04-22T00:00:27.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.898+0000 7f0853fff700 1 -- 172.21.15.38:0/2525558801 reap_dead start 2024-04-22T00:00:27.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.898+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 shutdown_connections 2024-04-22T00:00:27.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.898+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 >> 172.21.15.38:0/2525558801 conn(0x7f08540777a0 msgr2=0x7f085410dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:27.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.898+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 shutdown_connections 2024-04-22T00:00:27.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:27.898+0000 7f0859ca1700 1 -- 172.21.15.38:0/2525558801 wait complete. 2024-04-22T00:00:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.402+0000 7f2f02c67700 1 -- 172.21.15.38:0/3743981741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 msgr2=0x7f2efc1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.402+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3743981741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc1028f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009990 tx=0x7f2eec009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 -- 172.21.15.38:0/3743981741 shutdown_connections 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3743981741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2efc107590 0x7f2efc10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3743981741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc1028f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3743981741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2efc102ec0 0x7f2efc106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 -- 172.21.15.38:0/3743981741 >> 172.21.15.38:0/3743981741 conn(0x7f2efc0793a0 msgr2=0x7f2efc0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 -- 172.21.15.38:0/3743981741 shutdown_connections 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.403+0000 7f2f02c67700 1 -- 172.21.15.38:0/3743981741 wait complete. 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 Processor -- start 2024-04-22T00:00:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 -- start start 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2efc102ec0 0x7f2efc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2efc107590 0x7f2efc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc1125a0 con 0x7f2efc102510 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc112420 con 0x7f2efc102ec0 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f02c67700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc112720 con 0x7f2efc107590 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f00a03700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f00a03700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52764/0 (socket says 172.21.15.38:52764) 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f00a03700 1 -- 172.21.15.38:0/3079084556 learned_addr learned my addr 172.21.15.38:0/3079084556 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2efbfff700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2efc102ec0 0x7f2efc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.404+0000 7f2f01204700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2efc107590 0x7f2efc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 -- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2efc107590 msgr2=0x7f2efc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2efc107590 0x7f2efc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 -- 172.21.15.38:0/3079084556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2efc102ec0 msgr2=0x7f2efc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2efc102ec0 0x7f2efc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2efc198fc0 con 0x7f2efc102510 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f00a03700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc19e660 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2eec005910 tx=0x7f2eec0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec027070 con 0x7f2efc102510 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2efc199250 con 0x7f2efc102510 2024-04-22T00:00:28.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec003e40 con 0x7f2efc102510 2024-04-22T00:00:28.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec022d00 con 0x7f2efc102510 2024-04-22T00:00:28.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.405+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2efc1a54c0 con 0x7f2efc102510 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.406+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2efc0629c0 con 0x7f2efc102510 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.408+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2eec02a070 con 0x7f2efc102510 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.408+0000 7f2ef9ffb700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2ee40a2d30 0x7f2ee40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2efbfff700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2ee40a2d30 0x7f2ee40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2eec0d0c80 con 0x7f2efc102510 2024-04-22T00:00:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2ef9ffb700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] conn(0x7f2ee40ade90 0x7f2ee40b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2eec010ec0 con 0x7f2ee40ade90 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2eec0d10d0 con 0x7f2efc102510 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2f01204700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] conn(0x7f2ee40ade90 0x7f2ee40b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2efbfff700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2ee40a2d30 0x7f2ee40a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2efc108980 tx=0x7f2ef000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.409+0000 7f2f01204700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] conn(0x7f2ee40ade90 0x7f2ee40b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.410+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== osd.11 v2:172.21.15.203:6824/3896776627 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2eec010ec0 con 0x7f2ee40ade90 2024-04-22T00:00:28.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.429+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 --> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2efc062dc0 con 0x7f2ee40ade90 2024-04-22T00:00:28.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.429+0000 7f2ef9ffb700 1 -- 172.21.15.38:0/3079084556 <== osd.11 v2:172.21.15.203:6824/3896776627 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2efc062dc0 con 0x7f2ee40ade90 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] conn(0x7f2ee40ade90 msgr2=0x7f2ee40b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] conn(0x7f2ee40ade90 0x7f2ee40b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2ee40a2d30 msgr2=0x7f2ee40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f2ee40a2d30 0x7f2ee40a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2efc108980 tx=0x7f2ef000b410 comp rx=0 tx=0).stop 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 msgr2=0x7f2efc19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 --2- 172.21.15.38:0/3079084556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2efc102510 0x7f2efc19e660 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2eec005910 tx=0x7f2eec0047b0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f01204700 1 -- 172.21.15.38:0/3079084556 reap_dead start 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 shutdown_connections 2024-04-22T00:00:28.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 >> 172.21.15.38:0/3079084556 conn(0x7f2efc0793a0 msgr2=0x7f2efc100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:28.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 shutdown_connections 2024-04-22T00:00:28.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.430+0000 7f2f02c67700 1 -- 172.21.15.38:0/3079084556 wait complete. 2024-04-22T00:00:28.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:28 smithi038 ceph-mon[27972]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T00:00:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.778+0000 7f61920bb700 1 -- 172.21.15.38:0/1844702099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c1011c0 msgr2=0x7f618c10f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.778+0000 7f61920bb700 1 --2- 172.21.15.38:0/1844702099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c1011c0 0x7f618c10f100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f618400b0a0 tx=0x7f618400b3b0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 -- 172.21.15.38:0/1844702099 shutdown_connections 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 --2- 172.21.15.38:0/1844702099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c1011c0 0x7f618c10f100 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 --2- 172.21.15.38:0/1844702099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f618c1082c0 0x7f618c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 --2- 172.21.15.38:0/1844702099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c100820 0x7f618c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 -- 172.21.15.38:0/1844702099 >> 172.21.15.38:0/1844702099 conn(0x7f618c0777a0 msgr2=0x7f618c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 -- 172.21.15.38:0/1844702099 shutdown_connections 2024-04-22T00:00:28.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 -- 172.21.15.38:0/1844702099 wait complete. 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.779+0000 7f61920bb700 1 Processor -- start 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 -- start start 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 0x7f618c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c1011c0 0x7f618c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f618c1082c0 0x7f618c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112510 con 0x7f618c100820 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112390 con 0x7f618c1011c0 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f61920bb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f618c112690 con 0x7f618c1082c0 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 0x7f618c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 0x7f618c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52780/0 (socket says 172.21.15.38:52780) 2024-04-22T00:00:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 -- 172.21.15.38:0/3343826481 learned_addr learned my addr 172.21.15.38:0/3343826481 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618affd700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c1011c0 0x7f618c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618bfff700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f618c1082c0 0x7f618c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 -- 172.21.15.38:0/3343826481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f618c1082c0 msgr2=0x7f618c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f618c1082c0 0x7f618c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 -- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c1011c0 msgr2=0x7f618c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c1011c0 0x7f618c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618b7fe700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c198ff0 con 0x7f618c100820 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.780+0000 7f618affd700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f618c1011c0 0x7f618c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f618b7fe700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 0x7f618c19e690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f617c00c920 tx=0x7f617c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c007b00 con 0x7f618c100820 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617c004d10 con 0x7f618c100820 2024-04-22T00:00:28.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c1992e0 con 0x7f618c100820 2024-04-22T00:00:28.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c00f450 con 0x7f618c100820 2024-04-22T00:00:28.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.781+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c1a54f0 con 0x7f618c100820 2024-04-22T00:00:28.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.782+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f618c062900 con 0x7f618c100820 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.783+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f617c004750 con 0x7f618c100820 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.783+0000 7f6188ff9700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f61740a2cb0 0x7f61740a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f618affd700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f61740a2cb0 0x7f61740a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f617c0cf6a0 con 0x7f618c100820 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f6188ff9700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] conn(0x7f61740add90 0x7f61740b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f617c012ec0 con 0x7f61740add90 2024-04-22T00:00:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f617c0cfaf0 con 0x7f618c100820 2024-04-22T00:00:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f618affd700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f61740a2cb0 0x7f61740a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f618c107a00 tx=0x7f618000b5f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.784+0000 7f618bfff700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] conn(0x7f61740add90 0x7f61740b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.785+0000 7f618bfff700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] conn(0x7f61740add90 0x7f61740b0230 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.785+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== osd.4 v2:172.21.15.111:6808/753751366 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f617c012ec0 con 0x7f61740add90 2024-04-22T00:00:28.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:28 smithi203 ceph-mon[31431]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T00:00:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.802+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 --> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f618c062d80 con 0x7f61740add90 2024-04-22T00:00:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f6188ff9700 1 -- 172.21.15.38:0/3343826481 <== osd.4 v2:172.21.15.111:6808/753751366 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f618c062d80 con 0x7f61740add90 2024-04-22T00:00:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] conn(0x7f61740add90 msgr2=0x7f61740b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] conn(0x7f61740add90 0x7f61740b0230 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f61740a2cb0 msgr2=0x7f61740a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f61740a2cb0 0x7f61740a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f618c107a00 tx=0x7f618000b5f0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 msgr2=0x7f618c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 --2- 172.21.15.38:0/3343826481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f618c100820 0x7f618c19e690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f617c00c920 tx=0x7f617c00cce0 comp rx=0 tx=0).stop 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f618bfff700 1 -- 172.21.15.38:0/3343826481 reap_dead start 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 shutdown_connections 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.803+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 >> 172.21.15.38:0/3343826481 conn(0x7f618c0777a0 msgr2=0x7f618c0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.804+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 shutdown_connections 2024-04-22T00:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:28.804+0000 7f61920bb700 1 -- 172.21.15.38:0/3343826481 wait complete. 2024-04-22T00:00:28.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:28 smithi111 ceph-mon[31543]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T00:00:29.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:29 smithi038 ceph-mon[27972]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 511 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-04-22T00:00:29.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:29 smithi203 ceph-mon[31431]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 511 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-04-22T00:00:29.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:29 smithi111 ceph-mon[31543]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 511 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-04-22T00:00:31.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:31 smithi038 ceph-mon[27972]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:31.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:31 smithi203 ceph-mon[31431]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:31.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:31 smithi111 ceph-mon[31543]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:32.347 INFO:teuthology.orchestra.run.smithi038.stdout:141733920827 2024-04-22T00:00:32.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 -- 172.21.15.38:0/2220563024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac101130 msgr2=0x7f80ac1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 --2- 172.21.15.38:0/2220563024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac101130 0x7f80ac1015b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f809c009990 tx=0x7f809c009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 -- 172.21.15.38:0/2220563024 shutdown_connections 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 --2- 172.21.15.38:0/2220563024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101af0 0x7f80ac10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 --2- 172.21.15.38:0/2220563024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80ac1071a0 0x7f80ac107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 --2- 172.21.15.38:0/2220563024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac101130 0x7f80ac1015b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 -- 172.21.15.38:0/2220563024 >> 172.21.15.38:0/2220563024 conn(0x7f80ac0780e0 msgr2=0x7f80ac0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 -- 172.21.15.38:0/2220563024 shutdown_connections 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 -- 172.21.15.38:0/2220563024 wait complete. 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.392+0000 7f80b45cc700 1 Processor -- start 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 -- start start 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101130 0x7f80ac19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80ac101af0 0x7f80ac19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 0x7f80ac198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac111440 con 0x7f80ac1071a0 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac1112c0 con 0x7f80ac101af0 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b45cc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac1115c0 con 0x7f80ac101130 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 0x7f80ac198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:33.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 0x7f80ac198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52808/0 (socket says 172.21.15.38:52808) 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 -- 172.21.15.38:0/3586867578 learned_addr learned my addr 172.21.15.38:0/3586867578 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2368700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101130 0x7f80ac19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b1b67700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80ac101af0 0x7f80ac19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 -- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101130 msgr2=0x7f80ac19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101130 0x7f80ac19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 -- 172.21.15.38:0/3586867578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80ac101af0 msgr2=0x7f80ac19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80ac101af0 0x7f80ac19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac198f70 con 0x7f80ac1071a0 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2368700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80ac101130 0x7f80ac19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.393+0000 7f80b2b69700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 0x7f80ac198790 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80a4007ee0 tx=0x7f80a400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.394+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a400dbd0 con 0x7f80ac1071a0 2024-04-22T00:00:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.394+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a400dd70 con 0x7f80ac1071a0 2024-04-22T00:00:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.394+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a40133f0 con 0x7f80ac1071a0 2024-04-22T00:00:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.394+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ac199260 con 0x7f80ac1071a0 2024-04-22T00:00:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.394+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ac1a5470 con 0x7f80ac1071a0 2024-04-22T00:00:33.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.395+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80ac105260 con 0x7f80ac1071a0 2024-04-22T00:00:33.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.396+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f80a401a020 con 0x7f80ac1071a0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80a37fe700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80980a2d30 0x7f80980a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f80a40cf370 con 0x7f80ac1071a0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80b2368700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80980a2d30 0x7f80980a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80a37fe700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] conn(0x7f80980ade90 0x7f80980b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80a4004860 con 0x7f80980ade90 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f80a40cf7c0 con 0x7f80ac1071a0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80b1b67700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] conn(0x7f80980ade90 0x7f80980b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.397+0000 7f80b2368700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80980a2d30 0x7f80980a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80a8005e10 tx=0x7f80a8005d60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.398+0000 7f80b1b67700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] conn(0x7f80980ade90 0x7f80980b0330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:33.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.399+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== osd.2 v2:172.21.15.203:6800/2981966976 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80a4004860 con 0x7f80980ade90 2024-04-22T00:00:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.415+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 --> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80ac102850 con 0x7f80980ade90 2024-04-22T00:00:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80a37fe700 1 -- 172.21.15.38:0/3586867578 <== osd.2 v2:172.21.15.203:6800/2981966976 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f80ac102850 con 0x7f80980ade90 2024-04-22T00:00:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] conn(0x7f80980ade90 msgr2=0x7f80980b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] conn(0x7f80980ade90 0x7f80980b0330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80980a2d30 msgr2=0x7f80980a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f80980a2d30 0x7f80980a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80a8005e10 tx=0x7f80a8005d60 comp rx=0 tx=0).stop 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 msgr2=0x7f80ac198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 --2- 172.21.15.38:0/3586867578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ac1071a0 0x7f80ac198790 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80a4007ee0 tx=0x7f80a400d3b0 comp rx=0 tx=0).stop 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b2b69700 1 -- 172.21.15.38:0/3586867578 reap_dead start 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 shutdown_connections 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 >> 172.21.15.38:0/3586867578 conn(0x7f80ac0780e0 msgr2=0x7f80ac100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 shutdown_connections 2024-04-22T00:00:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:33.416+0000 7f80b45cc700 1 -- 172.21.15.38:0/3586867578 wait complete. 2024-04-22T00:00:33.592 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:33 smithi038 ceph-mon[27972]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:33.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:33 smithi203 ceph-mon[31431]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:33.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:33 smithi111 ceph-mon[31543]: pgmap v324: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 -- 172.21.15.38:0/263971151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da01082b0 msgr2=0x7f3da0108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 --2- 172.21.15.38:0/263971151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da01082b0 0x7f3da0108690 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009990 tx=0x7f3d90009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 -- 172.21.15.38:0/263971151 shutdown_connections 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 --2- 172.21.15.38:0/263971151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da006a8e0 0x7f3da010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 --2- 172.21.15.38:0/263971151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da01082b0 0x7f3da0108690 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 --2- 172.21.15.38:0/263971151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da0069f40 0x7f3da006a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 -- 172.21.15.38:0/263971151 >> 172.21.15.38:0/263971151 conn(0x7f3da00780e0 msgr2=0x7f3da00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 -- 172.21.15.38:0/263971151 shutdown_connections 2024-04-22T00:00:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.042+0000 7f3da4831700 1 -- 172.21.15.38:0/263971151 wait complete. 2024-04-22T00:00:34.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.043+0000 7f3da4831700 1 Processor -- start 2024-04-22T00:00:34.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.043+0000 7f3da4831700 1 -- start start 2024-04-22T00:00:34.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 0x7f3da0101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da006a8e0 0x7f3da0101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da01082b0 0x7f3da0101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3da01124f0 con 0x7f3da0069f40 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0112370 con 0x7f3da01082b0 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3da4831700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0112670 con 0x7f3da006a8e0 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3d9e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 0x7f3da0101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:34.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3d9e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 0x7f3da0101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53960/0 (socket says 172.21.15.38:53960) 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3d9e59c700 1 -- 172.21.15.38:0/1720351028 learned_addr learned my addr 172.21.15.38:0/1720351028 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.044+0000 7f3d9dd9b700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da006a8e0 0x7f3da0101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 -- 172.21.15.38:0/1720351028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da006a8e0 msgr2=0x7f3da0101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da006a8e0 0x7f3da0101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 -- 172.21.15.38:0/1720351028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da01082b0 msgr2=0x7f3da0101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3da01082b0 0x7f3da0101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da01ab3b0 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d9e59c700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 0x7f3da0101200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d9000bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90027070 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d90022470 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000f6c0 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da01ab5e0 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.045+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da01aba40 con 0x7f3da0069f40 2024-04-22T00:00:34.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.046+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3da0062940 con 0x7f3da0069f40 2024-04-22T00:00:34.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.048+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3d9002a070 con 0x7f3da0069f40 2024-04-22T00:00:34.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.048+0000 7f3d8f7fe700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3d880a2d30 0x7f3d880a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d9dd9b700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3d880a2d30 0x7f3d880a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3d900d0c10 con 0x7f3da0069f40 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d8f7fe700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] conn(0x7f3d880ade90 0x7f3d880b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d90023e50 con 0x7f3d880ade90 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3d900d1060 con 0x7f3da0069f40 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d9ed9d700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] conn(0x7f3d880ade90 0x7f3d880b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d9dd9b700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3d880a2d30 0x7f3d880a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3da01029c0 tx=0x7f3d94009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:34.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.049+0000 7f3d9ed9d700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] conn(0x7f3d880ade90 0x7f3d880b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:34.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.050+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== osd.3 v2:172.21.15.38:6810/1951374002 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d90023e50 con 0x7f3d880ade90 2024-04-22T00:00:34.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.066+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 --> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3da0062e10 con 0x7f3d880ade90 2024-04-22T00:00:34.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3d8f7fe700 1 -- 172.21.15.38:0/1720351028 <== osd.3 v2:172.21.15.38:6810/1951374002 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3da0062e10 con 0x7f3d880ade90 2024-04-22T00:00:34.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] conn(0x7f3d880ade90 msgr2=0x7f3d880b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] conn(0x7f3d880ade90 0x7f3d880b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:34.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3d880a2d30 msgr2=0x7f3d880a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3d880a2d30 0x7f3d880a51f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3da01029c0 tx=0x7f3d94009260 comp rx=0 tx=0).stop 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 msgr2=0x7f3da0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 --2- 172.21.15.38:0/1720351028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3da0069f40 0x7f3da0101200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3d90000c00 tx=0x7f3d9000bb60 comp rx=0 tx=0).stop 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3d9ed9d700 1 -- 172.21.15.38:0/1720351028 reap_dead start 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 shutdown_connections 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 >> 172.21.15.38:0/1720351028 conn(0x7f3da00780e0 msgr2=0x7f3da00fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.067+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 shutdown_connections 2024-04-22T00:00:34.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:34.068+0000 7f3da4831700 1 -- 172.21.15.38:0/1720351028 wait complete. 2024-04-22T00:00:35.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:35 smithi038 ceph-mon[27972]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:35 smithi203 ceph-mon[31431]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:35.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:35 smithi111 ceph-mon[31543]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T00:00:36.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:36 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:00:36.681 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:36 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:00:36.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:36 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:00:37.223 INFO:teuthology.orchestra.run.smithi038.stdout:107374182465 2024-04-22T00:00:37.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T00:00:37.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:37 smithi038 ceph-mon[27972]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 10 op/s; 37 B/s, 0 objects/s recovering 2024-04-22T00:00:37.688 INFO:teuthology.orchestra.run.smithi038.stdout:343597383704 2024-04-22T00:00:37.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T00:00:37.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:37 smithi203 ceph-mon[31431]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 10 op/s; 37 B/s, 0 objects/s recovering 2024-04-22T00:00:37.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:37 smithi111 ceph-mon[31543]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 329 B/s wr, 10 op/s; 37 B/s, 0 objects/s recovering 2024-04-22T00:00:38.056 INFO:teuthology.orchestra.run.smithi038.stdout:34359738448 2024-04-22T00:00:38.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T00:00:38.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:38.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:38.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:38 smithi038 ceph-mon[27972]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s; 33 B/s, 0 objects/s recovering 2024-04-22T00:00:38.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:38.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:38 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:38 smithi203 ceph-mon[31431]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s; 33 B/s, 0 objects/s recovering 2024-04-22T00:00:39.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:38 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:38 smithi111 ceph-mon[31543]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 297 B/s wr, 10 op/s; 33 B/s, 0 objects/s recovering 2024-04-22T00:00:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:38 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:00:39.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 -- 172.21.15.38:0/2411376673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf201024e0 msgr2=0x7fbf201028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 --2- 172.21.15.38:0/2411376673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf201024e0 0x7fbf201028c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009990 tx=0x7fbf10009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 -- 172.21.15.38:0/2411376673 shutdown_connections 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 --2- 172.21.15.38:0/2411376673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf20107570 0x7fbf2010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 --2- 172.21.15.38:0/2411376673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf201024e0 0x7fbf201028c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 --2- 172.21.15.38:0/2411376673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20102e90 0x7fbf20106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 -- 172.21.15.38:0/2411376673 >> 172.21.15.38:0/2411376673 conn(0x7fbf20079390 msgr2=0x7fbf200797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 -- 172.21.15.38:0/2411376673 shutdown_connections 2024-04-22T00:00:39.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.498+0000 7fbf2669e700 1 -- 172.21.15.38:0/2411376673 wait complete. 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.499+0000 7fbf2669e700 1 Processor -- start 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.499+0000 7fbf2669e700 1 -- start start 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf201024e0 0x7fbf2019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 0x7fbf2019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20107570 0x7fbf20198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbf201125b0 con 0x7fbf20102e90 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20112430 con 0x7fbf20107570 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf2669e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20112730 con 0x7fbf201024e0 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 0x7fbf2019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:39.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 0x7fbf2019ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53980/0 (socket says 172.21.15.38:53980) 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 -- 172.21.15.38:0/911696232 learned_addr learned my addr 172.21.15.38:0/911696232 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1ffff700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf201024e0 0x7fbf2019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf24c3b700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20107570 0x7fbf20198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 -- 172.21.15.38:0/911696232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf201024e0 msgr2=0x7fbf2019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf201024e0 0x7fbf2019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 -- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20107570 msgr2=0x7fbf20198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20107570 0x7fbf20198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf20199070 con 0x7fbf20102e90 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf24c3b700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf20107570 0x7fbf20198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.500+0000 7fbf1f7fe700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 0x7fbf2019ec90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400c8f0 tx=0x7fbf1400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.501+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14007940 con 0x7fbf20102e90 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.501+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf1400f450 con 0x7fbf20102e90 2024-04-22T00:00:39.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.501+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf20199360 con 0x7fbf20102e90 2024-04-22T00:00:39.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.501+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf14018660 con 0x7fbf20102e90 2024-04-22T00:00:39.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.501+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf201a5620 con 0x7fbf20102e90 2024-04-22T00:00:39.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.502+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf20062940 con 0x7fbf20102e90 2024-04-22T00:00:39.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.504+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbf1400f600 con 0x7fbf20102e90 2024-04-22T00:00:39.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.504+0000 7fbf1d7fa700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbf080a2cb0 0x7fbf080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbf140cf6d0 con 0x7fbf20102e90 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1ffff700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbf080a2cb0 0x7fbf080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1d7fa700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] conn(0x7fbf080ade10 0x7fbf080b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf14024ea0 con 0x7fbf080ade10 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbf140cfb20 con 0x7fbf20102e90 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf1ffff700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbf080a2cb0 0x7fbf080a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b5c0 tx=0x7fbf10005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.505+0000 7fbf24c3b700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] conn(0x7fbf080ade10 0x7fbf080b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:39.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.506+0000 7fbf24c3b700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] conn(0x7fbf080ade10 0x7fbf080b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:39.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.506+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== osd.1 v2:172.21.15.111:6800/4079345941 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf14024ea0 con 0x7fbf080ade10 2024-04-22T00:00:39.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.523+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 --> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf20062dc0 con 0x7fbf080ade10 2024-04-22T00:00:39.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf1d7fa700 1 -- 172.21.15.38:0/911696232 <== osd.1 v2:172.21.15.111:6800/4079345941 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbf20062dc0 con 0x7fbf080ade10 2024-04-22T00:00:39.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] conn(0x7fbf080ade10 msgr2=0x7fbf080b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] conn(0x7fbf080ade10 0x7fbf080b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbf080a2cb0 msgr2=0x7fbf080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbf080a2cb0 0x7fbf080a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000b5c0 tx=0x7fbf10005960 comp rx=0 tx=0).stop 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 msgr2=0x7fbf2019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 --2- 172.21.15.38:0/911696232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbf20102e90 0x7fbf2019ec90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbf1400c8f0 tx=0x7fbf1400cc00 comp rx=0 tx=0).stop 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf24c3b700 1 -- 172.21.15.38:0/911696232 reap_dead start 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 shutdown_connections 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 >> 172.21.15.38:0/911696232 conn(0x7fbf20079390 msgr2=0x7fbf20101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 shutdown_connections 2024-04-22T00:00:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:39.524+0000 7fbf2669e700 1 -- 172.21.15.38:0/911696232 wait complete. 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 -- 172.21.15.38:0/590714617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3074069f40 msgr2=0x7f307406a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 --2- 172.21.15.38:0/590714617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3074069f40 0x7f307406a3a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3064009990 tx=0x7f3064009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 -- 172.21.15.38:0/590714617 shutdown_connections 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 --2- 172.21.15.38:0/590714617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f307406a8e0 0x7f307410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 --2- 172.21.15.38:0/590714617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30741082b0 0x7f3074108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 --2- 172.21.15.38:0/590714617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3074069f40 0x7f307406a3a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 -- 172.21.15.38:0/590714617 >> 172.21.15.38:0/590714617 conn(0x7f30740780e0 msgr2=0x7f30740784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 -- 172.21.15.38:0/590714617 shutdown_connections 2024-04-22T00:00:40.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.161+0000 7f307b5ce700 1 -- 172.21.15.38:0/590714617 wait complete. 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 Processor -- start 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 -- start start 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3074069f40 0x7f307419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f307406a8e0 0x7f307419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 0x7f3074198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3074112510 con 0x7f30741082b0 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3074112390 con 0x7f307406a8e0 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307b5ce700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3074112690 con 0x7f3074069f40 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f3079b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 0x7f3074198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f3079b6b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 0x7f3074198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53998/0 (socket says 172.21.15.38:53998) 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f3079b6b700 1 -- 172.21.15.38:0/1606307745 learned_addr learned my addr 172.21.15.38:0/1606307745 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f3078b69700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f307406a8e0 0x7f307419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:40.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.162+0000 7f307936a700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3074069f40 0x7f307419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 -- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3074069f40 msgr2=0x7f307419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3074069f40 0x7f307419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 -- 172.21.15.38:0/1606307745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f307406a8e0 msgr2=0x7f307419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f307406a8e0 0x7f307419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3074199070 con 0x7f30741082b0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f3079b6b700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 0x7f3074198890 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f306c00ada0 tx=0x7f306c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c017020 con 0x7f30741082b0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f306c00cdb0 con 0x7f30741082b0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c0053e0 con 0x7f30741082b0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3074199360 con 0x7f30741082b0 2024-04-22T00:00:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.163+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30741a5520 con 0x7f30741082b0 2024-04-22T00:00:40.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.164+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3074062980 con 0x7f30741082b0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.165+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f306c00c8d0 con 0x7f30741082b0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f306a7fc700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30600a2cb0 0x7f30600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f306c0cf640 con 0x7f30741082b0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f307936a700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30600a2cb0 0x7f30600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f306a7fc700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] conn(0x7f30600ade10 0x7f30600b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f306c01ee50 con 0x7f30600ade10 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.166+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f306c0cfa90 con 0x7f30741082b0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.167+0000 7f3078b69700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] conn(0x7f30600ade10 0x7f30600b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.167+0000 7f307936a700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30600a2cb0 0x7f30600a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f30700045b0 tx=0x7f307000b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:40.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.167+0000 7f3078b69700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] conn(0x7f30600ade10 0x7f30600b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:40.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.168+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== osd.8 v2:172.21.15.203:6816/434241518 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f306c01ee50 con 0x7f30600ade10 2024-04-22T00:00:40.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.184+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 --> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3074062e00 con 0x7f30600ade10 2024-04-22T00:00:40.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.185+0000 7f306a7fc700 1 -- 172.21.15.38:0/1606307745 <== osd.8 v2:172.21.15.203:6816/434241518 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3074062e00 con 0x7f30600ade10 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.185+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] conn(0x7f30600ade10 msgr2=0x7f30600b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.185+0000 7f307b5ce700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] conn(0x7f30600ade10 0x7f30600b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30600a2cb0 msgr2=0x7f30600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f30600a2cb0 0x7f30600a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f30700045b0 tx=0x7f307000b3c0 comp rx=0 tx=0).stop 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 msgr2=0x7f3074198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 --2- 172.21.15.38:0/1606307745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30741082b0 0x7f3074198890 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f306c00ada0 tx=0x7f306c00c330 comp rx=0 tx=0).stop 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f3079b6b700 1 -- 172.21.15.38:0/1606307745 reap_dead start 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 shutdown_connections 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 >> 172.21.15.38:0/1606307745 conn(0x7f30740780e0 msgr2=0x7f30740ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:40.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 shutdown_connections 2024-04-22T00:00:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:40.186+0000 7f307b5ce700 1 -- 172.21.15.38:0/1606307745 wait complete. 2024-04-22T00:00:40.659 INFO:teuthology.orchestra.run.smithi038.stdout:77309411399 2024-04-22T00:00:40.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T00:00:40.919 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:40 smithi038 ceph-mon[27972]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s; 19 B/s, 0 objects/s recovering 2024-04-22T00:00:41.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:40 smithi203 ceph-mon[31431]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s; 19 B/s, 0 objects/s recovering 2024-04-22T00:00:41.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:40 smithi111 ceph-mon[31543]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s; 19 B/s, 0 objects/s recovering 2024-04-22T00:00:41.582 INFO:teuthology.orchestra.run.smithi038.stdout:55834574925 2024-04-22T00:00:41.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T00:00:42.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/1252092683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e4107590 msgr2=0x7fa7e410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/1252092683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e4107590 0x7fa7e410f160 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00b0d0 tx=0x7fa7dc00b3e0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/1252092683 shutdown_connections 2024-04-22T00:00:42.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/1252092683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e4107590 0x7fa7e410f160 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/1252092683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e406ac40 0x7fa7e406b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/1252092683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e406b5f0 0x7fa7e4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/1252092683 >> 172.21.15.38:0/1252092683 conn(0x7fa7e40781d0 msgr2=0x7fa7e40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/1252092683 shutdown_connections 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/1252092683 wait complete. 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.991+0000 7fa7ea9bb700 1 Processor -- start 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 -- start start 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 0x7fa7e419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e406b5f0 0x7fa7e419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e4107590 0x7fa7e4198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112590 con 0x7fa7e406ac40 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112410 con 0x7fa7e406b5f0 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7ea9bb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4112710 con 0x7fa7e4107590 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 0x7fa7e419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:42.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 0x7fa7e419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54016/0 (socket says 172.21.15.38:54016) 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 -- 172.21.15.38:0/3388265546 learned_addr learned my addr 172.21.15.38:0/3388265546 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e37fe700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e406b5f0 0x7fa7e419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e8f58700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e4107590 0x7fa7e4198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 -- 172.21.15.38:0/3388265546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e4107590 msgr2=0x7fa7e4198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e4107590 0x7fa7e4198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 -- 172.21.15.38:0/3388265546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e406b5f0 msgr2=0x7fa7e419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e406b5f0 0x7fa7e419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.992+0000 7fa7e3fff700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4199010 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7e3fff700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 0x7fa7e419e6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9a0 tx=0x7fa7d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d400ca10 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d400cbb0 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e4199300 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4018680 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.993+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a5510 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.994+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa7e4062940 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.995+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa7d4010430 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e17fa700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa7cc0a2cb0 0x7fa7cc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7d40d02a0 con 0x7fa7e406ac40 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e37fe700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa7cc0a2cb0 0x7fa7cc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e17fa700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] conn(0x7fa7cc0ade10 0x7fa7cc0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7d401bec0 con 0x7fa7cc0ade10 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa7d40d06f0 con 0x7fa7e406ac40 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e8f58700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] conn(0x7fa7cc0ade10 0x7fa7cc0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e37fe700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa7cc0a2cb0 0x7fa7cc0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4108910 tx=0x7fa7d800b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.996+0000 7fa7e8f58700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] conn(0x7fa7cc0ade10 0x7fa7cc0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:41.997+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== osd.9 v2:172.21.15.38:6826/654111402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7d401bec0 con 0x7fa7cc0ade10 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.013+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 --> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7e4062d40 con 0x7fa7cc0ade10 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7e17fa700 1 -- 172.21.15.38:0/3388265546 <== osd.9 v2:172.21.15.38:6826/654111402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa7e4062d40 con 0x7fa7cc0ade10 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] conn(0x7fa7cc0ade10 msgr2=0x7fa7cc0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] conn(0x7fa7cc0ade10 0x7fa7cc0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa7cc0a2cb0 msgr2=0x7fa7cc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa7cc0a2cb0 0x7fa7cc0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4108910 tx=0x7fa7d800b410 comp rx=0 tx=0).stop 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 msgr2=0x7fa7e419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:42.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 --2- 172.21.15.38:0/3388265546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e406ac40 0x7fa7e419e6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9a0 tx=0x7fa7d400ecb0 comp rx=0 tx=0).stop 2024-04-22T00:00:42.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7e8f58700 1 -- 172.21.15.38:0/3388265546 reap_dead start 2024-04-22T00:00:42.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 shutdown_connections 2024-04-22T00:00:42.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 >> 172.21.15.38:0/3388265546 conn(0x7fa7e40781d0 msgr2=0x7fa7e40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:42.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 shutdown_connections 2024-04-22T00:00:42.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:42.014+0000 7fa7ea9bb700 1 -- 172.21.15.38:0/3388265546 wait complete. 2024-04-22T00:00:42.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:42 smithi038 ceph-mon[27972]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:00:43.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:42 smithi203 ceph-mon[31431]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:00:43.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:42 smithi111 ceph-mon[31543]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 -- 172.21.15.38:0/2315024533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc9806ac40 msgr2=0x7fcc9806b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 --2- 172.21.15.38:0/2315024533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc9806ac40 0x7fcc9806b020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcc940099c0 tx=0x7fcc94009cd0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 -- 172.21.15.38:0/2315024533 shutdown_connections 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 --2- 172.21.15.38:0/2315024533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc98107590 0x7fcc9810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 --2- 172.21.15.38:0/2315024533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc9806ac40 0x7fcc9806b020 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 --2- 172.21.15.38:0/2315024533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc9806b5f0 0x7fcc98106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.622+0000 7fcc9f967700 1 -- 172.21.15.38:0/2315024533 >> 172.21.15.38:0/2315024533 conn(0x7fcc980781d0 msgr2=0x7fcc980785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.623+0000 7fcc9f967700 1 -- 172.21.15.38:0/2315024533 shutdown_connections 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.623+0000 7fcc9f967700 1 -- 172.21.15.38:0/2315024533 wait complete. 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.623+0000 7fcc9f967700 1 Processor -- start 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.623+0000 7fcc9f967700 1 -- start start 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc9806ac40 0x7fcc9819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc9806b5f0 0x7fcc9819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 0x7fcc98198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc98112590 con 0x7fcc98107590 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcc98112410 con 0x7fcc9806ac40 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9f967700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc98112710 con 0x7fcc9806b5f0 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9df04700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 0x7fcc98198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9df04700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 0x7fcc98198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54036/0 (socket says 172.21.15.38:54036) 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9df04700 1 -- 172.21.15.38:0/1889904507 learned_addr learned my addr 172.21.15.38:0/1889904507 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9d703700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc9806ac40 0x7fcc9819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.624+0000 7fcc9cf02700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc9806b5f0 0x7fcc9819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 -- 172.21.15.38:0/1889904507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc9806b5f0 msgr2=0x7fcc9819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc9806b5f0 0x7fcc9819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 -- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc9806ac40 msgr2=0x7fcc9819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc9806ac40 0x7fcc9819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc98198fd0 con 0x7fcc98107590 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9df04700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 0x7fcc98198870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcc90007ee0 tx=0x7fcc9000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9000dd20 con 0x7fcc98107590 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc981992c0 con 0x7fcc98107590 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc90004020 con 0x7fcc98107590 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc90015660 con 0x7fcc98107590 2024-04-22T00:00:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.625+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc981a5580 con 0x7fcc98107590 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.626+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc980629c0 con 0x7fcc98107590 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.627+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcc900041d0 con 0x7fcc98107590 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc8e7fc700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcc840a2cb0 0x7fcc840a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc9d703700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcc840a2cb0 0x7fcc840a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcc900cfa60 con 0x7fcc98107590 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc8e7fc700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] conn(0x7fcc840ade10 0x7fcc840b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc90020e50 con 0x7fcc840ade10 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcc900cfeb0 con 0x7fcc98107590 2024-04-22T00:00:43.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc9d703700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcc840a2cb0 0x7fcc840a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc94005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.628+0000 7fcc9cf02700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] conn(0x7fcc840ade10 0x7fcc840b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.629+0000 7fcc9cf02700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] conn(0x7fcc840ade10 0x7fcc840b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.630+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== osd.7 v2:172.21.15.111:6816/961343269 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc90020e50 con 0x7fcc840ade10 2024-04-22T00:00:43.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.646+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 --> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc98062dc0 con 0x7fcc840ade10 2024-04-22T00:00:43.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc8e7fc700 1 -- 172.21.15.38:0/1889904507 <== osd.7 v2:172.21.15.111:6816/961343269 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcc98062dc0 con 0x7fcc840ade10 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] conn(0x7fcc840ade10 msgr2=0x7fcc840b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] conn(0x7fcc840ade10 0x7fcc840b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcc840a2cb0 msgr2=0x7fcc840a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcc840a2cb0 0x7fcc840a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc94005fb0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 msgr2=0x7fcc98198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 --2- 172.21.15.38:0/1889904507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc98107590 0x7fcc98198870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcc90007ee0 tx=0x7fcc9000d3b0 comp rx=0 tx=0).stop 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9df04700 1 -- 172.21.15.38:0/1889904507 reap_dead start 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 shutdown_connections 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 >> 172.21.15.38:0/1889904507 conn(0x7fcc980781d0 msgr2=0x7fcc980ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 shutdown_connections 2024-04-22T00:00:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:43.647+0000 7fcc9f967700 1 -- 172.21.15.38:0/1889904507 wait complete. 2024-04-22T00:00:44.326 INFO:teuthology.orchestra.run.smithi038.stdout:253403070505 2024-04-22T00:00:44.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T00:00:44.725 INFO:teuthology.orchestra.run.smithi038.stdout:219043332143 2024-04-22T00:00:44.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T00:00:44.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:44 smithi038 ceph-mon[27972]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T00:00:45.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:44 smithi203 ceph-mon[31431]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T00:00:45.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:44 smithi111 ceph-mon[31543]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 -- 172.21.15.38:0/3146875595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d0102e90 msgr2=0x7f52d0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 --2- 172.21.15.38:0/3146875595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d0102e90 0x7f52d0106e90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009990 tx=0x7f52c0009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 -- 172.21.15.38:0/3146875595 shutdown_connections 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 --2- 172.21.15.38:0/3146875595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0107570 0x7f52d010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 --2- 172.21.15.38:0/3146875595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52d01024e0 0x7f52d01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 --2- 172.21.15.38:0/3146875595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d0102e90 0x7f52d0106e90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 -- 172.21.15.38:0/3146875595 >> 172.21.15.38:0/3146875595 conn(0x7f52d0079390 msgr2=0x7f52d00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 -- 172.21.15.38:0/3146875595 shutdown_connections 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 -- 172.21.15.38:0/3146875595 wait complete. 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.105+0000 7f52d65bf700 1 Processor -- start 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 -- start start 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 0x7f52d019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52d0102e90 0x7f52d019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0107570 0x7f52d0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52d0112530 con 0x7f52d01024e0 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52d01123b0 con 0x7f52d0107570 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d65bf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52d01126b0 con 0x7f52d0102e90 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 0x7f52d019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 0x7f52d019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58524/0 (socket says 172.21.15.38:58524) 2024-04-22T00:00:45.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 -- 172.21.15.38:0/1070640032 learned_addr learned my addr 172.21.15.38:0/1070640032 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52d4b5c700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0107570 0x7f52d0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cf7fe700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52d0102e90 0x7f52d019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 -- 172.21.15.38:0/1070640032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52d0102e90 msgr2=0x7f52d019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52d0102e90 0x7f52d019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 -- 172.21.15.38:0/1070640032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0107570 msgr2=0x7f52d0198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0107570 0x7f52d0198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.106+0000 7f52cffff700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d0199010 con 0x7f52d01024e0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52cffff700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 0x7f52d019e660 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c970 tx=0x7f52b800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b80078d0 con 0x7f52d01024e0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b8018480 con 0x7f52d01024e0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d0199300 con 0x7f52d01024e0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b800f6b0 con 0x7f52d01024e0 2024-04-22T00:00:45.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.107+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d01a54c0 con 0x7f52d01024e0 2024-04-22T00:00:45.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.108+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f52d00629c0 con 0x7f52d01024e0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.109+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f52b8018630 con 0x7f52d01024e0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.109+0000 7f52cd7fa700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f52bc0a2d30 0x7f52bc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cf7fe700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f52bc0a2d30 0x7f52bc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f52b80cf9f0 con 0x7f52d01024e0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cd7fa700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] conn(0x7f52bc0ade90 0x7f52bc0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52b8015e70 con 0x7f52bc0ade90 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f52b80cfe40 con 0x7f52d01024e0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52d4b5c700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] conn(0x7f52bc0ade90 0x7f52bc0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52cf7fe700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f52bc0a2d30 0x7f52bc0a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52c0006010 tx=0x7f52c0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.110+0000 7f52d4b5c700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] conn(0x7f52bc0ade90 0x7f52bc0b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.111+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== osd.6 v2:172.21.15.38:6818/1747049129 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52b8015e70 con 0x7f52bc0ade90 2024-04-22T00:00:45.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 --> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f52d0062dc0 con 0x7f52bc0ade90 2024-04-22T00:00:45.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52cd7fa700 1 -- 172.21.15.38:0/1070640032 <== osd.6 v2:172.21.15.38:6818/1747049129 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f52d0062dc0 con 0x7f52bc0ade90 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] conn(0x7f52bc0ade90 msgr2=0x7f52bc0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] conn(0x7f52bc0ade90 0x7f52bc0b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f52bc0a2d30 msgr2=0x7f52bc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f52bc0a2d30 0x7f52bc0a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f52c0006010 tx=0x7f52c0005960 comp rx=0 tx=0).stop 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 msgr2=0x7f52d019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d65bf700 1 --2- 172.21.15.38:0/1070640032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52d01024e0 0x7f52d019e660 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f52b800c970 tx=0x7f52b800cc80 comp rx=0 tx=0).stop 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.128+0000 7f52d4b5c700 1 -- 172.21.15.38:0/1070640032 reap_dead start 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.129+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 shutdown_connections 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.129+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 >> 172.21.15.38:0/1070640032 conn(0x7f52d0079390 msgr2=0x7f52d0101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.129+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 shutdown_connections 2024-04-22T00:00:45.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.129+0000 7f52d65bf700 1 -- 172.21.15.38:0/1070640032 wait complete. 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.792+0000 7f35a5e63700 1 -- 172.21.15.38:0/3676901266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 msgr2=0x7f35a01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.792+0000 7f35a5e63700 1 --2- 172.21.15.38:0/3676901266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a01028c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3590009990 tx=0x7f3590009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 -- 172.21.15.38:0/3676901266 shutdown_connections 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 --2- 172.21.15.38:0/3676901266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0107570 0x7f35a010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 --2- 172.21.15.38:0/3676901266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a01028c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 --2- 172.21.15.38:0/3676901266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0102e90 0x7f35a0106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 -- 172.21.15.38:0/3676901266 >> 172.21.15.38:0/3676901266 conn(0x7f35a0079390 msgr2=0x7f35a00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 -- 172.21.15.38:0/3676901266 shutdown_connections 2024-04-22T00:00:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 -- 172.21.15.38:0/3676901266 wait complete. 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.793+0000 7f35a5e63700 1 Processor -- start 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 -- start start 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a0074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102e90 0x7f35a0074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0107570 0x7f35a0074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35a01125b0 con 0x7f35a01024e0 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0112430 con 0x7f35a0102e90 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f35a5e63700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0112730 con 0x7f35a0107570 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a0074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a0074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58548/0 (socket says 172.21.15.38:58548) 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 -- 172.21.15.38:0/992115279 learned_addr learned my addr 172.21.15.38:0/992115279 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f3597fff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102e90 0x7f35a0074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359ffff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0107570 0x7f35a0074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 -- 172.21.15.38:0/992115279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0107570 msgr2=0x7f35a0074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0107570 0x7f35a0074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 -- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102e90 msgr2=0x7f35a0074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102e90 0x7f35a0074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359f7fe700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a01a6f70 con 0x7f35a01024e0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f3597fff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102e90 0x7f35a0074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.794+0000 7f359ffff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35a0107570 0x7f35a0074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f359f7fe700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a0074530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3590000c00 tx=0x7f3590003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3590027070 con 0x7f35a01024e0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3590004190 con 0x7f35a01024e0 2024-04-22T00:00:45.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359000f460 con 0x7f35a01024e0 2024-04-22T00:00:45.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a01a7200 con 0x7f35a01024e0 2024-04-22T00:00:45.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.795+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a01a7600 con 0x7f35a01024e0 2024-04-22T00:00:45.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.796+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35a0062940 con 0x7f35a01024e0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.797+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f359002a070 con 0x7f35a01024e0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f359d7fa700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f358c0a2cb0 0x7f358c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35900d0c90 con 0x7f35a01024e0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f3597fff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f358c0a2cb0 0x7f358c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f359d7fa700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] conn(0x7f358c0ade10 0x7f358c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3590001d30 con 0x7f358c0ade10 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.798+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f35900d1040 con 0x7f35a01024e0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.799+0000 7f3597fff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f358c0a2cb0 0x7f358c0a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3588005fd0 tx=0x7f3588005dc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.799+0000 7f359ffff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] conn(0x7f358c0ade10 0x7f358c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:45.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.799+0000 7f359ffff700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] conn(0x7f358c0ade10 0x7f358c0b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:45.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.800+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== osd.10 v2:172.21.15.111:6824/4030828796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3590001d30 con 0x7f358c0ade10 2024-04-22T00:00:45.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.818+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 --> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f35a0062d40 con 0x7f358c0ade10 2024-04-22T00:00:45.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f359d7fa700 1 -- 172.21.15.38:0/992115279 <== osd.10 v2:172.21.15.111:6824/4030828796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f35a0062d40 con 0x7f358c0ade10 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] conn(0x7f358c0ade10 msgr2=0x7f358c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] conn(0x7f358c0ade10 0x7f358c0b02b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f358c0a2cb0 msgr2=0x7f358c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f358c0a2cb0 0x7f358c0a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3588005fd0 tx=0x7f3588005dc0 comp rx=0 tx=0).stop 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 msgr2=0x7f35a0074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 --2- 172.21.15.38:0/992115279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35a01024e0 0x7f35a0074530 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3590000c00 tx=0x7f3590003730 comp rx=0 tx=0).stop 2024-04-22T00:00:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f359ffff700 1 -- 172.21.15.38:0/992115279 reap_dead start 2024-04-22T00:00:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 shutdown_connections 2024-04-22T00:00:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 >> 172.21.15.38:0/992115279 conn(0x7f35a0079390 msgr2=0x7f35a01086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 shutdown_connections 2024-04-22T00:00:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:45.819+0000 7f35a5e63700 1 -- 172.21.15.38:0/992115279 wait complete. 2024-04-22T00:00:46.272 INFO:teuthology.orchestra.run.smithi038.stdout:197568495669 2024-04-22T00:00:46.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T00:00:46.862 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:46 smithi038 ceph-mon[27972]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-04-22T00:00:46.889 INFO:teuthology.orchestra.run.smithi038.stdout:287762808869 2024-04-22T00:00:46.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T00:00:47.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:46 smithi203 ceph-mon[31431]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-04-22T00:00:47.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:46 smithi111 ceph-mon[31543]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-04-22T00:00:47.282 INFO:teuthology.orchestra.run.smithi038.stdout:309237645344 2024-04-22T00:00:47.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T00:00:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 -- 172.21.15.38:0/694002330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08102510 msgr2=0x7f3f081028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/694002330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08102510 0x7f3f081028f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009990 tx=0x7f3f04009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 -- 172.21.15.38:0/694002330 shutdown_connections 2024-04-22T00:00:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/694002330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08107590 0x7f3f0810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/694002330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08102510 0x7f3f081028f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/694002330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f08102ec0 0x7f3f08106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 -- 172.21.15.38:0/694002330 >> 172.21.15.38:0/694002330 conn(0x7f3f080793a0 msgr2=0x7f3f080797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 -- 172.21.15.38:0/694002330 shutdown_connections 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.564+0000 7f3f0fd54700 1 -- 172.21.15.38:0/694002330 wait complete. 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 Processor -- start 2024-04-22T00:00:47.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 -- start start 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f08102510 0x7f3f0819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08102ec0 0x7f3f0819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 0x7f3f08198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08112620 con 0x7f3f08107590 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081124a0 con 0x7f3f08102ec0 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0fd54700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081127a0 con 0x7f3f08102510 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0e2f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 0x7f3f08198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0e2f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 0x7f3f08198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58560/0 (socket says 172.21.15.38:58560) 2024-04-22T00:00:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0e2f1700 1 -- 172.21.15.38:0/3404361310 learned_addr learned my addr 172.21.15.38:0/3404361310 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0daf0700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f08102510 0x7f3f0819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.565+0000 7f3f0d2ef700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08102ec0 0x7f3f0819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 -- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f08102510 msgr2=0x7f3f0819e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f08102510 0x7f3f0819e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 -- 172.21.15.38:0/3404361310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08102ec0 msgr2=0x7f3f0819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08102ec0 0x7f3f0819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f08199060 con 0x7f3f08107590 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0d2ef700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f08102ec0 0x7f3f0819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0e2f1700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 0x7f3f08198900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000ada0 tx=0x7f3f0000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00017020 con 0x7f3f08107590 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f08199350 con 0x7f3f08107590 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f081a5610 con 0x7f3f08107590 2024-04-22T00:00:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00004bb0 con 0x7f3f08107590 2024-04-22T00:00:47.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.566+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00005520 con 0x7f3f08107590 2024-04-22T00:00:47.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.567+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f080629c0 con 0x7f3f08107590 2024-04-22T00:00:47.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.570+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3f000056c0 con 0x7f3f08107590 2024-04-22T00:00:47.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3efeffd700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3ef40a2d30 0x7f3ef40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:47.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3f000cf690 con 0x7f3f08107590 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3efeffd700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] conn(0x7f3ef40ade90 0x7f3ef40b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ef40b0a00 con 0x7f3ef40ade90 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3f000cfa40 con 0x7f3f08107590 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.571+0000 7f3f0daf0700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3ef40a2d30 0x7f3ef40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.572+0000 7f3f0d2ef700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] conn(0x7f3ef40ade90 0x7f3ef40b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.572+0000 7f3f0daf0700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3ef40a2d30 0x7f3ef40a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f04005f50 tx=0x7f3f04005e80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:47.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.573+0000 7f3f0d2ef700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] conn(0x7f3ef40ade90 0x7f3ef40b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:47.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.574+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== osd.5 v2:172.21.15.203:6808/1090281315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ef40b0a00 con 0x7f3ef40ade90 2024-04-22T00:00:47.574 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:47.574 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:47.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:47.587 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:47.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.613+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 --> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3f08062bd0 con 0x7f3ef40ade90 2024-04-22T00:00:47.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.615+0000 7f3efeffd700 1 -- 172.21.15.38:0/3404361310 <== osd.5 v2:172.21.15.203:6808/1090281315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3f08062bd0 con 0x7f3ef40ade90 2024-04-22T00:00:47.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.621+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] conn(0x7f3ef40ade90 msgr2=0x7f3ef40b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.621+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] conn(0x7f3ef40ade90 0x7f3ef40b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:47.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.624+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3ef40a2d30 msgr2=0x7f3ef40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.624+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3ef40a2d30 0x7f3ef40a51f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f04005f50 tx=0x7f3f04005e80 comp rx=0 tx=0).stop 2024-04-22T00:00:47.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.624+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 msgr2=0x7f3f08198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:47.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.624+0000 7f3f0fd54700 1 --2- 172.21.15.38:0/3404361310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f08107590 0x7f3f08198900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000ada0 tx=0x7f3f0000c330 comp rx=0 tx=0).stop 2024-04-22T00:00:47.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.624+0000 7f3f0e2f1700 1 -- 172.21.15.38:0/3404361310 reap_dead start 2024-04-22T00:00:47.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.626+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 shutdown_connections 2024-04-22T00:00:47.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.626+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 >> 172.21.15.38:0/3404361310 conn(0x7f3f080793a0 msgr2=0x7f3f08101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:47.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.628+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 shutdown_connections 2024-04-22T00:00:47.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:47.629+0000 7f3f0fd54700 1 -- 172.21.15.38:0/3404361310 wait complete. 2024-04-22T00:00:48.379 INFO:teuthology.orchestra.run.smithi038.stdout:163208757306 2024-04-22T00:00:48.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T00:00:48.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:48 smithi038 ceph-mon[27972]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:00:49.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:48 smithi203 ceph-mon[31431]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:00:49.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:48 smithi111 ceph-mon[31543]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:00:49.132 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:49.572 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:49.574 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:49.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:50.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:50 smithi038 ceph-mon[27972]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:00:51.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:50 smithi203 ceph-mon[31431]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:00:51.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:50 smithi111 ceph-mon[31543]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:00:53.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:52 smithi203 ceph-mon[31431]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:00:53.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:52 smithi111 ceph-mon[31543]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:00:53.102 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:52 smithi038 ceph-mon[27972]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:00:54.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:54 smithi038 ceph-mon[27972]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:00:54.980 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:54.981 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:55.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:54 smithi203 ceph-mon[31431]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:00:55.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:54 smithi111 ceph-mon[31543]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:00:55.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 -- 172.21.15.38:0/732155644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04101120 msgr2=0x7fde041015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 --2- 172.21.15.38:0/732155644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04101120 0x7fde041015a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009990 tx=0x7fddf4009ca0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 -- 172.21.15.38:0/732155644 shutdown_connections 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 --2- 172.21.15.38:0/732155644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101ae0 0x7fde0410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 --2- 172.21.15.38:0/732155644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04107190 0x7fde04107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 --2- 172.21.15.38:0/732155644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04101120 0x7fde041015a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.736+0000 7fde0b963700 1 -- 172.21.15.38:0/732155644 >> 172.21.15.38:0/732155644 conn(0x7fde040780a0 msgr2=0x7fde040784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.737+0000 7fde0b963700 1 -- 172.21.15.38:0/732155644 shutdown_connections 2024-04-22T00:00:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.737+0000 7fde0b963700 1 -- 172.21.15.38:0/732155644 wait complete. 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.737+0000 7fde0b963700 1 Processor -- start 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.737+0000 7fde0b963700 1 -- start start 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 0x7fde0419d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04101ae0 0x7fde0419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:55.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fde04111420 con 0x7fde04107190 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fde041112a0 con 0x7fde04101120 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde0b963700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde041115a0 con 0x7fde04101ae0 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50354/0 (socket says 172.21.15.38:50354) 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 -- 172.21.15.38:0/3801736272 learned_addr learned my addr 172.21.15.38:0/3801736272 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde08efe700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04101ae0 0x7fde0419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde096ff700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 0x7fde0419d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 -- 172.21.15.38:0/3801736272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04101ae0 msgr2=0x7fde0419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04101ae0 0x7fde0419dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 -- 172.21.15.38:0/3801736272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 msgr2=0x7fde0419d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 0x7fde0419d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 -- 172.21.15.38:0/3801736272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde04197f40 con 0x7fde04107190 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde096ff700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 0x7fde0419d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:00:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.738+0000 7fde09f00700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fddfc007d70 tx=0x7fddfc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.739+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc016020 con 0x7fde04107190 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.739+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddfc00ad40 con 0x7fde04107190 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.739+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc011600 con 0x7fde04107190 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.739+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde04198230 con 0x7fde04107190 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.739+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde041a4510 con 0x7fde04107190 2024-04-22T00:00:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.740+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde04102840 con 0x7fde04107190 2024-04-22T00:00:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.741+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fddfc019020 con 0x7fde04107190 2024-04-22T00:00:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.741+0000 7fddfa7fc700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 0x7fddf00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.742+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fddfc0cf710 con 0x7fde04107190 2024-04-22T00:00:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.742+0000 7fde096ff700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 0x7fddf00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.742+0000 7fde096ff700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 0x7fddf00a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde00005e10 tx=0x7fde00005d80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:55.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.745+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fddfc020020 con 0x7fde04107190 2024-04-22T00:00:55.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.958+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fde04198b90 con 0x7fde04107190 2024-04-22T00:00:55.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.958+0000 7fddfa7fc700 1 -- 172.21.15.38:0/3801736272 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fddfc022300 con 0x7fde04107190 2024-04-22T00:00:55.951 INFO:teuthology.orchestra.run.smithi038.stdout:107374182470 2024-04-22T00:00:55.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 msgr2=0x7fddf00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:55.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 0x7fddf00a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fde00005e10 tx=0x7fde00005d80 comp rx=0 tx=0).stop 2024-04-22T00:00:55.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 msgr2=0x7fde04197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:55.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fddfc007d70 tx=0x7fddfc00a340 comp rx=0 tx=0).stop 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 shutdown_connections 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fddf00a2d30 0x7fddf00a51f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde04107190 0x7fde04197660 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde04101120 0x7fde0419d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 --2- 172.21.15.38:0/3801736272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde04101ae0 0x7fde0419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.960+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 >> 172.21.15.38:0/3801736272 conn(0x7fde040780a0 msgr2=0x7fde040ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.961+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 shutdown_connections 2024-04-22T00:00:55.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:55.961+0000 7fde0b963700 1 -- 172.21.15.38:0/3801736272 wait complete. 2024-04-22T00:00:56.887 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:56 smithi038 ceph-mon[27972]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T00:00:56.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:56 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3801736272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T00:00:57.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:56 smithi203 ceph-mon[31431]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T00:00:57.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:56 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3801736272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T00:00:57.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:56 smithi111 ceph-mon[31543]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T00:00:57.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:56 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3801736272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T00:00:57.570 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:57.570 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 -- 172.21.15.38:0/689548967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d01013b0 msgr2=0x7f05d0101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 --2- 172.21.15.38:0/689548967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d01013b0 0x7f05d0101790 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f05c00099c0 tx=0x7f05c0009cd0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 -- 172.21.15.38:0/689548967 shutdown_connections 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 --2- 172.21.15.38:0/689548967 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d0106490 0x7f05d010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 --2- 172.21.15.38:0/689548967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d01013b0 0x7f05d0101790 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 --2- 172.21.15.38:0/689548967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0101d60 0x7f05d0105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 -- 172.21.15.38:0/689548967 >> 172.21.15.38:0/689548967 conn(0x7f05d00781d0 msgr2=0x7f05d00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:58.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 -- 172.21.15.38:0/689548967 shutdown_connections 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.500+0000 7f05d5e66700 1 -- 172.21.15.38:0/689548967 wait complete. 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 Processor -- start 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 -- start start 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d01013b0 0x7f05d0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0106490 0x7f05d01a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05d01114f0 con 0x7f05d0101d60 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0111370 con 0x7f05d0106490 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05d5e66700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05d0111670 con 0x7f05d01013b0 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50368/0 (socket says 172.21.15.38:50368) 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 -- 172.21.15.38:0/1816031154 learned_addr learned my addr 172.21.15.38:0/1816031154 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:00:58.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05cf7fe700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d01013b0 0x7f05d0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05cffff700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0106490 0x7f05d01a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 -- 172.21.15.38:0/1816031154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d01013b0 msgr2=0x7f05d0199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d01013b0 0x7f05d0199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 -- 172.21.15.38:0/1816031154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0106490 msgr2=0x7f05d01a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0106490 0x7f05d01a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.501+0000 7f05c7fff700 1 -- 172.21.15.38:0/1816031154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d01a4ed0 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05c7fff700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f05b800d980 tx=0x7f05b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8017070 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05b800fd70 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d01a5100 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b800e3b0 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.502+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d01a5690 con 0x7f05d0101d60 2024-04-22T00:00:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.503+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d0068f20 con 0x7f05d0101d60 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.504+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f05b800f890 con 0x7f05d0101d60 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.505+0000 7f05cd7fa700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 0x7f05bc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.505+0000 7f05cf7fe700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 0x7f05bc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.505+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f05b80ceee0 con 0x7f05d0101d60 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.506+0000 7f05cf7fe700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 0x7f05bc0a51f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f05c0000c00 tx=0x7f05c0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:00:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.509+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f05b809ba00 con 0x7f05d0101d60 2024-04-22T00:00:58.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.719+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f05d002db30 con 0x7f05d0101d60 2024-04-22T00:00:58.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.719+0000 7f05cd7fa700 1 -- 172.21.15.38:0/1816031154 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f05b809b300 con 0x7f05d0101d60 2024-04-22T00:00:58.712 INFO:teuthology.orchestra.run.smithi038.stdout:77309411404 2024-04-22T00:00:58.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.721+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 msgr2=0x7f05bc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:58.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.721+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 0x7f05bc0a51f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f05c0000c00 tx=0x7f05c0005fb0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.721+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 msgr2=0x7f05d0199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:00:58.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.721+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f05b800d980 tx=0x7f05b800dc90 comp rx=0 tx=0).stop 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 shutdown_connections 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f05bc0a2d30 0x7f05bc0a51f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d0106490 0x7f05d01a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05d01013b0 0x7f05d0199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 --2- 172.21.15.38:0/1816031154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05d0101d60 0x7f05d0199e50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 >> 172.21.15.38:0/1816031154 conn(0x7f05d00781d0 msgr2=0x7f05d00ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 shutdown_connections 2024-04-22T00:00:58.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:00:58.722+0000 7f05d5e66700 1 -- 172.21.15.38:0/1816031154 wait complete. 2024-04-22T00:00:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:58 smithi038 ceph-mon[27972]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:00:58.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:00:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1816031154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T00:00:59.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:58 smithi203 ceph-mon[31431]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:00:59.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:00:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1816031154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T00:00:59.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:58 smithi111 ceph-mon[31543]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:00:59.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:00:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1816031154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T00:00:59.568 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-04-22T00:00:59.568 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:00.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:00 smithi038 ceph-mon[27972]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:00 smithi203 ceph-mon[31431]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:01.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:00 smithi111 ceph-mon[31543]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 -- 172.21.15.38:0/4016940136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f38069f40 msgr2=0x7f9f3806a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 --2- 172.21.15.38:0/4016940136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f38069f40 0x7f9f3806a3a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009990 tx=0x7f9f2c009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 -- 172.21.15.38:0/4016940136 shutdown_connections 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 --2- 172.21.15.38:0/4016940136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f3806a8e0 0x7f9f3810f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 --2- 172.21.15.38:0/4016940136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f381082b0 0x7f9f38108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 --2- 172.21.15.38:0/4016940136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f38069f40 0x7f9f3806a3a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 -- 172.21.15.38:0/4016940136 >> 172.21.15.38:0/4016940136 conn(0x7f9f380780e0 msgr2=0x7f9f380784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 -- 172.21.15.38:0/4016940136 shutdown_connections 2024-04-22T00:01:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.511+0000 7f9f3db72700 1 -- 172.21.15.38:0/4016940136 wait complete. 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 Processor -- start 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 -- start start 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f38069f40 0x7f9f3819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 0x7f9f3819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38112580 con 0x7f9f381082b0 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38112400 con 0x7f9f3806a8e0 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f3db72700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38112700 con 0x7f9f38069f40 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.512+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f37fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50378/0 (socket says 172.21.15.38:50378) 2024-04-22T00:01:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f37fff700 1 -- 172.21.15.38:0/2896990800 learned_addr learned my addr 172.21.15.38:0/2896990800 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 0x7f9f3819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f377fe700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f38069f40 0x7f9f3819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 -- 172.21.15.38:0/2896990800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f38069f40 msgr2=0x7f9f3819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f38069f40 0x7f9f3819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 -- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 msgr2=0x7f9f381988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 -- 172.21.15.38:0/2896990800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f38199000 con 0x7f9f3806a8e0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f37fff700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f36ffd700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 0x7f9f3819ec20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00b2e0 tx=0x7f9f2c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c00be50 con 0x7f9f3806a8e0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f38199290 con 0x7f9f3806a8e0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f2c004830 con 0x7f9f3806a8e0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.513+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f381a55b0 con 0x7f9f3806a8e0 2024-04-22T00:01:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.514+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c005670 con 0x7f9f3806a8e0 2024-04-22T00:01:01.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.514+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f38068f20 con 0x7f9f3806a8e0 2024-04-22T00:01:01.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.516+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9f2c028060 con 0x7f9f3806a8e0 2024-04-22T00:01:01.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.516+0000 7f9f34ff9700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 0x7f9f200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:01.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.517+0000 7f9f377fe700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 0x7f9f200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:01.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.517+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f2c0d0220 con 0x7f9f3806a8e0 2024-04-22T00:01:01.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.517+0000 7f9f377fe700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 0x7f9f200a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9f381999e0 tx=0x7f9f28009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:01.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.520+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9f2c05c6f0 con 0x7f9f3806a8e0 2024-04-22T00:01:01.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.731+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f9f3802db30 con 0x7f9f3806a8e0 2024-04-22T00:01:01.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.731+0000 7f9f34ff9700 1 -- 172.21.15.38:0/2896990800 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f2c013070 con 0x7f9f3806a8e0 2024-04-22T00:01:01.724 INFO:teuthology.orchestra.run.smithi038.stdout:55834574929 2024-04-22T00:01:01.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 msgr2=0x7f9f200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:01.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 0x7f9f200a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9f381999e0 tx=0x7f9f28009260 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 msgr2=0x7f9f3819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 0x7f9f3819ec20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00b2e0 tx=0x7f9f2c00b840 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 shutdown_connections 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f9f200a2cb0 0x7f9f200a5170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f381082b0 0x7f9f381988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f38069f40 0x7f9f3819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 --2- 172.21.15.38:0/2896990800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f3806a8e0 0x7f9f3819ec20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 >> 172.21.15.38:0/2896990800 conn(0x7f9f380780e0 msgr2=0x7f9f380ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 shutdown_connections 2024-04-22T00:01:01.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.734+0000 7f9f3db72700 1 -- 172.21.15.38:0/2896990800 wait complete. 2024-04-22T00:01:01.904 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:01 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2896990800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T00:01:02.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:01 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2896990800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T00:01:02.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:01 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2896990800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T00:01:02.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574929 for osd.1 2024-04-22T00:01:02.412 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/3382864596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 msgr2=0x7f7abc1028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/3382864596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc1028f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00b0d0 tx=0x7f7aac00b3e0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/3382864596 shutdown_connections 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/3382864596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/3382864596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc1028f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/3382864596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/3382864596 >> 172.21.15.38:0/3382864596 conn(0x7f7abc0793e0 msgr2=0x7f7abc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/3382864596 shutdown_connections 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.698+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/3382864596 wait complete. 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 Processor -- start 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 -- start start 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc112640 con 0x7f7abc102510 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc1124c0 con 0x7f7abc102ec0 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac3f7e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc1127c0 con 0x7f7abc1075e0 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50394/0 (socket says 172.21.15.38:50394) 2024-04-22T00:01:02.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 -- 172.21.15.38:0/4252725491 learned_addr learned my addr 172.21.15.38:0/4252725491 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stdout:34359738455 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1519700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac251b700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 -- 172.21.15.38:0/4252725491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 msgr2=0x7f7abc074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 -- 172.21.15.38:0/4252725491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 msgr2=0x7f7abc074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:02.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1d1a700 1 -- 172.21.15.38:0/4252725491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc1a7060 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac251b700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.699+0000 7f7ac1519700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7ac1d1a700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00b820 tx=0x7f7aac004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac021020 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7abc1a72f0 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aac007d00 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac021020 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.700+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7abc1a77f0 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.701+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7abc068f20 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.703+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7aac024020 con 0x7f7abc102510 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.703+0000 7f7abaffd700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 0x7f7aa40a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:02.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.703+0000 7f7ac1519700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 0x7f7aa40a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.703+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7aac00f020 con 0x7f7abc102510 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.704+0000 7f7ac1519700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 0x7f7aa40a5370 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7abc108a10 tx=0x7f7ab0005f50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.708+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7aac09d010 con 0x7f7abc102510 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.923+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7abc02db30 con 0x7f7abc102510 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.923+0000 7f7abaffd700 1 -- 172.21.15.38:0/4252725491 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7aac09c910 con 0x7f7abc102510 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 msgr2=0x7f7aa40a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 0x7f7aa40a5370 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7abc108a10 tx=0x7f7ab0005f50 comp rx=0 tx=0).stop 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 msgr2=0x7f7abc074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00b820 tx=0x7f7aac004ed0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 shutdown_connections 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7aa40a2eb0 0x7f7aa40a5370 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7abc1075e0 0x7f7abc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7abc102510 0x7f7abc074530 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 --2- 172.21.15.38:0/4252725491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7abc102ec0 0x7f7abc074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:02.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 >> 172.21.15.38:0/4252725491 conn(0x7f7abc0793e0 msgr2=0x7f7abc100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:02.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 shutdown_connections 2024-04-22T00:01:02.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:01.925+0000 7f7ac3f7e700 1 -- 172.21.15.38:0/4252725491 wait complete. 2024-04-22T00:01:03.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:02 smithi203 ceph-mon[31431]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:03.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:02 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4252725491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T00:01:03.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:02 smithi111 ceph-mon[31543]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:03.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:02 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4252725491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T00:01:03.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:02 smithi038 ceph-mon[27972]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:03.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:02 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4252725491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T00:01:04.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738455 for osd.0 2024-04-22T00:01:04.607 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:04.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:04 smithi038 ceph-mon[27972]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:05.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:04 smithi203 ceph-mon[31431]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:05.040 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182465 got 107374182470 for osd.3 2024-04-22T00:01:05.040 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:05.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:04 smithi111 ceph-mon[31543]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:06.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:05 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/664314446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T00:01:06.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:05 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/664314446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T00:01:06.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:05 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/664314446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T00:01:06.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.714+0000 7f9971646700 1 -- 172.21.15.38:0/1163065464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c106480 msgr2=0x7f996c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.714+0000 7f9971646700 1 --2- 172.21.15.38:0/1163065464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c106480 0x7f996c10e050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f995c00b0d0 tx=0x7f995c00b3e0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 -- 172.21.15.38:0/1163065464 shutdown_connections 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 --2- 172.21.15.38:0/1163065464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c106480 0x7f996c10e050 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 --2- 172.21.15.38:0/1163065464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f996c1013c0 0x7f996c1017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 --2- 172.21.15.38:0/1163065464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c101d70 0x7f996c105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 -- 172.21.15.38:0/1163065464 >> 172.21.15.38:0/1163065464 conn(0x7f996c0771d0 msgr2=0x7f996c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 -- 172.21.15.38:0/1163065464 shutdown_connections 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 -- 172.21.15.38:0/1163065464 wait complete. 2024-04-22T00:01:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 Processor -- start 2024-04-22T00:01:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.715+0000 7f9971646700 1 -- start start 2024-04-22T00:01:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c1013c0 0x7f996c1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f996c106480 0x7f996c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f996c114780 con 0x7f996c101d70 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f996c114600 con 0x7f996c106480 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9971646700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f996c114900 con 0x7f996c1013c0 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58420/0 (socket says 172.21.15.38:58420) 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 -- 172.21.15.38:0/4088874371 learned_addr learned my addr 172.21.15.38:0/4088874371 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996affd700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c1013c0 0x7f996c1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 -- 172.21.15.38:0/4088874371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c1013c0 msgr2=0x7f996c1a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c1013c0 0x7f996c1a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 -- 172.21.15.38:0/4088874371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f996c106480 msgr2=0x7f996c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f996c106480 0x7f996c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 -- 172.21.15.38:0/4088874371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c19b120 con 0x7f996c101d70 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f996a7fc700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f995400d980 tx=0x7f995400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.716+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954017070 con 0x7f996c101d70 2024-04-22T00:01:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.717+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996c19b410 con 0x7f996c101d70 2024-04-22T00:01:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.717+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9954009e10 con 0x7f996c101d70 2024-04-22T00:01:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.717+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995400f620 con 0x7f996c101d70 2024-04-22T00:01:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.717+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996c1a7680 con 0x7f996c101d70 2024-04-22T00:01:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.718+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f996c102880 con 0x7f996c101d70 2024-04-22T00:01:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.720+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9954010460 con 0x7f996c101d70 2024-04-22T00:01:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.720+0000 7f9963fff700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 0x7f99580a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.720+0000 7f996affd700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 0x7f99580a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.720+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f99540cefa0 con 0x7f996c101d70 2024-04-22T00:01:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.721+0000 7f996affd700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 0x7f99580a5150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f996c106280 tx=0x7f996400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:06.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.724+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f995409ba40 con 0x7f996c101d70 2024-04-22T00:01:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/3723965256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101af0 msgr2=0x7f0acc10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/3723965256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101af0 0x7f0acc10dfe0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400b0a0 tx=0x7f0ac400b3b0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/3723965256 shutdown_connections 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/3723965256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101af0 0x7f0acc10dfe0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/3723965256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc1071a0 0x7f0acc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/3723965256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc101130 0x7f0acc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/3723965256 >> 172.21.15.38:0/3723965256 conn(0x7f0acc0780e0 msgr2=0x7f0acc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/3723965256 shutdown_connections 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.915+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/3723965256 wait complete. 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 Processor -- start 2024-04-22T00:01:06.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 -- start start 2024-04-22T00:01:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc101af0 0x7f0acc103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc1071a0 0x7f0acc103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc111440 con 0x7f0acc101130 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1112c0 con 0x7f0acc101af0 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad3a2f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1115c0 con 0x7f0acc1071a0 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad17cb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad17cb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58440/0 (socket says 172.21.15.38:58440) 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad17cb700 1 -- 172.21.15.38:0/2224203643 learned_addr learned my addr 172.21.15.38:0/2224203643 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad0fca700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc101af0 0x7f0acc103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.916+0000 7f0ad1fcc700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc1071a0 0x7f0acc103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 -- 172.21.15.38:0/2224203643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc1071a0 msgr2=0x7f0acc103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc1071a0 0x7f0acc103890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 -- 172.21.15.38:0/2224203643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc101af0 msgr2=0x7f0acc103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc101af0 0x7f0acc103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 -- 172.21.15.38:0/2224203643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc1a6e60 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad17cb700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9a0 tx=0x7f0ac800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac800ca10 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac800cbb0 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc1a7150 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8018680 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.917+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1a7690 con 0x7f0acc101130 2024-04-22T00:01:06.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.918+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc068f20 con 0x7f0acc101130 2024-04-22T00:01:06.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.920+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0ac8010430 con 0x7f0acc101130 2024-04-22T00:01:06.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.920+0000 7f0ac27fc700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 0x7f0ab80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:06.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.921+0000 7f0ad0fca700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 0x7f0ab80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:06.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.921+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0ac80cfba0 con 0x7f0acc101130 2024-04-22T00:01:06.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.921+0000 7f0ad0fca700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 0x7f0ab80a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0ff850 tx=0x7f0abc00b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:06.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.924+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0ac809c5e0 con 0x7f0acc101130 2024-04-22T00:01:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.937+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f996c02db30 con 0x7f996c101d70 2024-04-22T00:01:06.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.937+0000 7f9963fff700 1 -- 172.21.15.38:0/4088874371 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f995409b340 con 0x7f996c101d70 2024-04-22T00:01:06.930 INFO:teuthology.orchestra.run.smithi038.stdout:343597383711 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 msgr2=0x7f99580a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 0x7f99580a5150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f996c106280 tx=0x7f996400b410 comp rx=0 tx=0).stop 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 msgr2=0x7f996c1a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f995400d980 tx=0x7f995400dc90 comp rx=0 tx=0).stop 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 shutdown_connections 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f99580a2c90 0x7f99580a5150 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f996c106480 0x7f996c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f996c1013c0 0x7f996c1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 --2- 172.21.15.38:0/4088874371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f996c101d70 0x7f996c1a0cf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:06.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 >> 172.21.15.38:0/4088874371 conn(0x7f996c0771d0 msgr2=0x7f996c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:06.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 shutdown_connections 2024-04-22T00:01:06.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:06.939+0000 7f9971646700 1 -- 172.21.15.38:0/4088874371 wait complete. 2024-04-22T00:01:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:06 smithi038 ceph-mon[27972]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:07.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:06 smithi203 ceph-mon[31431]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:07.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:06 smithi111 ceph-mon[31543]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:07.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.135+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0acc1a78f0 con 0x7f0acc101130 2024-04-22T00:01:07.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.135+0000 7f0ac27fc700 1 -- 172.21.15.38:0/2224203643 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0ac809bee0 con 0x7f0acc101130 2024-04-22T00:01:07.128 INFO:teuthology.orchestra.run.smithi038.stdout:163208757310 2024-04-22T00:01:07.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 msgr2=0x7f0ab80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 0x7f0ab80a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0ff850 tx=0x7f0abc00b3c0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 msgr2=0x7f0acc102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9a0 tx=0x7f0ac800ecb0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 shutdown_connections 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ab80a2cb0 0x7f0ab80a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0acc1071a0 0x7f0acc103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0acc101130 0x7f0acc102e10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 --2- 172.21.15.38:0/2224203643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0acc101af0 0x7f0acc103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.137+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 >> 172.21.15.38:0/2224203643 conn(0x7f0acc0780e0 msgr2=0x7f0acc1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.138+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 shutdown_connections 2024-04-22T00:01:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:07.138+0000 7f0ad3a2f700 1 -- 172.21.15.38:0/2224203643 wait complete. 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 -- 172.21.15.38:0/2049389789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101130 msgr2=0x7fa4941015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 --2- 172.21.15.38:0/2049389789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101130 0x7fa4941015b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa488009990 tx=0x7fa488009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 -- 172.21.15.38:0/2049389789 shutdown_connections 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 --2- 172.21.15.38:0/2049389789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494101af0 0x7fa49410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 --2- 172.21.15.38:0/2049389789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4941071a0 0x7fa494107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 --2- 172.21.15.38:0/2049389789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101130 0x7fa4941015b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 -- 172.21.15.38:0/2049389789 >> 172.21.15.38:0/2049389789 conn(0x7fa4940780e0 msgr2=0x7fa4940784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 -- 172.21.15.38:0/2049389789 shutdown_connections 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.242+0000 7fa49acc8700 1 -- 172.21.15.38:0/2049389789 wait complete. 2024-04-22T00:01:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 Processor -- start 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 -- start start 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 0x7fa49419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 0x7fa494198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa494111490 con 0x7fa494101af0 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa494111310 con 0x7fa494101130 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa49acc8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa494111610 con 0x7fa4941071a0 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58408/0 (socket says 172.21.15.38:58408) 2024-04-22T00:01:07.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 -- 172.21.15.38:0/664314446 learned_addr learned my addr 172.21.15.38:0/664314446 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:07.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa498a64700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 0x7fa49419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:07.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa499265700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 0x7fa494198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:07.955 INFO:teuthology.orchestra.run.smithi038.stdout:197568495673 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 -- 172.21.15.38:0/664314446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 msgr2=0x7fa494198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 0x7fa494198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 -- 172.21.15.38:0/664314446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 msgr2=0x7fa49419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 0x7fa49419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 -- 172.21.15.38:0/664314446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa494199030 con 0x7fa494101af0 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa499265700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 0x7fa494198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:07.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa498a64700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 0x7fa49419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.243+0000 7fa493fff700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bff0 tx=0x7fa488005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.244+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488027070 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.244+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4941992c0 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.244+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa488004150 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.244+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48800f460 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.244+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4941a5530 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.245+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa494102850 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.247+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa48802a070 con 0x7fa494101af0 2024-04-22T00:01:07.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.247+0000 7fa491ffb700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 0x7fa47c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.248+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa4880d0c50 con 0x7fa494101af0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.248+0000 7fa498a64700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 0x7fa47c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.248+0000 7fa498a64700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 0x7fa47c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa484005fd0 tx=0x7fa484005e20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.251+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa488031020 con 0x7fa494101af0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.463+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa4941a5700 con 0x7fa494101af0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.464+0000 7fa491ffb700 1 -- 172.21.15.38:0/664314446 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa488013070 con 0x7fa494101af0 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 msgr2=0x7fa47c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 0x7fa47c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa484005fd0 tx=0x7fa484005e20 comp rx=0 tx=0).stop 2024-04-22T00:01:07.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 msgr2=0x7fa49419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa48800bff0 tx=0x7fa488005640 comp rx=0 tx=0).stop 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 shutdown_connections 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa47c0a2cb0 0x7fa47c0a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4941071a0 0x7fa494198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa494101130 0x7fa49419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 --2- 172.21.15.38:0/664314446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa494101af0 0x7fa49419ec10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 >> 172.21.15.38:0/664314446 conn(0x7fa4940780e0 msgr2=0x7fa4940fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 shutdown_connections 2024-04-22T00:01:07.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:05.466+0000 7fa49acc8700 1 -- 172.21.15.38:0/664314446 wait complete. 2024-04-22T00:01:08.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4088874371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T00:01:08.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2224203643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T00:01:08.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4088874371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T00:01:08.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2224203643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T00:01:08.208 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4088874371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T00:01:08.208 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2224203643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T00:01:09.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:08 smithi111 ceph-mon[31543]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:09.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:08 smithi203 ceph-mon[31431]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:09.187 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:08 smithi038 ceph-mon[27972]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:09.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383711 for osd.11 2024-04-22T00:01:09.217 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:09.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 -- 172.21.15.38:0/3844413271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff006a7d0 msgr2=0x7fbff006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:09.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 --2- 172.21.15.38:0/3844413271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff006a7d0 0x7fbff006abb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009990 tx=0x7fbfe4009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 -- 172.21.15.38:0/3844413271 shutdown_connections 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 --2- 172.21.15.38:0/3844413271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff010ecd0 0x7fbff01110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 --2- 172.21.15.38:0/3844413271 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff006a7d0 0x7fbff006abb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 --2- 172.21.15.38:0/3844413271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006b0f0 0x7fbff010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 -- 172.21.15.38:0/3844413271 >> 172.21.15.38:0/3844413271 conn(0x7fbff00780a0 msgr2=0x7fbff00784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 -- 172.21.15.38:0/3844413271 shutdown_connections 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.953+0000 7fbff70dd700 1 -- 172.21.15.38:0/3844413271 wait complete. 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 Processor -- start 2024-04-22T00:01:09.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 -- start start 2024-04-22T00:01:09.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 0x7fbff01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff006b0f0 0x7fbff01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 0x7fbff019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0114730 con 0x7fbff010ecd0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01145b0 con 0x7fbff006b0f0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff70dd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01148b0 con 0x7fbff006a7d0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff567a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 0x7fbff019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff567a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 0x7fbff019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58456/0 (socket says 172.21.15.38:58456) 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff567a700 1 -- 172.21.15.38:0/2041691638 learned_addr learned my addr 172.21.15.38:0/2041691638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbff4e79700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 0x7fbff01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.954+0000 7fbfeffff700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff006b0f0 0x7fbff01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 -- 172.21.15.38:0/2041691638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff006b0f0 msgr2=0x7fbff01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:09.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff006b0f0 0x7fbff01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 -- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 msgr2=0x7fbff019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 0x7fbff019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 -- 172.21.15.38:0/2041691638 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff019b0d0 con 0x7fbff006a7d0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff4e79700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 0x7fbff01a07b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40059d0 tx=0x7fbfe40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4027070 con 0x7fbff006a7d0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff019b360 con 0x7fbff006a7d0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe4003740 con 0x7fbff006a7d0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.955+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4022b50 con 0x7fbff006a7d0 2024-04-22T00:01:09.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.956+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff01a7680 con 0x7fbff006a7d0 2024-04-22T00:01:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.956+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff0101740 con 0x7fbff006a7d0 2024-04-22T00:01:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.958+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbfe4022d10 con 0x7fbff006a7d0 2024-04-22T00:01:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.958+0000 7fbfedffb700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 0x7fbfe00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.959+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbfe40d0a40 con 0x7fbff006a7d0 2024-04-22T00:01:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.959+0000 7fbfeffff700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 0x7fbfe00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:09.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.959+0000 7fbfeffff700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 0x7fbfe00a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbff0102530 tx=0x7fbfdc0073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:09.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:09.962+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbfe4013070 con 0x7fbff006a7d0 2024-04-22T00:01:10.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.181+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbff019bea0 con 0x7fbff006a7d0 2024-04-22T00:01:10.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.182+0000 7fbfedffb700 1 -- 172.21.15.38:0/2041691638 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbfe409d510 con 0x7fbff006a7d0 2024-04-22T00:01:10.175 INFO:teuthology.orchestra.run.smithi038.stdout:141733920835 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 msgr2=0x7fbfe00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 0x7fbfe00a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbff0102530 tx=0x7fbfdc0073d0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 msgr2=0x7fbff01a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 0x7fbff01a07b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40059d0 tx=0x7fbfe40047b0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 shutdown_connections 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbfe00a2cb0 0x7fbfe00a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbff010ecd0 0x7fbff019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff006a7d0 0x7fbff01a07b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 --2- 172.21.15.38:0/2041691638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbff006b0f0 0x7fbff01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 >> 172.21.15.38:0/2041691638 conn(0x7fbff00780a0 msgr2=0x7fbff0110080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:10.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 shutdown_connections 2024-04-22T00:01:10.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.185+0000 7fbff70dd700 1 -- 172.21.15.38:0/2041691638 wait complete. 2024-04-22T00:01:10.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/1684636237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 msgr2=0x7fcbf8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/1684636237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf8106eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009990 tx=0x7fcbec009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/1684636237 shutdown_connections 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/1684636237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8107590 0x7fcbf810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/1684636237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8102510 0x7fcbf81028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/1684636237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf8106eb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/1684636237 >> 172.21.15.38:0/1684636237 conn(0x7fcbf80793a0 msgr2=0x7fcbf80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/1684636237 shutdown_connections 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.814+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/1684636237 wait complete. 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 Processor -- start 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 -- start start 2024-04-22T00:01:10.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8102510 0x7fcbf819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 0x7fcbf8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf81125b0 con 0x7fcbf8102ec0 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf8112430 con 0x7fcbf8107590 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbfdf0a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf8112730 con 0x7fcbf8102510 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.815+0000 7fcbf6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58470/0 (socket says 172.21.15.38:58470) 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 -- 172.21.15.38:0/2804041403 learned_addr learned my addr 172.21.15.38:0/2804041403 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf77fe700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8102510 0x7fcbf819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf7fff700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 0x7fcbf8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 -- 172.21.15.38:0/2804041403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8102510 msgr2=0x7fcbf819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8102510 0x7fcbf819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 -- 172.21.15.38:0/2804041403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 msgr2=0x7fcbf8198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:10.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 0x7fcbf8198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 -- 172.21.15.38:0/2804041403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf8198fd0 con 0x7fcbf8102ec0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf7fff700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 0x7fcbf8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf6ffd700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec027070 con 0x7fcbf8102ec0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf8199260 con 0x7fcbf8102ec0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbec003780 con 0x7fcbf8102ec0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec022db0 con 0x7fcbf8102ec0 2024-04-22T00:01:10.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.816+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf81a5580 con 0x7fcbf8102ec0 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.817+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf8068f20 con 0x7fcbf8102ec0 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.819+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcbec02a070 con 0x7fcbf8102ec0 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.820+0000 7fcbf4ff9700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 0x7fcbe00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.820+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcbec0d0250 con 0x7fcbf8102ec0 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.820+0000 7fcbf77fe700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 0x7fcbe00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:10.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.821+0000 7fcbf77fe700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 0x7fcbe00a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbf81999b0 tx=0x7fcbe800b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:10.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:10.826+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcbec013070 con 0x7fcbf8102ec0 2024-04-22T00:01:10.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:10 smithi038 ceph-mon[27972]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:10.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:10 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2041691638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T00:01:11.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1184849476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 msgr2=0x7f4a8406a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1184849476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8406a3a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009990 tx=0x7f4a74009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1184849476 shutdown_connections 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1184849476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 0x7f4a8410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1184849476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 0x7f4a84108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1184849476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8406a3a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.006+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1184849476 >> 172.21.15.38:0/1184849476 conn(0x7f4a840780e0 msgr2=0x7f4a840784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.007+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1184849476 shutdown_connections 2024-04-22T00:01:11.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.007+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1184849476 wait complete. 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 Processor -- start 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 -- start start 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 0x7f4a8419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 0x7f4a84198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84112560 con 0x7f4a84069f40 2024-04-22T00:01:11.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841123e0 con 0x7f4a841082b0 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a8b2b7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841126e0 con 0x7f4a8406a8e0 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a89053700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a89053700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58492/0 (socket says 172.21.15.38:58492) 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a89053700 1 -- 172.21.15.38:0/1619068992 learned_addr learned my addr 172.21.15.38:0/1619068992 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a88852700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 0x7f4a8419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.008+0000 7f4a89854700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 0x7f4a84198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 -- 172.21.15.38:0/1619068992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 msgr2=0x7f4a8419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 0x7f4a8419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 -- 172.21.15.38:0/1619068992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 msgr2=0x7f4a84198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 0x7f4a84198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 -- 172.21.15.38:0/1619068992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a84199000 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a89053700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000d950 tx=0x7f4a8000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80016070 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a8000fd70 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000e610 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a841992f0 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.009+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a841a54b0 con 0x7f4a84069f40 2024-04-22T00:01:11.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.010+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a84103960 con 0x7f4a84069f40 2024-04-22T00:01:11.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.012+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4a80019070 con 0x7f4a84069f40 2024-04-22T00:01:11.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.013+0000 7f4a7a7fc700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 0x7f4a700a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:11.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.013+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4a800cefb0 con 0x7f4a84069f40 2024-04-22T00:01:11.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.013+0000 7f4a88852700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 0x7f4a700a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:11.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.013+0000 7f4a88852700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 0x7f4a700a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4a84078670 tx=0x7f4a74005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:11.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.017+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4a8009b9f0 con 0x7f4a84069f40 2024-04-22T00:01:11.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.045+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fcbf802db30 con 0x7fcbf8102ec0 2024-04-22T00:01:11.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.045+0000 7fcbf4ff9700 1 -- 172.21.15.38:0/2804041403 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcbec09cd20 con 0x7fcbf8102ec0 2024-04-22T00:01:11.039 INFO:teuthology.orchestra.run.smithi038.stdout:309237645349 2024-04-22T00:01:11.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 msgr2=0x7fcbe00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 0x7fcbe00a51f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbf81999b0 tx=0x7fcbe800b410 comp rx=0 tx=0).stop 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 msgr2=0x7fcbf819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec0047b0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 shutdown_connections 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcbe00a2d30 0x7fcbe00a51f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8107590 0x7fcbf8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbf8102510 0x7fcbf819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 --2- 172.21.15.38:0/2804041403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf8102ec0 0x7fcbf819ebf0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 >> 172.21.15.38:0/2804041403 conn(0x7fcbf80793a0 msgr2=0x7fcbf8100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:11.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 shutdown_connections 2024-04-22T00:01:11.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.048+0000 7fcbfdf0a700 1 -- 172.21.15.38:0/2804041403 wait complete. 2024-04-22T00:01:11.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:10 smithi111 ceph-mon[31543]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:11.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:10 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2041691638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T00:01:11.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.228+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4a84199d00 con 0x7f4a84069f40 2024-04-22T00:01:11.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.228+0000 7f4a7a7fc700 1 -- 172.21.15.38:0/1619068992 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4a8009b2f0 con 0x7f4a84069f40 2024-04-22T00:01:11.221 INFO:teuthology.orchestra.run.smithi038.stdout:219043332148 2024-04-22T00:01:11.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 msgr2=0x7f4a700a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 0x7f4a700a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4a84078670 tx=0x7f4a74005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 msgr2=0x7f4a8419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:11.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000d950 tx=0x7f4a8000dc60 comp rx=0 tx=0).stop 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 shutdown_connections 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4a700a2d30 0x7f4a700a51f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a841082b0 0x7f4a84198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069f40 0x7f4a8419e650 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 --2- 172.21.15.38:0/1619068992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a8e0 0x7f4a8419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.230+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 >> 172.21.15.38:0/1619068992 conn(0x7f4a840780e0 msgr2=0x7f4a840ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.231+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 shutdown_connections 2024-04-22T00:01:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:11.231+0000 7f4a8b2b7700 1 -- 172.21.15.38:0/1619068992 wait complete. 2024-04-22T00:01:11.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:10 smithi203 ceph-mon[31431]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:11.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:10 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2041691638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T00:01:11.584 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757306 got 163208757310 for osd.5 2024-04-22T00:01:11.584 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:11.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2804041403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T00:01:11.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1619068992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T00:01:12.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2804041403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T00:01:12.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1619068992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T00:01:12.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2804041403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T00:01:12.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1619068992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T00:01:12.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645344 got 309237645349 for osd.10 2024-04-22T00:01:12.486 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:12 smithi038 ceph-mon[27972]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:13.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:12 smithi111 ceph-mon[31543]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:13.130 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920827 got 141733920835 for osd.4 2024-04-22T00:01:13.130 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:13.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:12 smithi203 ceph-mon[31431]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 -- 172.21.15.38:0/4166438524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 msgr2=0x7fa9dc106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 --2- 172.21.15.38:0/4166438524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc106e90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009990 tx=0x7fa9cc009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 -- 172.21.15.38:0/4166438524 shutdown_connections 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 --2- 172.21.15.38:0/4166438524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 --2- 172.21.15.38:0/4166438524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 --2- 172.21.15.38:0/4166438524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc106e90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.073+0000 7fa9e316d700 1 -- 172.21.15.38:0/4166438524 >> 172.21.15.38:0/4166438524 conn(0x7fa9dc079390 msgr2=0x7fa9dc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:14.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- 172.21.15.38:0/4166438524 shutdown_connections 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- 172.21.15.38:0/4166438524 wait complete. 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 Processor -- start 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- start start 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc112610 con 0x7fa9dc102e90 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc112490 con 0x7fa9dc107570 2024-04-22T00:01:14.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.074+0000 7fa9e316d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc112790 con 0x7fa9dc1024e0 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39132/0 (socket says 172.21.15.38:39132) 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 -- 172.21.15.38:0/2465412078 learned_addr learned my addr 172.21.15.38:0/2465412078 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9e170a700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9e0f09700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 -- 172.21.15.38:0/2465412078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 msgr2=0x7fa9dc074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 -- 172.21.15.38:0/2465412078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 msgr2=0x7fa9dc074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 -- 172.21.15.38:0/2465412078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc1a70b0 con 0x7fa9dc102e90 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9e0f09700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9e170a700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9dbfff700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc000c00 tx=0x7fa9cc003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc027070 con 0x7fa9dc102e90 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc1a7340 con 0x7fa9dc102e90 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc004190 con 0x7fa9dc102e90 2024-04-22T00:01:14.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.075+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00f460 con 0x7fa9dc102e90 2024-04-22T00:01:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.076+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9dc1a77f0 con 0x7fa9dc102e90 2024-04-22T00:01:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.076+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9dc103970 con 0x7fa9dc102e90 2024-04-22T00:01:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.078+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa9cc02a070 con 0x7fa9dc102e90 2024-04-22T00:01:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.079+0000 7fa9d9ffb700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 0x7fa9c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.079+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa9cc0d0850 con 0x7fa9dc102e90 2024-04-22T00:01:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.079+0000 7fa9e0f09700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 0x7fa9c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.079+0000 7fa9e0f09700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 0x7fa9c80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4007900 tx=0x7fa9c4008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.082+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa9cc013070 con 0x7fa9dc102e90 2024-04-22T00:01:14.107 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332143 got 219043332148 for osd.7 2024-04-22T00:01:14.108 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:14.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.289+0000 7f237cfa7700 1 -- 172.21.15.38:0/2174822598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23781071a0 msgr2=0x7f2378107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.289+0000 7f237cfa7700 1 --2- 172.21.15.38:0/2174822598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23781071a0 0x7f2378107580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2368009990 tx=0x7f2368009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 -- 172.21.15.38:0/2174822598 shutdown_connections 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 --2- 172.21.15.38:0/2174822598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2378101af0 0x7f237810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 --2- 172.21.15.38:0/2174822598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23781071a0 0x7f2378107580 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 --2- 172.21.15.38:0/2174822598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101130 0x7f23781015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 -- 172.21.15.38:0/2174822598 >> 172.21.15.38:0/2174822598 conn(0x7f23780780e0 msgr2=0x7f23780784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 -- 172.21.15.38:0/2174822598 shutdown_connections 2024-04-22T00:01:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 -- 172.21.15.38:0/2174822598 wait complete. 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 Processor -- start 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.290+0000 7f237cfa7700 1 -- start start 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101af0 0x7f237819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23781071a0 0x7f2378198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111490 con 0x7f2378101130 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111310 con 0x7f23781071a0 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237cfa7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2378111610 con 0x7f2378101af0 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39146/0 (socket says 172.21.15.38:39146) 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 -- 172.21.15.38:0/789454905 learned_addr learned my addr 172.21.15.38:0/789454905 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f2375d9b700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101af0 0x7f237819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f2376d9d700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23781071a0 0x7f2378198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 -- 172.21.15.38:0/789454905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101af0 msgr2=0x7f237819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101af0 0x7f237819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 -- 172.21.15.38:0/789454905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23781071a0 msgr2=0x7f2378198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23781071a0 0x7f2378198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.291+0000 7f237659c700 1 -- 172.21.15.38:0/789454905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2378199030 con 0x7f2378101130 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f237659c700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2368005910 tx=0x7f236800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368027070 con 0x7f2378101130 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23781992c0 con 0x7f2378101130 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2368003d70 con 0x7f2378101130 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236800f6c0 con 0x7f2378101130 2024-04-22T00:01:14.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.292+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2378107fb0 con 0x7f2378101130 2024-04-22T00:01:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.293+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2378102850 con 0x7f2378101130 2024-04-22T00:01:14.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.294+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f236802a070 con 0x7f2378101130 2024-04-22T00:01:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.295+0000 7f236f7fe700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 0x7f23640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.295+0000 7f2375d9b700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 0x7f23640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.295+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f23680d0b10 con 0x7f2378101130 2024-04-22T00:01:14.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.296+0000 7f2375d9b700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 0x7f23640a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f23780fe250 tx=0x7f236000b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:14.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.298+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa9dc02db30 con 0x7fa9dc102e90 2024-04-22T00:01:14.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.299+0000 7fa9d9ffb700 1 -- 172.21.15.38:0/2465412078 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa9cc09d380 con 0x7fa9dc102e90 2024-04-22T00:01:14.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.299+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2368031070 con 0x7f2378101130 2024-04-22T00:01:14.292 INFO:teuthology.orchestra.run.smithi038.stdout:253403070512 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 msgr2=0x7fa9c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 0x7fa9c80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4007900 tx=0x7fa9c4008040 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 msgr2=0x7fa9dc074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc000c00 tx=0x7fa9cc003730 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 shutdown_connections 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa9c80a2cb0 0x7fa9c80a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9dc107570 0x7fa9dc074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9dc1024e0 0x7fa9dc074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 --2- 172.21.15.38:0/2465412078 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9dc102e90 0x7fa9dc074a70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 >> 172.21.15.38:0/2465412078 conn(0x7fa9dc079390 msgr2=0x7fa9dc100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 shutdown_connections 2024-04-22T00:01:14.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.301+0000 7fa9e316d700 1 -- 172.21.15.38:0/2465412078 wait complete. 2024-04-22T00:01:14.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.514+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f2378199d80 con 0x7f2378101130 2024-04-22T00:01:14.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.514+0000 7f236f7fe700 1 -- 172.21.15.38:0/789454905 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2368022e20 con 0x7f2378101130 2024-04-22T00:01:14.507 INFO:teuthology.orchestra.run.smithi038.stdout:287762808875 2024-04-22T00:01:14.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 msgr2=0x7f23640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 0x7f23640a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f23780fe250 tx=0x7f236000b3c0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 msgr2=0x7f237819e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2368005910 tx=0x7f236800bab0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 shutdown_connections 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f23640a2cb0 0x7f23640a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23781071a0 0x7f2378198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2378101130 0x7f237819e6d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 --2- 172.21.15.38:0/789454905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2378101af0 0x7f237819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.516+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 >> 172.21.15.38:0/789454905 conn(0x7f23780780e0 msgr2=0x7f23780fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.517+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 shutdown_connections 2024-04-22T00:01:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:14.517+0000 7f237cfa7700 1 -- 172.21.15.38:0/789454905 wait complete. 2024-04-22T00:01:14.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495673 for osd.6 2024-04-22T00:01:14.921 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:15.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:14 smithi111 ceph-mon[31543]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:15.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2465412078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T00:01:15.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/789454905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T00:01:15.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:14 smithi038 ceph-mon[27972]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:15.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2465412078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T00:01:15.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/789454905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T00:01:15.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:14 smithi203 ceph-mon[31431]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:15.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2465412078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T00:01:15.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/789454905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T00:01:15.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070512 for osd.8 2024-04-22T00:01:15.621 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:16.040 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808869 got 287762808875 for osd.9 2024-04-22T00:01:16.041 DEBUG:teuthology.parallel:result is None 2024-04-22T00:01:16.041 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T00:01:16.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T00:01:16.316 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:01:17.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:17 smithi038 ceph-mon[27972]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:01:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:01:17.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:17 smithi203 ceph-mon[31431]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:17.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:17.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:01:17.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:01:17.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:17 smithi111 ceph-mon[31543]: from='mgr.14148 172.21.15.38:0/2079114760' entity='mgr.x' 2024-04-22T00:01:18.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 -- 172.21.15.38:0/2723350249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101130 msgr2=0x7f55b01015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/2723350249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101130 0x7f55b01015b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f559c009990 tx=0x7f559c009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 -- 172.21.15.38:0/2723350249 shutdown_connections 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/2723350249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101af0 0x7f55b010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/2723350249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 0x7f55b0107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/2723350249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101130 0x7f55b01015b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 -- 172.21.15.38:0/2723350249 >> 172.21.15.38:0/2723350249 conn(0x7f55b00780e0 msgr2=0x7f55b00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 -- 172.21.15.38:0/2723350249 shutdown_connections 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.065+0000 7f55b6dd0700 1 -- 172.21.15.38:0/2723350249 wait complete. 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 Processor -- start 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 -- start start 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101130 0x7f55b019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:18.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 0x7f55b01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f55b0111410 con 0x7f55b0101af0 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f55b0111290 con 0x7f55b01071a0 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55b6dd0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55b0111590 con 0x7f55b0101130 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39172/0 (socket says 172.21.15.38:39172) 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.066+0000 7f55affff700 1 -- 172.21.15.38:0/1695472155 learned_addr learned my addr 172.21.15.38:0/1695472155 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55b536d700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 0x7f55b01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55b4b6c700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101130 0x7f55b019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 -- 172.21.15.38:0/1695472155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101130 msgr2=0x7f55b019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101130 0x7f55b019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 -- 172.21.15.38:0/1695472155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 msgr2=0x7f55b01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 0x7f55b01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 -- 172.21.15.38:0/1695472155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b0198f90 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55affff700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f559c0038b0 tx=0x7f559c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c027070 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c004050 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b0199220 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c022c00 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.067+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b01a5490 con 0x7f55b0101af0 2024-04-22T00:01:18.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.068+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b0102850 con 0x7f55b0101af0 2024-04-22T00:01:18.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.070+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f559c02a070 con 0x7f55b0101af0 2024-04-22T00:01:18.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.070+0000 7f55adffb700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 0x7f55a00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:18.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.071+0000 7f55b4b6c700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 0x7f55a00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:18.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.071+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f559c0d0b40 con 0x7f55b0101af0 2024-04-22T00:01:18.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.071+0000 7f55b4b6c700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 0x7f55a00a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f55a4005950 tx=0x7f55a400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:18.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.074+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f559c031020 con 0x7f55b0101af0 2024-04-22T00:01:18.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.277+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f55b0069f30 con 0x7f55a00a2d30 2024-04-22T00:01:18.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.287+0000 7f55adffb700 1 -- 172.21.15.38:0/1695472155 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340411 (secure 0 0 0) 0x7f55b0069f30 con 0x7f55a00a2d30 2024-04-22T00:01:18.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:18.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 msgr2=0x7f55a00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:18.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 0x7f55a00a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f55a4005950 tx=0x7f55a400b410 comp rx=0 tx=0).stop 2024-04-22T00:01:18.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 msgr2=0x7f55b019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f559c0038b0 tx=0x7f559c009f50 comp rx=0 tx=0).stop 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 shutdown_connections 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f55a00a2d30 0x7f55a00a51f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55b01071a0 0x7f55b01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55b0101130 0x7f55b019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.290+0000 7f55b6dd0700 1 --2- 172.21.15.38:0/1695472155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55b0101af0 0x7f55b019eb70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.291+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 >> 172.21.15.38:0/1695472155 conn(0x7f55b00780e0 msgr2=0x7f55b01002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.291+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 shutdown_connections 2024-04-22T00:01:18.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:18.291+0000 7f55b6dd0700 1 -- 172.21.15.38:0/1695472155 wait complete. 2024-04-22T00:01:18.292 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:01:18.982 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-22T00:01:17.675095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660139,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":886,"num_read_kb":572,"num_write":426,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181082,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363452,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578372,"statfs":{"total":1151940427776,"available":1151568252928,"internally_reserved":0,"allocated":14581760,"data_stored":7966223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003604"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356786+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-22T00:00:27.356786+0000","last_peered":"2024-04-22T00:00:27.356786+0000","last_clean":"2024-04-22T00:00:27.356786+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-22T00:00:27.356786+0000","last_undegraded":"2024-04-22T00:00:27.356786+0000","last_fullsized":"2024-04-22T00:00:27.356786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357853+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-22T00:00:27.357853+0000","last_peered":"2024-04-22T00:00:27.357853+0000","last_clean":"2024-04-22T00:00:27.357853+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-22T00:00:27.357853+0000","last_undegraded":"2024-04-22T00:00:27.357853+0000","last_fullsized":"2024-04-22T00:00:27.357853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357305+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-22T00:00:27.357305+0000","last_peered":"2024-04-22T00:00:27.357305+0000","last_clean":"2024-04-22T00:00:27.357305+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-22T00:00:27.357305+0000","last_undegraded":"2024-04-22T00:00:27.357305+0000","last_fullsized":"2024-04-22T00:00:27.357305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359224+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-22T00:00:27.359224+0000","last_peered":"2024-04-22T00:00:27.359224+0000","last_clean":"2024-04-22T00:00:27.359224+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-22T00:00:27.359224+0000","last_undegraded":"2024-04-22T00:00:27.359224+0000","last_fullsized":"2024-04-22T00:00:27.359224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354642+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-22T00:00:27.354642+0000","last_peered":"2024-04-22T00:00:27.354642+0000","last_clean":"2024-04-22T00:00:27.354642+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-22T00:00:27.354642+0000","last_undegraded":"2024-04-22T00:00:27.354642+0000","last_fullsized":"2024-04-22T00:00:27.354642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491579+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-22T00:00:27.491579+0000","last_peered":"2024-04-22T00:00:27.491579+0000","last_clean":"2024-04-22T00:00:27.491579+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-22T00:00:27.491579+0000","last_undegraded":"2024-04-22T00:00:27.491579+0000","last_fullsized":"2024-04-22T00:00:27.491579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356054+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-22T00:00:27.356054+0000","last_peered":"2024-04-22T00:00:27.356054+0000","last_clean":"2024-04-22T00:00:27.356054+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-22T00:00:27.356054+0000","last_undegraded":"2024-04-22T00:00:27.356054+0000","last_fullsized":"2024-04-22T00:00:27.356054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490436+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-22T00:00:27.490436+0000","last_peered":"2024-04-22T00:00:27.490436+0000","last_clean":"2024-04-22T00:00:27.490436+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-22T00:00:27.490436+0000","last_undegraded":"2024-04-22T00:00:27.490436+0000","last_fullsized":"2024-04-22T00:00:27.490436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355228+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-22T00:00:27.355228+0000","last_peered":"2024-04-22T00:00:27.355228+0000","last_clean":"2024-04-22T00:00:27.355228+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-22T00:00:27.355228+0000","last_undegraded":"2024-04-22T00:00:27.355228+0000","last_fullsized":"2024-04-22T00:00:27.355228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491800+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-22T00:00:27.491800+0000","last_peered":"2024-04-22T00:00:27.491800+0000","last_clean":"2024-04-22T00:00:27.491800+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-22T00:00:27.491800+0000","last_undegraded":"2024-04-22T00:00:27.491800+0000","last_fullsized":"2024-04-22T00:00:27.491800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357900+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-22T00:00:27.357900+0000","last_peered":"2024-04-22T00:00:27.357900+0000","last_clean":"2024-04-22T00:00:27.357900+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-22T00:00:27.357900+0000","last_undegraded":"2024-04-22T00:00:27.357900+0000","last_fullsized":"2024-04-22T00:00:27.357900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359175+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-22T00:00:27.359175+0000","last_peered":"2024-04-22T00:00:27.359175+0000","last_clean":"2024-04-22T00:00:27.359175+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-22T00:00:27.359175+0000","last_undegraded":"2024-04-22T00:00:27.359175+0000","last_fullsized":"2024-04-22T00:00:27.359175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492882+0000","last_change":"2024-04-22T00:00:27.492882+0000","last_active":"2024-04-22T00:00:27.492882+0000","last_peered":"2024-04-22T00:00:27.492882+0000","last_clean":"2024-04-22T00:00:27.492882+0000","last_became_active":"2024-04-22T00:00:27.492451+0000","last_became_peered":"2024-04-22T00:00:27.492451+0000","last_unstale":"2024-04-22T00:00:27.492882+0000","last_undegraded":"2024-04-22T00:00:27.492882+0000","last_fullsized":"2024-04-22T00:00:27.492882+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:10.283575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357546+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-22T00:00:27.357546+0000","last_peered":"2024-04-22T00:00:27.357546+0000","last_clean":"2024-04-22T00:00:27.357546+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-22T00:00:27.357546+0000","last_undegraded":"2024-04-22T00:00:27.357546+0000","last_fullsized":"2024-04-22T00:00:27.357546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357367+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-22T00:00:27.357367+0000","last_peered":"2024-04-22T00:00:27.357367+0000","last_clean":"2024-04-22T00:00:27.357367+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-22T00:00:27.357367+0000","last_undegraded":"2024-04-22T00:00:27.357367+0000","last_fullsized":"2024-04-22T00:00:27.357367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354448+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-22T00:00:27.354448+0000","last_peered":"2024-04-22T00:00:27.354448+0000","last_clean":"2024-04-22T00:00:27.354448+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-22T00:00:27.354448+0000","last_undegraded":"2024-04-22T00:00:27.354448+0000","last_fullsized":"2024-04-22T00:00:27.354448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356396+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-22T00:00:27.356396+0000","last_peered":"2024-04-22T00:00:27.356396+0000","last_clean":"2024-04-22T00:00:27.356396+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-22T00:00:27.356396+0000","last_undegraded":"2024-04-22T00:00:27.356396+0000","last_fullsized":"2024-04-22T00:00:27.356396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353369+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-22T00:00:27.353369+0000","last_peered":"2024-04-22T00:00:27.353369+0000","last_clean":"2024-04-22T00:00:27.353369+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-22T00:00:27.353369+0000","last_undegraded":"2024-04-22T00:00:27.353369+0000","last_fullsized":"2024-04-22T00:00:27.353369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352831+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-22T00:00:27.352831+0000","last_peered":"2024-04-22T00:00:27.352831+0000","last_clean":"2024-04-22T00:00:27.352831+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-22T00:00:27.352831+0000","last_undegraded":"2024-04-22T00:00:27.352831+0000","last_fullsized":"2024-04-22T00:00:27.352831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352249+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-22T00:00:27.352249+0000","last_peered":"2024-04-22T00:00:27.352249+0000","last_clean":"2024-04-22T00:00:27.352249+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-22T00:00:27.352249+0000","last_undegraded":"2024-04-22T00:00:27.352249+0000","last_fullsized":"2024-04-22T00:00:27.352249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:14.375260+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:01:14.375260+0000","last_peered":"2024-04-22T00:01:14.375260+0000","last_clean":"2024-04-22T00:01:14.375260+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:01:14.375260+0000","last_undegraded":"2024-04-22T00:01:14.375260+0000","last_fullsized":"2024-04-22T00:01:14.375260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491009+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-22T00:00:27.491009+0000","last_peered":"2024-04-22T00:00:27.491009+0000","last_clean":"2024-04-22T00:00:27.491009+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-22T00:00:27.491009+0000","last_undegraded":"2024-04-22T00:00:27.491009+0000","last_fullsized":"2024-04-22T00:00:27.491009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356909+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-22T00:00:27.356909+0000","last_peered":"2024-04-22T00:00:27.356909+0000","last_clean":"2024-04-22T00:00:27.356909+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-22T00:00:27.356909+0000","last_undegraded":"2024-04-22T00:00:27.356909+0000","last_fullsized":"2024-04-22T00:00:27.356909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491065+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-22T00:00:27.491065+0000","last_peered":"2024-04-22T00:00:27.491065+0000","last_clean":"2024-04-22T00:00:27.491065+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-22T00:00:27.491065+0000","last_undegraded":"2024-04-22T00:00:27.491065+0000","last_fullsized":"2024-04-22T00:00:27.491065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358173+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-22T00:00:27.358173+0000","last_peered":"2024-04-22T00:00:27.358173+0000","last_clean":"2024-04-22T00:00:27.358173+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-22T00:00:27.358173+0000","last_undegraded":"2024-04-22T00:00:27.358173+0000","last_fullsized":"2024-04-22T00:00:27.358173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492457+0000","last_change":"2024-04-22T00:00:27.492457+0000","last_active":"2024-04-22T00:00:27.492457+0000","last_peered":"2024-04-22T00:00:27.492457+0000","last_clean":"2024-04-22T00:00:27.492457+0000","last_became_active":"2024-04-22T00:00:27.491828+0000","last_became_peered":"2024-04-22T00:00:27.491828+0000","last_unstale":"2024-04-22T00:00:27.492457+0000","last_undegraded":"2024-04-22T00:00:27.492457+0000","last_fullsized":"2024-04-22T00:00:27.492457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:09:18.512103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354779+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:00:27.354779+0000","last_peered":"2024-04-22T00:00:27.354779+0000","last_clean":"2024-04-22T00:00:27.354779+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:00:27.354779+0000","last_undegraded":"2024-04-22T00:00:27.354779+0000","last_fullsized":"2024-04-22T00:00:27.354779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354584+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-22T00:00:27.354584+0000","last_peered":"2024-04-22T00:00:27.354584+0000","last_clean":"2024-04-22T00:00:27.354584+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-22T00:00:27.354584+0000","last_undegraded":"2024-04-22T00:00:27.354584+0000","last_fullsized":"2024-04-22T00:00:27.354584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357308+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-22T00:00:27.357308+0000","last_peered":"2024-04-22T00:00:27.357308+0000","last_clean":"2024-04-22T00:00:27.357308+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-22T00:00:27.357308+0000","last_undegraded":"2024-04-22T00:00:27.357308+0000","last_fullsized":"2024-04-22T00:00:27.357308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355422+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-22T00:00:27.355422+0000","last_peered":"2024-04-22T00:00:27.355422+0000","last_clean":"2024-04-22T00:00:27.355422+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-22T00:00:27.355422+0000","last_undegraded":"2024-04-22T00:00:27.355422+0000","last_fullsized":"2024-04-22T00:00:27.355422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360370+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-22T00:00:27.360370+0000","last_peered":"2024-04-22T00:00:27.360370+0000","last_clean":"2024-04-22T00:00:27.360370+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-22T00:00:27.360370+0000","last_undegraded":"2024-04-22T00:00:27.360370+0000","last_fullsized":"2024-04-22T00:00:27.360370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352694+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-22T00:00:27.352694+0000","last_peered":"2024-04-22T00:00:27.352694+0000","last_clean":"2024-04-22T00:00:27.352694+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-22T00:00:27.352694+0000","last_undegraded":"2024-04-22T00:00:27.352694+0000","last_fullsized":"2024-04-22T00:00:27.352694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:13.375360+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:01:13.375360+0000","last_peered":"2024-04-22T00:01:13.375360+0000","last_clean":"2024-04-22T00:01:13.375360+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:01:13.375360+0000","last_undegraded":"2024-04-22T00:01:13.375360+0000","last_fullsized":"2024-04-22T00:01:13.375360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355075+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-22T00:00:27.355075+0000","last_peered":"2024-04-22T00:00:27.355075+0000","last_clean":"2024-04-22T00:00:27.355075+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-22T00:00:27.355075+0000","last_undegraded":"2024-04-22T00:00:27.355075+0000","last_fullsized":"2024-04-22T00:00:27.355075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352432+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-22T00:00:27.352432+0000","last_peered":"2024-04-22T00:00:27.352432+0000","last_clean":"2024-04-22T00:00:27.352432+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-22T00:00:27.352432+0000","last_undegraded":"2024-04-22T00:00:27.352432+0000","last_fullsized":"2024-04-22T00:00:27.352432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491319+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-22T00:00:27.491319+0000","last_peered":"2024-04-22T00:00:27.491319+0000","last_clean":"2024-04-22T00:00:27.491319+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-22T00:00:27.491319+0000","last_undegraded":"2024-04-22T00:00:27.491319+0000","last_fullsized":"2024-04-22T00:00:27.491319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360197+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-22T00:00:27.360197+0000","last_peered":"2024-04-22T00:00:27.360197+0000","last_clean":"2024-04-22T00:00:27.360197+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-22T00:00:27.360197+0000","last_undegraded":"2024-04-22T00:00:27.360197+0000","last_fullsized":"2024-04-22T00:00:27.360197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356273+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-22T00:00:27.356273+0000","last_peered":"2024-04-22T00:00:27.356273+0000","last_clean":"2024-04-22T00:00:27.356273+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-22T00:00:27.356273+0000","last_undegraded":"2024-04-22T00:00:27.356273+0000","last_fullsized":"2024-04-22T00:00:27.356273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355336+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-22T00:00:27.355336+0000","last_peered":"2024-04-22T00:00:27.355336+0000","last_clean":"2024-04-22T00:00:27.355336+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-22T00:00:27.355336+0000","last_undegraded":"2024-04-22T00:00:27.355336+0000","last_fullsized":"2024-04-22T00:00:27.355336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490300+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-22T00:00:27.490300+0000","last_peered":"2024-04-22T00:00:27.490300+0000","last_clean":"2024-04-22T00:00:27.490300+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-22T00:00:27.490300+0000","last_undegraded":"2024-04-22T00:00:27.490300+0000","last_fullsized":"2024-04-22T00:00:27.490300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357732+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-22T00:00:27.357732+0000","last_peered":"2024-04-22T00:00:27.357732+0000","last_clean":"2024-04-22T00:00:27.357732+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-22T00:00:27.357732+0000","last_undegraded":"2024-04-22T00:00:27.357732+0000","last_fullsized":"2024-04-22T00:00:27.357732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360631+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:00:27.360631+0000","last_peered":"2024-04-22T00:00:27.360631+0000","last_clean":"2024-04-22T00:00:27.360631+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:00:27.360631+0000","last_undegraded":"2024-04-22T00:00:27.360631+0000","last_fullsized":"2024-04-22T00:00:27.360631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352579+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-22T00:00:27.352579+0000","last_peered":"2024-04-22T00:00:27.352579+0000","last_clean":"2024-04-22T00:00:27.352579+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-22T00:00:27.352579+0000","last_undegraded":"2024-04-22T00:00:27.352579+0000","last_fullsized":"2024-04-22T00:00:27.352579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490506+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-22T00:00:27.490506+0000","last_peered":"2024-04-22T00:00:27.490506+0000","last_clean":"2024-04-22T00:00:27.490506+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-22T00:00:27.490506+0000","last_undegraded":"2024-04-22T00:00:27.490506+0000","last_fullsized":"2024-04-22T00:00:27.490506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378673+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:01:16.378673+0000","last_peered":"2024-04-22T00:01:16.378673+0000","last_clean":"2024-04-22T00:01:16.378673+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:01:16.378673+0000","last_undegraded":"2024-04-22T00:01:16.378673+0000","last_fullsized":"2024-04-22T00:01:16.378673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355061+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-22T00:00:27.355061+0000","last_peered":"2024-04-22T00:00:27.355061+0000","last_clean":"2024-04-22T00:00:27.355061+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-22T00:00:27.355061+0000","last_undegraded":"2024-04-22T00:00:27.355061+0000","last_fullsized":"2024-04-22T00:00:27.355061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352388+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-22T00:00:27.352388+0000","last_peered":"2024-04-22T00:00:27.352388+0000","last_clean":"2024-04-22T00:00:27.352388+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-22T00:00:27.352388+0000","last_undegraded":"2024-04-22T00:00:27.352388+0000","last_fullsized":"2024-04-22T00:00:27.352388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356489+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:00:27.356489+0000","last_peered":"2024-04-22T00:00:27.356489+0000","last_clean":"2024-04-22T00:00:27.356489+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:00:27.356489+0000","last_undegraded":"2024-04-22T00:00:27.356489+0000","last_fullsized":"2024-04-22T00:00:27.356489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358076+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-22T00:00:27.358076+0000","last_peered":"2024-04-22T00:00:27.358076+0000","last_clean":"2024-04-22T00:00:27.358076+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-22T00:00:27.358076+0000","last_undegraded":"2024-04-22T00:00:27.358076+0000","last_fullsized":"2024-04-22T00:00:27.358076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357668+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-22T00:00:27.357668+0000","last_peered":"2024-04-22T00:00:27.357668+0000","last_clean":"2024-04-22T00:00:27.357668+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-22T00:00:27.357668+0000","last_undegraded":"2024-04-22T00:00:27.357668+0000","last_fullsized":"2024-04-22T00:00:27.357668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360162+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:00:27.360162+0000","last_peered":"2024-04-22T00:00:27.360162+0000","last_clean":"2024-04-22T00:00:27.360162+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:00:27.360162+0000","last_undegraded":"2024-04-22T00:00:27.360162+0000","last_fullsized":"2024-04-22T00:00:27.360162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355018+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-22T00:00:27.355018+0000","last_peered":"2024-04-22T00:00:27.355018+0000","last_clean":"2024-04-22T00:00:27.355018+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-22T00:00:27.355018+0000","last_undegraded":"2024-04-22T00:00:27.355018+0000","last_fullsized":"2024-04-22T00:00:27.355018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352622+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-22T00:00:27.352622+0000","last_peered":"2024-04-22T00:00:27.352622+0000","last_clean":"2024-04-22T00:00:27.352622+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-22T00:00:27.352622+0000","last_undegraded":"2024-04-22T00:00:27.352622+0000","last_fullsized":"2024-04-22T00:00:27.352622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":214,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378457+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:01:16.378457+0000","last_peered":"2024-04-22T00:01:16.378457+0000","last_clean":"2024-04-22T00:01:16.378457+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:01:16.378457+0000","last_undegraded":"2024-04-22T00:01:16.378457+0000","last_fullsized":"2024-04-22T00:01:16.378457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":137,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355154+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-22T00:00:27.355154+0000","last_peered":"2024-04-22T00:00:27.355154+0000","last_clean":"2024-04-22T00:00:27.355154+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-22T00:00:27.355154+0000","last_undegraded":"2024-04-22T00:00:27.355154+0000","last_fullsized":"2024-04-22T00:00:27.355154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361264+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-22T00:00:27.361264+0000","last_peered":"2024-04-22T00:00:27.361264+0000","last_clean":"2024-04-22T00:00:27.361264+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-22T00:00:27.361264+0000","last_undegraded":"2024-04-22T00:00:27.361264+0000","last_fullsized":"2024-04-22T00:00:27.361264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352497+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:00:27.352497+0000","last_peered":"2024-04-22T00:00:27.352497+0000","last_clean":"2024-04-22T00:00:27.352497+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:00:27.352497+0000","last_undegraded":"2024-04-22T00:00:27.352497+0000","last_fullsized":"2024-04-22T00:00:27.352497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354975+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-22T00:00:27.354975+0000","last_peered":"2024-04-22T00:00:27.354975+0000","last_clean":"2024-04-22T00:00:27.354975+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-22T00:00:27.354975+0000","last_undegraded":"2024-04-22T00:00:27.354975+0000","last_fullsized":"2024-04-22T00:00:27.354975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355607+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-22T00:00:27.355607+0000","last_peered":"2024-04-22T00:00:27.355607+0000","last_clean":"2024-04-22T00:00:27.355607+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-22T00:00:27.355607+0000","last_undegraded":"2024-04-22T00:00:27.355607+0000","last_fullsized":"2024-04-22T00:00:27.355607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359646+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:00:27.359646+0000","last_peered":"2024-04-22T00:00:27.359646+0000","last_clean":"2024-04-22T00:00:27.359646+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:00:27.359646+0000","last_undegraded":"2024-04-22T00:00:27.359646+0000","last_fullsized":"2024-04-22T00:00:27.359646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360979+0000","last_change":"2024-04-22T00:00:27.360979+0000","last_active":"2024-04-22T00:00:27.360979+0000","last_peered":"2024-04-22T00:00:27.360979+0000","last_clean":"2024-04-22T00:00:27.360979+0000","last_became_active":"2024-04-22T00:00:27.360589+0000","last_became_peered":"2024-04-22T00:00:27.360589+0000","last_unstale":"2024-04-22T00:00:27.360979+0000","last_undegraded":"2024-04-22T00:00:27.360979+0000","last_fullsized":"2024-04-22T00:00:27.360979+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:25.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353018+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-22T00:00:27.353018+0000","last_peered":"2024-04-22T00:00:27.353018+0000","last_clean":"2024-04-22T00:00:27.353018+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-22T00:00:27.353018+0000","last_undegraded":"2024-04-22T00:00:27.353018+0000","last_fullsized":"2024-04-22T00:00:27.353018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:14.375988+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-22T00:01:14.375988+0000","last_peered":"2024-04-22T00:01:14.375988+0000","last_clean":"2024-04-22T00:01:14.375988+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-22T00:01:14.375988+0000","last_undegraded":"2024-04-22T00:01:14.375988+0000","last_fullsized":"2024-04-22T00:01:14.375988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355982+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-22T00:00:27.355982+0000","last_peered":"2024-04-22T00:00:27.355982+0000","last_clean":"2024-04-22T00:00:27.355982+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-22T00:00:27.355982+0000","last_undegraded":"2024-04-22T00:00:27.355982+0000","last_fullsized":"2024-04-22T00:00:27.355982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356239+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-22T00:00:27.356239+0000","last_peered":"2024-04-22T00:00:27.356239+0000","last_clean":"2024-04-22T00:00:27.356239+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-22T00:00:27.356239+0000","last_undegraded":"2024-04-22T00:00:27.356239+0000","last_fullsized":"2024-04-22T00:00:27.356239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354526+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:00:27.354526+0000","last_peered":"2024-04-22T00:00:27.354526+0000","last_clean":"2024-04-22T00:00:27.354526+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:00:27.354526+0000","last_undegraded":"2024-04-22T00:00:27.354526+0000","last_fullsized":"2024-04-22T00:00:27.354526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359881+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-22T00:00:27.359881+0000","last_peered":"2024-04-22T00:00:27.359881+0000","last_clean":"2024-04-22T00:00:27.359881+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-22T00:00:27.359881+0000","last_undegraded":"2024-04-22T00:00:27.359881+0000","last_fullsized":"2024-04-22T00:00:27.359881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359718+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-22T00:00:27.359718+0000","last_peered":"2024-04-22T00:00:27.359718+0000","last_clean":"2024-04-22T00:00:27.359718+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-22T00:00:27.359718+0000","last_undegraded":"2024-04-22T00:00:27.359718+0000","last_fullsized":"2024-04-22T00:00:27.359718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355292+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-22T00:00:27.355292+0000","last_peered":"2024-04-22T00:00:27.355292+0000","last_clean":"2024-04-22T00:00:27.355292+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-22T00:00:27.355292+0000","last_undegraded":"2024-04-22T00:00:27.355292+0000","last_fullsized":"2024-04-22T00:00:27.355292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356170+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-22T00:00:27.356170+0000","last_peered":"2024-04-22T00:00:27.356170+0000","last_clean":"2024-04-22T00:00:27.356170+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-22T00:00:27.356170+0000","last_undegraded":"2024-04-22T00:00:27.356170+0000","last_fullsized":"2024-04-22T00:00:27.356170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354964+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-22T00:00:27.354964+0000","last_peered":"2024-04-22T00:00:27.354964+0000","last_clean":"2024-04-22T00:00:27.354964+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-22T00:00:27.354964+0000","last_undegraded":"2024-04-22T00:00:27.354964+0000","last_fullsized":"2024-04-22T00:00:27.354964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360956+0000","last_change":"2024-04-22T00:00:27.360956+0000","last_active":"2024-04-22T00:00:27.360956+0000","last_peered":"2024-04-22T00:00:27.360956+0000","last_clean":"2024-04-22T00:00:27.360956+0000","last_became_active":"2024-04-22T00:00:27.360481+0000","last_became_peered":"2024-04-22T00:00:27.360481+0000","last_unstale":"2024-04-22T00:00:27.360956+0000","last_undegraded":"2024-04-22T00:00:27.360956+0000","last_fullsized":"2024-04-22T00:00:27.360956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:16.624366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491742+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-22T00:00:27.491742+0000","last_peered":"2024-04-22T00:00:27.491742+0000","last_clean":"2024-04-22T00:00:27.491742+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-22T00:00:27.491742+0000","last_undegraded":"2024-04-22T00:00:27.491742+0000","last_fullsized":"2024-04-22T00:00:27.491742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355981+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-22T00:00:27.355981+0000","last_peered":"2024-04-22T00:00:27.355981+0000","last_clean":"2024-04-22T00:00:27.355981+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-22T00:00:27.355981+0000","last_undegraded":"2024-04-22T00:00:27.355981+0000","last_fullsized":"2024-04-22T00:00:27.355981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378518+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-22T00:01:16.378518+0000","last_peered":"2024-04-22T00:01:16.378518+0000","last_clean":"2024-04-22T00:01:16.378518+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-22T00:01:16.378518+0000","last_undegraded":"2024-04-22T00:01:16.378518+0000","last_fullsized":"2024-04-22T00:01:16.378518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491288+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-22T00:00:27.491288+0000","last_peered":"2024-04-22T00:00:27.491288+0000","last_clean":"2024-04-22T00:00:27.491288+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-22T00:00:27.491288+0000","last_undegraded":"2024-04-22T00:00:27.491288+0000","last_fullsized":"2024-04-22T00:00:27.491288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355981+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-22T00:00:27.355981+0000","last_peered":"2024-04-22T00:00:27.355981+0000","last_clean":"2024-04-22T00:00:27.355981+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-22T00:00:27.355981+0000","last_undegraded":"2024-04-22T00:00:27.355981+0000","last_fullsized":"2024-04-22T00:00:27.355981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361537+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-22T00:00:27.361537+0000","last_peered":"2024-04-22T00:00:27.361537+0000","last_clean":"2024-04-22T00:00:27.361537+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-22T00:00:27.361537+0000","last_undegraded":"2024-04-22T00:00:27.361537+0000","last_fullsized":"2024-04-22T00:00:27.361537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:15.376607+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-22T00:01:15.376607+0000","last_peered":"2024-04-22T00:01:15.376607+0000","last_clean":"2024-04-22T00:01:15.376607+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-22T00:01:15.376607+0000","last_undegraded":"2024-04-22T00:01:15.376607+0000","last_fullsized":"2024-04-22T00:01:15.376607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.369850+0000","last_change":"2024-04-22T00:00:27.369850+0000","last_active":"2024-04-22T00:00:27.369850+0000","last_peered":"2024-04-22T00:00:27.369850+0000","last_clean":"2024-04-22T00:00:27.369850+0000","last_became_active":"2024-04-22T00:00:27.361664+0000","last_became_peered":"2024-04-22T00:00:27.361664+0000","last_unstale":"2024-04-22T00:00:27.369850+0000","last_undegraded":"2024-04-22T00:00:27.369850+0000","last_fullsized":"2024-04-22T00:00:27.369850+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:42:13.794809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490505+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-22T00:00:27.490505+0000","last_peered":"2024-04-22T00:00:27.490505+0000","last_clean":"2024-04-22T00:00:27.490505+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-22T00:00:27.490505+0000","last_undegraded":"2024-04-22T00:00:27.490505+0000","last_fullsized":"2024-04-22T00:00:27.490505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357610+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-22T00:00:27.357610+0000","last_peered":"2024-04-22T00:00:27.357610+0000","last_clean":"2024-04-22T00:00:27.357610+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-22T00:00:27.357610+0000","last_undegraded":"2024-04-22T00:00:27.357610+0000","last_fullsized":"2024-04-22T00:00:27.357610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355819+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-22T00:00:27.355819+0000","last_peered":"2024-04-22T00:00:27.355819+0000","last_clean":"2024-04-22T00:00:27.355819+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-22T00:00:27.355819+0000","last_undegraded":"2024-04-22T00:00:27.355819+0000","last_fullsized":"2024-04-22T00:00:27.355819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359733+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-22T00:00:27.359733+0000","last_peered":"2024-04-22T00:00:27.359733+0000","last_clean":"2024-04-22T00:00:27.359733+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-22T00:00:27.359733+0000","last_undegraded":"2024-04-22T00:00:27.359733+0000","last_fullsized":"2024-04-22T00:00:27.359733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359209+0000","last_change":"2024-04-22T00:00:27.359209+0000","last_active":"2024-04-22T00:00:27.359209+0000","last_peered":"2024-04-22T00:00:27.359209+0000","last_clean":"2024-04-22T00:00:27.359209+0000","last_became_active":"2024-04-22T00:00:27.358750+0000","last_became_peered":"2024-04-22T00:00:27.358750+0000","last_unstale":"2024-04-22T00:00:27.359209+0000","last_undegraded":"2024-04-22T00:00:27.359209+0000","last_fullsized":"2024-04-22T00:00:27.359209+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:05.053039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355880+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:00:27.355880+0000","last_peered":"2024-04-22T00:00:27.355880+0000","last_clean":"2024-04-22T00:00:27.355880+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:00:27.355880+0000","last_undegraded":"2024-04-22T00:00:27.355880+0000","last_fullsized":"2024-04-22T00:00:27.355880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354859+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-22T00:00:27.354859+0000","last_peered":"2024-04-22T00:00:27.354859+0000","last_clean":"2024-04-22T00:00:27.354859+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-22T00:00:27.354859+0000","last_undegraded":"2024-04-22T00:00:27.354859+0000","last_fullsized":"2024-04-22T00:00:27.354859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.369109+0000","last_change":"2024-04-22T00:00:27.368767+0000","last_active":"2024-04-22T00:00:27.369109+0000","last_peered":"2024-04-22T00:00:27.369109+0000","last_clean":"2024-04-22T00:00:27.369109+0000","last_became_active":"2024-04-22T00:00:27.362160+0000","last_became_peered":"2024-04-22T00:00:27.362160+0000","last_unstale":"2024-04-22T00:00:27.369109+0000","last_undegraded":"2024-04-22T00:00:27.369109+0000","last_fullsized":"2024-04-22T00:00:27.369109+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:38.606596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356383+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-22T00:00:27.356383+0000","last_peered":"2024-04-22T00:00:27.356383+0000","last_clean":"2024-04-22T00:00:27.356383+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-22T00:00:27.356383+0000","last_undegraded":"2024-04-22T00:00:27.356383+0000","last_fullsized":"2024-04-22T00:00:27.356383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490300+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-22T00:00:27.490300+0000","last_peered":"2024-04-22T00:00:27.490300+0000","last_clean":"2024-04-22T00:00:27.490300+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-22T00:00:27.490300+0000","last_undegraded":"2024-04-22T00:00:27.490300+0000","last_fullsized":"2024-04-22T00:00:27.490300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353337+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:00:27.353337+0000","last_peered":"2024-04-22T00:00:27.353337+0000","last_clean":"2024-04-22T00:00:27.353337+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:00:27.353337+0000","last_undegraded":"2024-04-22T00:00:27.353337+0000","last_fullsized":"2024-04-22T00:00:27.353337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490676+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-22T00:00:27.490676+0000","last_peered":"2024-04-22T00:00:27.490676+0000","last_clean":"2024-04-22T00:00:27.490676+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-22T00:00:27.490676+0000","last_undegraded":"2024-04-22T00:00:27.490676+0000","last_fullsized":"2024-04-22T00:00:27.490676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352065+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-22T00:00:27.352065+0000","last_peered":"2024-04-22T00:00:27.352065+0000","last_clean":"2024-04-22T00:00:27.352065+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-22T00:00:27.352065+0000","last_undegraded":"2024-04-22T00:00:27.352065+0000","last_fullsized":"2024-04-22T00:00:27.352065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359020+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-22T00:00:27.359020+0000","last_peered":"2024-04-22T00:00:27.359020+0000","last_clean":"2024-04-22T00:00:27.359020+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-22T00:00:27.359020+0000","last_undegraded":"2024-04-22T00:00:27.359020+0000","last_fullsized":"2024-04-22T00:00:27.359020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357957+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-22T00:00:27.357957+0000","last_peered":"2024-04-22T00:00:27.357957+0000","last_clean":"2024-04-22T00:00:27.357957+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-22T00:00:27.357957+0000","last_undegraded":"2024-04-22T00:00:27.357957+0000","last_fullsized":"2024-04-22T00:00:27.357957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358936+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-22T00:00:27.358936+0000","last_peered":"2024-04-22T00:00:27.358936+0000","last_clean":"2024-04-22T00:00:27.358936+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-22T00:00:27.358936+0000","last_undegraded":"2024-04-22T00:00:27.358936+0000","last_fullsized":"2024-04-22T00:00:27.358936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354792+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:00:27.354792+0000","last_peered":"2024-04-22T00:00:27.354792+0000","last_clean":"2024-04-22T00:00:27.354792+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:00:27.354792+0000","last_undegraded":"2024-04-22T00:00:27.354792+0000","last_fullsized":"2024-04-22T00:00:27.354792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354661+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-22T00:00:27.354661+0000","last_peered":"2024-04-22T00:00:27.354661+0000","last_clean":"2024-04-22T00:00:27.354661+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-22T00:00:27.354661+0000","last_undegraded":"2024-04-22T00:00:27.354661+0000","last_fullsized":"2024-04-22T00:00:27.354661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492722+0000","last_change":"2024-04-22T00:00:27.492100+0000","last_active":"2024-04-22T00:00:27.492722+0000","last_peered":"2024-04-22T00:00:27.492722+0000","last_clean":"2024-04-22T00:00:27.492722+0000","last_became_active":"2024-04-22T00:00:27.491414+0000","last_became_peered":"2024-04-22T00:00:27.491414+0000","last_unstale":"2024-04-22T00:00:27.492722+0000","last_undegraded":"2024-04-22T00:00:27.492722+0000","last_fullsized":"2024-04-22T00:00:27.492722+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:16:32.280602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490378+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:00:27.490378+0000","last_peered":"2024-04-22T00:00:27.490378+0000","last_clean":"2024-04-22T00:00:27.490378+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:00:27.490378+0000","last_undegraded":"2024-04-22T00:00:27.490378+0000","last_fullsized":"2024-04-22T00:00:27.490378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354802+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-22T00:00:27.354802+0000","last_peered":"2024-04-22T00:00:27.354802+0000","last_clean":"2024-04-22T00:00:27.354802+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-22T00:00:27.354802+0000","last_undegraded":"2024-04-22T00:00:27.354802+0000","last_fullsized":"2024-04-22T00:00:27.354802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.497233+0000","last_change":"2024-04-22T00:00:27.497233+0000","last_active":"2024-04-22T00:00:27.497233+0000","last_peered":"2024-04-22T00:00:27.497233+0000","last_clean":"2024-04-22T00:00:27.497233+0000","last_became_active":"2024-04-22T00:00:27.492847+0000","last_became_peered":"2024-04-22T00:00:27.492847+0000","last_unstale":"2024-04-22T00:00:27.497233+0000","last_undegraded":"2024-04-22T00:00:27.497233+0000","last_fullsized":"2024-04-22T00:00:27.497233+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:25.742746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352162+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:00:27.352162+0000","last_peered":"2024-04-22T00:00:27.352162+0000","last_clean":"2024-04-22T00:00:27.352162+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:00:27.352162+0000","last_undegraded":"2024-04-22T00:00:27.352162+0000","last_fullsized":"2024-04-22T00:00:27.352162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490781+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-22T00:00:27.490781+0000","last_peered":"2024-04-22T00:00:27.490781+0000","last_clean":"2024-04-22T00:00:27.490781+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-22T00:00:27.490781+0000","last_undegraded":"2024-04-22T00:00:27.490781+0000","last_fullsized":"2024-04-22T00:00:27.490781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358857+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-22T00:00:27.358857+0000","last_peered":"2024-04-22T00:00:27.358857+0000","last_clean":"2024-04-22T00:00:27.358857+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-22T00:00:27.358857+0000","last_undegraded":"2024-04-22T00:00:27.358857+0000","last_fullsized":"2024-04-22T00:00:27.358857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354505+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:00:27.354505+0000","last_peered":"2024-04-22T00:00:27.354505+0000","last_clean":"2024-04-22T00:00:27.354505+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:00:27.354505+0000","last_undegraded":"2024-04-22T00:00:27.354505+0000","last_fullsized":"2024-04-22T00:00:27.354505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357045+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-22T00:00:27.357045+0000","last_peered":"2024-04-22T00:00:27.357045+0000","last_clean":"2024-04-22T00:00:27.357045+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-22T00:00:27.357045+0000","last_undegraded":"2024-04-22T00:00:27.357045+0000","last_fullsized":"2024-04-22T00:00:27.357045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354396+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-22T00:00:27.354396+0000","last_peered":"2024-04-22T00:00:27.354396+0000","last_clean":"2024-04-22T00:00:27.354396+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-22T00:00:27.354396+0000","last_undegraded":"2024-04-22T00:00:27.354396+0000","last_fullsized":"2024-04-22T00:00:27.354396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:12.401417+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:01:12.401417+0000","last_peered":"2024-04-22T00:01:12.401417+0000","last_clean":"2024-04-22T00:01:12.401417+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:01:12.401417+0000","last_undegraded":"2024-04-22T00:01:12.401417+0000","last_fullsized":"2024-04-22T00:01:12.401417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358151+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-22T00:00:27.358151+0000","last_peered":"2024-04-22T00:00:27.358151+0000","last_clean":"2024-04-22T00:00:27.358151+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-22T00:00:27.358151+0000","last_undegraded":"2024-04-22T00:00:27.358151+0000","last_fullsized":"2024-04-22T00:00:27.358151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358746+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-22T00:00:27.358746+0000","last_peered":"2024-04-22T00:00:27.358746+0000","last_clean":"2024-04-22T00:00:27.358746+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-22T00:00:27.358746+0000","last_undegraded":"2024-04-22T00:00:27.358746+0000","last_fullsized":"2024-04-22T00:00:27.358746+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360489+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:00:27.360489+0000","last_peered":"2024-04-22T00:00:27.360489+0000","last_clean":"2024-04-22T00:00:27.360489+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:00:27.360489+0000","last_undegraded":"2024-04-22T00:00:27.360489+0000","last_fullsized":"2024-04-22T00:00:27.360489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356513+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-22T00:00:27.356513+0000","last_peered":"2024-04-22T00:00:27.356513+0000","last_clean":"2024-04-22T00:00:27.356513+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-22T00:00:27.356513+0000","last_undegraded":"2024-04-22T00:00:27.356513+0000","last_fullsized":"2024-04-22T00:00:27.356513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360458+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-22T00:00:27.360458+0000","last_peered":"2024-04-22T00:00:27.360458+0000","last_clean":"2024-04-22T00:00:27.360458+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-22T00:00:27.360458+0000","last_undegraded":"2024-04-22T00:00:27.360458+0000","last_fullsized":"2024-04-22T00:00:27.360458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356119+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-22T00:00:27.356119+0000","last_peered":"2024-04-22T00:00:27.356119+0000","last_clean":"2024-04-22T00:00:27.356119+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-22T00:00:27.356119+0000","last_undegraded":"2024-04-22T00:00:27.356119+0000","last_fullsized":"2024-04-22T00:00:27.356119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356817+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-22T00:00:27.356817+0000","last_peered":"2024-04-22T00:00:27.356817+0000","last_clean":"2024-04-22T00:00:27.356817+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-22T00:00:27.356817+0000","last_undegraded":"2024-04-22T00:00:27.356817+0000","last_fullsized":"2024-04-22T00:00:27.356817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354922+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-22T00:00:27.354922+0000","last_peered":"2024-04-22T00:00:27.354922+0000","last_clean":"2024-04-22T00:00:27.354922+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-22T00:00:27.354922+0000","last_undegraded":"2024-04-22T00:00:27.354922+0000","last_fullsized":"2024-04-22T00:00:27.354922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352219+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-22T00:00:27.352219+0000","last_peered":"2024-04-22T00:00:27.352219+0000","last_clean":"2024-04-22T00:00:27.352219+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-22T00:00:27.352219+0000","last_undegraded":"2024-04-22T00:00:27.352219+0000","last_fullsized":"2024-04-22T00:00:27.352219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352156+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-22T00:00:27.352156+0000","last_peered":"2024-04-22T00:00:27.352156+0000","last_clean":"2024-04-22T00:00:27.352156+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-22T00:00:27.352156+0000","last_undegraded":"2024-04-22T00:00:27.352156+0000","last_fullsized":"2024-04-22T00:00:27.352156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.362810+0000","last_change":"2024-04-22T00:00:27.362810+0000","last_active":"2024-04-22T00:00:27.362810+0000","last_peered":"2024-04-22T00:00:27.362810+0000","last_clean":"2024-04-22T00:00:27.362810+0000","last_became_active":"2024-04-22T00:00:27.362380+0000","last_became_peered":"2024-04-22T00:00:27.362380+0000","last_unstale":"2024-04-22T00:00:27.362810+0000","last_undegraded":"2024-04-22T00:00:27.362810+0000","last_fullsized":"2024-04-22T00:00:27.362810+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:09:30.391961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356294+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-22T00:00:27.356294+0000","last_peered":"2024-04-22T00:00:27.356294+0000","last_clean":"2024-04-22T00:00:27.356294+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-22T00:00:27.356294+0000","last_undegraded":"2024-04-22T00:00:27.356294+0000","last_fullsized":"2024-04-22T00:00:27.356294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356715+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-22T00:00:27.356715+0000","last_peered":"2024-04-22T00:00:27.356715+0000","last_clean":"2024-04-22T00:00:27.356715+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-22T00:00:27.356715+0000","last_undegraded":"2024-04-22T00:00:27.356715+0000","last_fullsized":"2024-04-22T00:00:27.356715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352827+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-22T00:00:27.352827+0000","last_peered":"2024-04-22T00:00:27.352827+0000","last_clean":"2024-04-22T00:00:27.352827+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-22T00:00:27.352827+0000","last_undegraded":"2024-04-22T00:00:27.352827+0000","last_fullsized":"2024-04-22T00:00:27.352827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490869+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-22T00:00:27.490869+0000","last_peered":"2024-04-22T00:00:27.490869+0000","last_clean":"2024-04-22T00:00:27.490869+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-22T00:00:27.490869+0000","last_undegraded":"2024-04-22T00:00:27.490869+0000","last_fullsized":"2024-04-22T00:00:27.490869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361417+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-22T00:00:27.361417+0000","last_peered":"2024-04-22T00:00:27.361417+0000","last_clean":"2024-04-22T00:00:27.361417+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-22T00:00:27.361417+0000","last_undegraded":"2024-04-22T00:00:27.361417+0000","last_fullsized":"2024-04-22T00:00:27.361417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490327+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-22T00:00:27.490327+0000","last_peered":"2024-04-22T00:00:27.490327+0000","last_clean":"2024-04-22T00:00:27.490327+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-22T00:00:27.490327+0000","last_undegraded":"2024-04-22T00:00:27.490327+0000","last_fullsized":"2024-04-22T00:00:27.490327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354899+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-22T00:00:27.354899+0000","last_peered":"2024-04-22T00:00:27.354899+0000","last_clean":"2024-04-22T00:00:27.354899+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-22T00:00:27.354899+0000","last_undegraded":"2024-04-22T00:00:27.354899+0000","last_fullsized":"2024-04-22T00:00:27.354899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355985+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-22T00:00:27.355985+0000","last_peered":"2024-04-22T00:00:27.355985+0000","last_clean":"2024-04-22T00:00:27.355985+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-22T00:00:27.355985+0000","last_undegraded":"2024-04-22T00:00:27.355985+0000","last_fullsized":"2024-04-22T00:00:27.355985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355867+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-22T00:00:27.355867+0000","last_peered":"2024-04-22T00:00:27.355867+0000","last_clean":"2024-04-22T00:00:27.355867+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-22T00:00:27.355867+0000","last_undegraded":"2024-04-22T00:00:27.355867+0000","last_fullsized":"2024-04-22T00:00:27.355867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69537,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":750,"num_read_kb":453,"num_write":243,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":501936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1155069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":501363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":505725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1098238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":560230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":498714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":560841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":501332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1654784,"data_stored":1089333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:01:18.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T00:01:19.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:19 smithi038 ceph-mon[27972]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:19.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:19 smithi038 ceph-mon[27972]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:19.411 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:19.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:19 smithi203 ceph-mon[31431]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:19.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:19 smithi203 ceph-mon[31431]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:19.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:19 smithi111 ceph-mon[31543]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:19.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:19 smithi111 ceph-mon[31543]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:21.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- 172.21.15.38:0/1586565080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 msgr2=0x7ff9e8106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:21.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1586565080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e8106e90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009990 tx=0x7ff9d8009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- 172.21.15.38:0/1586565080 shutdown_connections 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1586565080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e8107570 0x7ff9e810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1586565080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e81024e0 0x7ff9e81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1586565080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e8106e90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- 172.21.15.38:0/1586565080 >> 172.21.15.38:0/1586565080 conn(0x7ff9e8079390 msgr2=0x7ff9e80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- 172.21.15.38:0/1586565080 shutdown_connections 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- 172.21.15.38:0/1586565080 wait complete. 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 Processor -- start 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.017+0000 7ff9f0088700 1 -- start start 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 0x7ff9e819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:21.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 0x7ff9e8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e8112530 con 0x7ff9e8102e90 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81123b0 con 0x7ff9e8107570 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9f0088700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e81126b0 con 0x7ff9e81024e0 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39190/0 (socket says 172.21.15.38:39190) 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 -- 172.21.15.38:0/1982464006 learned_addr learned my addr 172.21.15.38:0/1982464006 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ee625700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 0x7ff9e8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ede24700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 0x7ff9e819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 -- 172.21.15.38:0/1982464006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 msgr2=0x7ff9e819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 0x7ff9e819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 -- 172.21.15.38:0/1982464006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 msgr2=0x7ff9e8198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 0x7ff9e8198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 -- 172.21.15.38:0/1982464006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e8198ff0 con 0x7ff9e8102e90 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ee625700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 0x7ff9e8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ede24700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 0x7ff9e819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.018+0000 7ff9ed623700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8000c00 tx=0x7ff9d8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.019+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8027070 con 0x7ff9e8102e90 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.019+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e8199280 con 0x7ff9e8102e90 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.019+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d8004110 con 0x7ff9e8102e90 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.019+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8022d70 con 0x7ff9e8102e90 2024-04-22T00:01:21.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.019+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e81a54f0 con 0x7ff9e8102e90 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.020+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e8103970 con 0x7ff9e8102e90 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.021+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff9d802a070 con 0x7ff9e8102e90 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.021+0000 7ff9deffd700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 0x7ff9d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.021+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff9d80d0360 con 0x7ff9e8102e90 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.021+0000 7ff9ede24700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 0x7ff9d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:21.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.022+0000 7ff9ede24700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 0x7ff9d40a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005fd0 tx=0x7ff9e4005e20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:21.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.026+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff9d805e6f0 con 0x7ff9e8102e90 2024-04-22T00:01:21.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.228+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9e8062670 con 0x7ff9d40a2cb0 2024-04-22T00:01:21.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.238+0000 7ff9deffd700 1 -- 172.21.15.38:0/1982464006 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340411 (secure 0 0 0) 0x7ff9e8062670 con 0x7ff9d40a2cb0 2024-04-22T00:01:21.235 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 msgr2=0x7ff9d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 0x7ff9d40a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005fd0 tx=0x7ff9e4005e20 comp rx=0 tx=0).stop 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 msgr2=0x7ff9e819ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8000c00 tx=0x7ff9d8009f90 comp rx=0 tx=0).stop 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 shutdown_connections 2024-04-22T00:01:21.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff9d40a2cb0 0x7ff9d40a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e8107570 0x7ff9e8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9e81024e0 0x7ff9e819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 --2- 172.21.15.38:0/1982464006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff9e8102e90 0x7ff9e819ebd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 >> 172.21.15.38:0/1982464006 conn(0x7ff9e8079390 msgr2=0x7ff9e8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 shutdown_connections 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:21.241+0000 7ff9f0088700 1 -- 172.21.15.38:0/1982464006 wait complete. 2024-04-22T00:01:21.239 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:01:21.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:21 smithi038 ceph-mon[27972]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:21.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:21 smithi203 ceph-mon[31431]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:21.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:21 smithi111 ceph-mon[31543]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:22.016 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-22T00:01:19.675674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660139,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":891,"num_read_kb":572,"num_write":426,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181082,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363452,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578372,"statfs":{"total":1151940427776,"available":1151568252928,"internally_reserved":0,"allocated":14581760,"data_stored":7966223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003607"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356786+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-22T00:00:27.356786+0000","last_peered":"2024-04-22T00:00:27.356786+0000","last_clean":"2024-04-22T00:00:27.356786+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-22T00:00:27.356786+0000","last_undegraded":"2024-04-22T00:00:27.356786+0000","last_fullsized":"2024-04-22T00:00:27.356786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357853+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-22T00:00:27.357853+0000","last_peered":"2024-04-22T00:00:27.357853+0000","last_clean":"2024-04-22T00:00:27.357853+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-22T00:00:27.357853+0000","last_undegraded":"2024-04-22T00:00:27.357853+0000","last_fullsized":"2024-04-22T00:00:27.357853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357305+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-22T00:00:27.357305+0000","last_peered":"2024-04-22T00:00:27.357305+0000","last_clean":"2024-04-22T00:00:27.357305+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-22T00:00:27.357305+0000","last_undegraded":"2024-04-22T00:00:27.357305+0000","last_fullsized":"2024-04-22T00:00:27.357305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359224+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-22T00:00:27.359224+0000","last_peered":"2024-04-22T00:00:27.359224+0000","last_clean":"2024-04-22T00:00:27.359224+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-22T00:00:27.359224+0000","last_undegraded":"2024-04-22T00:00:27.359224+0000","last_fullsized":"2024-04-22T00:00:27.359224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354642+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-22T00:00:27.354642+0000","last_peered":"2024-04-22T00:00:27.354642+0000","last_clean":"2024-04-22T00:00:27.354642+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-22T00:00:27.354642+0000","last_undegraded":"2024-04-22T00:00:27.354642+0000","last_fullsized":"2024-04-22T00:00:27.354642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491579+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-22T00:00:27.491579+0000","last_peered":"2024-04-22T00:00:27.491579+0000","last_clean":"2024-04-22T00:00:27.491579+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-22T00:00:27.491579+0000","last_undegraded":"2024-04-22T00:00:27.491579+0000","last_fullsized":"2024-04-22T00:00:27.491579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356054+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-22T00:00:27.356054+0000","last_peered":"2024-04-22T00:00:27.356054+0000","last_clean":"2024-04-22T00:00:27.356054+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-22T00:00:27.356054+0000","last_undegraded":"2024-04-22T00:00:27.356054+0000","last_fullsized":"2024-04-22T00:00:27.356054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490436+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-22T00:00:27.490436+0000","last_peered":"2024-04-22T00:00:27.490436+0000","last_clean":"2024-04-22T00:00:27.490436+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-22T00:00:27.490436+0000","last_undegraded":"2024-04-22T00:00:27.490436+0000","last_fullsized":"2024-04-22T00:00:27.490436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355228+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-22T00:00:27.355228+0000","last_peered":"2024-04-22T00:00:27.355228+0000","last_clean":"2024-04-22T00:00:27.355228+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-22T00:00:27.355228+0000","last_undegraded":"2024-04-22T00:00:27.355228+0000","last_fullsized":"2024-04-22T00:00:27.355228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491800+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-22T00:00:27.491800+0000","last_peered":"2024-04-22T00:00:27.491800+0000","last_clean":"2024-04-22T00:00:27.491800+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-22T00:00:27.491800+0000","last_undegraded":"2024-04-22T00:00:27.491800+0000","last_fullsized":"2024-04-22T00:00:27.491800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357900+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-22T00:00:27.357900+0000","last_peered":"2024-04-22T00:00:27.357900+0000","last_clean":"2024-04-22T00:00:27.357900+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-22T00:00:27.357900+0000","last_undegraded":"2024-04-22T00:00:27.357900+0000","last_fullsized":"2024-04-22T00:00:27.357900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359175+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-22T00:00:27.359175+0000","last_peered":"2024-04-22T00:00:27.359175+0000","last_clean":"2024-04-22T00:00:27.359175+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-22T00:00:27.359175+0000","last_undegraded":"2024-04-22T00:00:27.359175+0000","last_fullsized":"2024-04-22T00:00:27.359175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492882+0000","last_change":"2024-04-22T00:00:27.492882+0000","last_active":"2024-04-22T00:00:27.492882+0000","last_peered":"2024-04-22T00:00:27.492882+0000","last_clean":"2024-04-22T00:00:27.492882+0000","last_became_active":"2024-04-22T00:00:27.492451+0000","last_became_peered":"2024-04-22T00:00:27.492451+0000","last_unstale":"2024-04-22T00:00:27.492882+0000","last_undegraded":"2024-04-22T00:00:27.492882+0000","last_fullsized":"2024-04-22T00:00:27.492882+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:10.283575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357546+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-22T00:00:27.357546+0000","last_peered":"2024-04-22T00:00:27.357546+0000","last_clean":"2024-04-22T00:00:27.357546+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-22T00:00:27.357546+0000","last_undegraded":"2024-04-22T00:00:27.357546+0000","last_fullsized":"2024-04-22T00:00:27.357546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357367+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-22T00:00:27.357367+0000","last_peered":"2024-04-22T00:00:27.357367+0000","last_clean":"2024-04-22T00:00:27.357367+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-22T00:00:27.357367+0000","last_undegraded":"2024-04-22T00:00:27.357367+0000","last_fullsized":"2024-04-22T00:00:27.357367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354448+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-22T00:00:27.354448+0000","last_peered":"2024-04-22T00:00:27.354448+0000","last_clean":"2024-04-22T00:00:27.354448+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-22T00:00:27.354448+0000","last_undegraded":"2024-04-22T00:00:27.354448+0000","last_fullsized":"2024-04-22T00:00:27.354448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356396+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-22T00:00:27.356396+0000","last_peered":"2024-04-22T00:00:27.356396+0000","last_clean":"2024-04-22T00:00:27.356396+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-22T00:00:27.356396+0000","last_undegraded":"2024-04-22T00:00:27.356396+0000","last_fullsized":"2024-04-22T00:00:27.356396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353369+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-22T00:00:27.353369+0000","last_peered":"2024-04-22T00:00:27.353369+0000","last_clean":"2024-04-22T00:00:27.353369+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-22T00:00:27.353369+0000","last_undegraded":"2024-04-22T00:00:27.353369+0000","last_fullsized":"2024-04-22T00:00:27.353369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352831+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-22T00:00:27.352831+0000","last_peered":"2024-04-22T00:00:27.352831+0000","last_clean":"2024-04-22T00:00:27.352831+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-22T00:00:27.352831+0000","last_undegraded":"2024-04-22T00:00:27.352831+0000","last_fullsized":"2024-04-22T00:00:27.352831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352249+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-22T00:00:27.352249+0000","last_peered":"2024-04-22T00:00:27.352249+0000","last_clean":"2024-04-22T00:00:27.352249+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-22T00:00:27.352249+0000","last_undegraded":"2024-04-22T00:00:27.352249+0000","last_fullsized":"2024-04-22T00:00:27.352249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:14.375260+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:01:14.375260+0000","last_peered":"2024-04-22T00:01:14.375260+0000","last_clean":"2024-04-22T00:01:14.375260+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:01:14.375260+0000","last_undegraded":"2024-04-22T00:01:14.375260+0000","last_fullsized":"2024-04-22T00:01:14.375260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491009+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-22T00:00:27.491009+0000","last_peered":"2024-04-22T00:00:27.491009+0000","last_clean":"2024-04-22T00:00:27.491009+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-22T00:00:27.491009+0000","last_undegraded":"2024-04-22T00:00:27.491009+0000","last_fullsized":"2024-04-22T00:00:27.491009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356909+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-22T00:00:27.356909+0000","last_peered":"2024-04-22T00:00:27.356909+0000","last_clean":"2024-04-22T00:00:27.356909+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-22T00:00:27.356909+0000","last_undegraded":"2024-04-22T00:00:27.356909+0000","last_fullsized":"2024-04-22T00:00:27.356909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491065+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-22T00:00:27.491065+0000","last_peered":"2024-04-22T00:00:27.491065+0000","last_clean":"2024-04-22T00:00:27.491065+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-22T00:00:27.491065+0000","last_undegraded":"2024-04-22T00:00:27.491065+0000","last_fullsized":"2024-04-22T00:00:27.491065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358173+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-22T00:00:27.358173+0000","last_peered":"2024-04-22T00:00:27.358173+0000","last_clean":"2024-04-22T00:00:27.358173+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-22T00:00:27.358173+0000","last_undegraded":"2024-04-22T00:00:27.358173+0000","last_fullsized":"2024-04-22T00:00:27.358173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492457+0000","last_change":"2024-04-22T00:00:27.492457+0000","last_active":"2024-04-22T00:00:27.492457+0000","last_peered":"2024-04-22T00:00:27.492457+0000","last_clean":"2024-04-22T00:00:27.492457+0000","last_became_active":"2024-04-22T00:00:27.491828+0000","last_became_peered":"2024-04-22T00:00:27.491828+0000","last_unstale":"2024-04-22T00:00:27.492457+0000","last_undegraded":"2024-04-22T00:00:27.492457+0000","last_fullsized":"2024-04-22T00:00:27.492457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:09:18.512103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354779+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:00:27.354779+0000","last_peered":"2024-04-22T00:00:27.354779+0000","last_clean":"2024-04-22T00:00:27.354779+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:00:27.354779+0000","last_undegraded":"2024-04-22T00:00:27.354779+0000","last_fullsized":"2024-04-22T00:00:27.354779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354584+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-22T00:00:27.354584+0000","last_peered":"2024-04-22T00:00:27.354584+0000","last_clean":"2024-04-22T00:00:27.354584+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-22T00:00:27.354584+0000","last_undegraded":"2024-04-22T00:00:27.354584+0000","last_fullsized":"2024-04-22T00:00:27.354584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357308+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-22T00:00:27.357308+0000","last_peered":"2024-04-22T00:00:27.357308+0000","last_clean":"2024-04-22T00:00:27.357308+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-22T00:00:27.357308+0000","last_undegraded":"2024-04-22T00:00:27.357308+0000","last_fullsized":"2024-04-22T00:00:27.357308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355422+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-22T00:00:27.355422+0000","last_peered":"2024-04-22T00:00:27.355422+0000","last_clean":"2024-04-22T00:00:27.355422+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-22T00:00:27.355422+0000","last_undegraded":"2024-04-22T00:00:27.355422+0000","last_fullsized":"2024-04-22T00:00:27.355422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360370+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-22T00:00:27.360370+0000","last_peered":"2024-04-22T00:00:27.360370+0000","last_clean":"2024-04-22T00:00:27.360370+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-22T00:00:27.360370+0000","last_undegraded":"2024-04-22T00:00:27.360370+0000","last_fullsized":"2024-04-22T00:00:27.360370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352694+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-22T00:00:27.352694+0000","last_peered":"2024-04-22T00:00:27.352694+0000","last_clean":"2024-04-22T00:00:27.352694+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-22T00:00:27.352694+0000","last_undegraded":"2024-04-22T00:00:27.352694+0000","last_fullsized":"2024-04-22T00:00:27.352694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:13.375360+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:01:13.375360+0000","last_peered":"2024-04-22T00:01:13.375360+0000","last_clean":"2024-04-22T00:01:13.375360+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:01:13.375360+0000","last_undegraded":"2024-04-22T00:01:13.375360+0000","last_fullsized":"2024-04-22T00:01:13.375360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355075+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-22T00:00:27.355075+0000","last_peered":"2024-04-22T00:00:27.355075+0000","last_clean":"2024-04-22T00:00:27.355075+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-22T00:00:27.355075+0000","last_undegraded":"2024-04-22T00:00:27.355075+0000","last_fullsized":"2024-04-22T00:00:27.355075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352432+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-22T00:00:27.352432+0000","last_peered":"2024-04-22T00:00:27.352432+0000","last_clean":"2024-04-22T00:00:27.352432+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-22T00:00:27.352432+0000","last_undegraded":"2024-04-22T00:00:27.352432+0000","last_fullsized":"2024-04-22T00:00:27.352432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491319+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-22T00:00:27.491319+0000","last_peered":"2024-04-22T00:00:27.491319+0000","last_clean":"2024-04-22T00:00:27.491319+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-22T00:00:27.491319+0000","last_undegraded":"2024-04-22T00:00:27.491319+0000","last_fullsized":"2024-04-22T00:00:27.491319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360197+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-22T00:00:27.360197+0000","last_peered":"2024-04-22T00:00:27.360197+0000","last_clean":"2024-04-22T00:00:27.360197+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-22T00:00:27.360197+0000","last_undegraded":"2024-04-22T00:00:27.360197+0000","last_fullsized":"2024-04-22T00:00:27.360197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356273+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-22T00:00:27.356273+0000","last_peered":"2024-04-22T00:00:27.356273+0000","last_clean":"2024-04-22T00:00:27.356273+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-22T00:00:27.356273+0000","last_undegraded":"2024-04-22T00:00:27.356273+0000","last_fullsized":"2024-04-22T00:00:27.356273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355336+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-22T00:00:27.355336+0000","last_peered":"2024-04-22T00:00:27.355336+0000","last_clean":"2024-04-22T00:00:27.355336+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-22T00:00:27.355336+0000","last_undegraded":"2024-04-22T00:00:27.355336+0000","last_fullsized":"2024-04-22T00:00:27.355336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490300+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-22T00:00:27.490300+0000","last_peered":"2024-04-22T00:00:27.490300+0000","last_clean":"2024-04-22T00:00:27.490300+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-22T00:00:27.490300+0000","last_undegraded":"2024-04-22T00:00:27.490300+0000","last_fullsized":"2024-04-22T00:00:27.490300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357732+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-22T00:00:27.357732+0000","last_peered":"2024-04-22T00:00:27.357732+0000","last_clean":"2024-04-22T00:00:27.357732+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-22T00:00:27.357732+0000","last_undegraded":"2024-04-22T00:00:27.357732+0000","last_fullsized":"2024-04-22T00:00:27.357732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360631+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:00:27.360631+0000","last_peered":"2024-04-22T00:00:27.360631+0000","last_clean":"2024-04-22T00:00:27.360631+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:00:27.360631+0000","last_undegraded":"2024-04-22T00:00:27.360631+0000","last_fullsized":"2024-04-22T00:00:27.360631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352579+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-22T00:00:27.352579+0000","last_peered":"2024-04-22T00:00:27.352579+0000","last_clean":"2024-04-22T00:00:27.352579+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-22T00:00:27.352579+0000","last_undegraded":"2024-04-22T00:00:27.352579+0000","last_fullsized":"2024-04-22T00:00:27.352579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490506+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-22T00:00:27.490506+0000","last_peered":"2024-04-22T00:00:27.490506+0000","last_clean":"2024-04-22T00:00:27.490506+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-22T00:00:27.490506+0000","last_undegraded":"2024-04-22T00:00:27.490506+0000","last_fullsized":"2024-04-22T00:00:27.490506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378673+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:01:16.378673+0000","last_peered":"2024-04-22T00:01:16.378673+0000","last_clean":"2024-04-22T00:01:16.378673+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:01:16.378673+0000","last_undegraded":"2024-04-22T00:01:16.378673+0000","last_fullsized":"2024-04-22T00:01:16.378673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355061+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-22T00:00:27.355061+0000","last_peered":"2024-04-22T00:00:27.355061+0000","last_clean":"2024-04-22T00:00:27.355061+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-22T00:00:27.355061+0000","last_undegraded":"2024-04-22T00:00:27.355061+0000","last_fullsized":"2024-04-22T00:00:27.355061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352388+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-22T00:00:27.352388+0000","last_peered":"2024-04-22T00:00:27.352388+0000","last_clean":"2024-04-22T00:00:27.352388+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-22T00:00:27.352388+0000","last_undegraded":"2024-04-22T00:00:27.352388+0000","last_fullsized":"2024-04-22T00:00:27.352388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356489+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:00:27.356489+0000","last_peered":"2024-04-22T00:00:27.356489+0000","last_clean":"2024-04-22T00:00:27.356489+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:00:27.356489+0000","last_undegraded":"2024-04-22T00:00:27.356489+0000","last_fullsized":"2024-04-22T00:00:27.356489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358076+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-22T00:00:27.358076+0000","last_peered":"2024-04-22T00:00:27.358076+0000","last_clean":"2024-04-22T00:00:27.358076+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-22T00:00:27.358076+0000","last_undegraded":"2024-04-22T00:00:27.358076+0000","last_fullsized":"2024-04-22T00:00:27.358076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357668+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-22T00:00:27.357668+0000","last_peered":"2024-04-22T00:00:27.357668+0000","last_clean":"2024-04-22T00:00:27.357668+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-22T00:00:27.357668+0000","last_undegraded":"2024-04-22T00:00:27.357668+0000","last_fullsized":"2024-04-22T00:00:27.357668+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360162+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:00:27.360162+0000","last_peered":"2024-04-22T00:00:27.360162+0000","last_clean":"2024-04-22T00:00:27.360162+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:00:27.360162+0000","last_undegraded":"2024-04-22T00:00:27.360162+0000","last_fullsized":"2024-04-22T00:00:27.360162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355018+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-22T00:00:27.355018+0000","last_peered":"2024-04-22T00:00:27.355018+0000","last_clean":"2024-04-22T00:00:27.355018+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-22T00:00:27.355018+0000","last_undegraded":"2024-04-22T00:00:27.355018+0000","last_fullsized":"2024-04-22T00:00:27.355018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352622+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-22T00:00:27.352622+0000","last_peered":"2024-04-22T00:00:27.352622+0000","last_clean":"2024-04-22T00:00:27.352622+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-22T00:00:27.352622+0000","last_undegraded":"2024-04-22T00:00:27.352622+0000","last_fullsized":"2024-04-22T00:00:27.352622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":214,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378457+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:01:16.378457+0000","last_peered":"2024-04-22T00:01:16.378457+0000","last_clean":"2024-04-22T00:01:16.378457+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:01:16.378457+0000","last_undegraded":"2024-04-22T00:01:16.378457+0000","last_fullsized":"2024-04-22T00:01:16.378457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":137,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355154+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-22T00:00:27.355154+0000","last_peered":"2024-04-22T00:00:27.355154+0000","last_clean":"2024-04-22T00:00:27.355154+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-22T00:00:27.355154+0000","last_undegraded":"2024-04-22T00:00:27.355154+0000","last_fullsized":"2024-04-22T00:00:27.355154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361264+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-22T00:00:27.361264+0000","last_peered":"2024-04-22T00:00:27.361264+0000","last_clean":"2024-04-22T00:00:27.361264+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-22T00:00:27.361264+0000","last_undegraded":"2024-04-22T00:00:27.361264+0000","last_fullsized":"2024-04-22T00:00:27.361264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352497+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:00:27.352497+0000","last_peered":"2024-04-22T00:00:27.352497+0000","last_clean":"2024-04-22T00:00:27.352497+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:00:27.352497+0000","last_undegraded":"2024-04-22T00:00:27.352497+0000","last_fullsized":"2024-04-22T00:00:27.352497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354975+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-22T00:00:27.354975+0000","last_peered":"2024-04-22T00:00:27.354975+0000","last_clean":"2024-04-22T00:00:27.354975+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-22T00:00:27.354975+0000","last_undegraded":"2024-04-22T00:00:27.354975+0000","last_fullsized":"2024-04-22T00:00:27.354975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355607+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-22T00:00:27.355607+0000","last_peered":"2024-04-22T00:00:27.355607+0000","last_clean":"2024-04-22T00:00:27.355607+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-22T00:00:27.355607+0000","last_undegraded":"2024-04-22T00:00:27.355607+0000","last_fullsized":"2024-04-22T00:00:27.355607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359646+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:00:27.359646+0000","last_peered":"2024-04-22T00:00:27.359646+0000","last_clean":"2024-04-22T00:00:27.359646+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:00:27.359646+0000","last_undegraded":"2024-04-22T00:00:27.359646+0000","last_fullsized":"2024-04-22T00:00:27.359646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360979+0000","last_change":"2024-04-22T00:00:27.360979+0000","last_active":"2024-04-22T00:00:27.360979+0000","last_peered":"2024-04-22T00:00:27.360979+0000","last_clean":"2024-04-22T00:00:27.360979+0000","last_became_active":"2024-04-22T00:00:27.360589+0000","last_became_peered":"2024-04-22T00:00:27.360589+0000","last_unstale":"2024-04-22T00:00:27.360979+0000","last_undegraded":"2024-04-22T00:00:27.360979+0000","last_fullsized":"2024-04-22T00:00:27.360979+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:25.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353018+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-22T00:00:27.353018+0000","last_peered":"2024-04-22T00:00:27.353018+0000","last_clean":"2024-04-22T00:00:27.353018+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-22T00:00:27.353018+0000","last_undegraded":"2024-04-22T00:00:27.353018+0000","last_fullsized":"2024-04-22T00:00:27.353018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:14.375988+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-22T00:01:14.375988+0000","last_peered":"2024-04-22T00:01:14.375988+0000","last_clean":"2024-04-22T00:01:14.375988+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-22T00:01:14.375988+0000","last_undegraded":"2024-04-22T00:01:14.375988+0000","last_fullsized":"2024-04-22T00:01:14.375988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355982+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-22T00:00:27.355982+0000","last_peered":"2024-04-22T00:00:27.355982+0000","last_clean":"2024-04-22T00:00:27.355982+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-22T00:00:27.355982+0000","last_undegraded":"2024-04-22T00:00:27.355982+0000","last_fullsized":"2024-04-22T00:00:27.355982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356239+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-22T00:00:27.356239+0000","last_peered":"2024-04-22T00:00:27.356239+0000","last_clean":"2024-04-22T00:00:27.356239+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-22T00:00:27.356239+0000","last_undegraded":"2024-04-22T00:00:27.356239+0000","last_fullsized":"2024-04-22T00:00:27.356239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354526+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:00:27.354526+0000","last_peered":"2024-04-22T00:00:27.354526+0000","last_clean":"2024-04-22T00:00:27.354526+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:00:27.354526+0000","last_undegraded":"2024-04-22T00:00:27.354526+0000","last_fullsized":"2024-04-22T00:00:27.354526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359881+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-22T00:00:27.359881+0000","last_peered":"2024-04-22T00:00:27.359881+0000","last_clean":"2024-04-22T00:00:27.359881+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-22T00:00:27.359881+0000","last_undegraded":"2024-04-22T00:00:27.359881+0000","last_fullsized":"2024-04-22T00:00:27.359881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359718+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-22T00:00:27.359718+0000","last_peered":"2024-04-22T00:00:27.359718+0000","last_clean":"2024-04-22T00:00:27.359718+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-22T00:00:27.359718+0000","last_undegraded":"2024-04-22T00:00:27.359718+0000","last_fullsized":"2024-04-22T00:00:27.359718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355292+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-22T00:00:27.355292+0000","last_peered":"2024-04-22T00:00:27.355292+0000","last_clean":"2024-04-22T00:00:27.355292+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-22T00:00:27.355292+0000","last_undegraded":"2024-04-22T00:00:27.355292+0000","last_fullsized":"2024-04-22T00:00:27.355292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356170+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-22T00:00:27.356170+0000","last_peered":"2024-04-22T00:00:27.356170+0000","last_clean":"2024-04-22T00:00:27.356170+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-22T00:00:27.356170+0000","last_undegraded":"2024-04-22T00:00:27.356170+0000","last_fullsized":"2024-04-22T00:00:27.356170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354964+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-22T00:00:27.354964+0000","last_peered":"2024-04-22T00:00:27.354964+0000","last_clean":"2024-04-22T00:00:27.354964+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-22T00:00:27.354964+0000","last_undegraded":"2024-04-22T00:00:27.354964+0000","last_fullsized":"2024-04-22T00:00:27.354964+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360956+0000","last_change":"2024-04-22T00:00:27.360956+0000","last_active":"2024-04-22T00:00:27.360956+0000","last_peered":"2024-04-22T00:00:27.360956+0000","last_clean":"2024-04-22T00:00:27.360956+0000","last_became_active":"2024-04-22T00:00:27.360481+0000","last_became_peered":"2024-04-22T00:00:27.360481+0000","last_unstale":"2024-04-22T00:00:27.360956+0000","last_undegraded":"2024-04-22T00:00:27.360956+0000","last_fullsized":"2024-04-22T00:00:27.360956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:16.624366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491742+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-22T00:00:27.491742+0000","last_peered":"2024-04-22T00:00:27.491742+0000","last_clean":"2024-04-22T00:00:27.491742+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-22T00:00:27.491742+0000","last_undegraded":"2024-04-22T00:00:27.491742+0000","last_fullsized":"2024-04-22T00:00:27.491742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355981+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-22T00:00:27.355981+0000","last_peered":"2024-04-22T00:00:27.355981+0000","last_clean":"2024-04-22T00:00:27.355981+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-22T00:00:27.355981+0000","last_undegraded":"2024-04-22T00:00:27.355981+0000","last_fullsized":"2024-04-22T00:00:27.355981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:16.378518+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-22T00:01:16.378518+0000","last_peered":"2024-04-22T00:01:16.378518+0000","last_clean":"2024-04-22T00:01:16.378518+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-22T00:01:16.378518+0000","last_undegraded":"2024-04-22T00:01:16.378518+0000","last_fullsized":"2024-04-22T00:01:16.378518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.491288+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-22T00:00:27.491288+0000","last_peered":"2024-04-22T00:00:27.491288+0000","last_clean":"2024-04-22T00:00:27.491288+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-22T00:00:27.491288+0000","last_undegraded":"2024-04-22T00:00:27.491288+0000","last_fullsized":"2024-04-22T00:00:27.491288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355981+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-22T00:00:27.355981+0000","last_peered":"2024-04-22T00:00:27.355981+0000","last_clean":"2024-04-22T00:00:27.355981+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-22T00:00:27.355981+0000","last_undegraded":"2024-04-22T00:00:27.355981+0000","last_fullsized":"2024-04-22T00:00:27.355981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361537+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-22T00:00:27.361537+0000","last_peered":"2024-04-22T00:00:27.361537+0000","last_clean":"2024-04-22T00:00:27.361537+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-22T00:00:27.361537+0000","last_undegraded":"2024-04-22T00:00:27.361537+0000","last_fullsized":"2024-04-22T00:00:27.361537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:15.376607+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-22T00:01:15.376607+0000","last_peered":"2024-04-22T00:01:15.376607+0000","last_clean":"2024-04-22T00:01:15.376607+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-22T00:01:15.376607+0000","last_undegraded":"2024-04-22T00:01:15.376607+0000","last_fullsized":"2024-04-22T00:01:15.376607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.369850+0000","last_change":"2024-04-22T00:00:27.369850+0000","last_active":"2024-04-22T00:00:27.369850+0000","last_peered":"2024-04-22T00:00:27.369850+0000","last_clean":"2024-04-22T00:00:27.369850+0000","last_became_active":"2024-04-22T00:00:27.361664+0000","last_became_peered":"2024-04-22T00:00:27.361664+0000","last_unstale":"2024-04-22T00:00:27.369850+0000","last_undegraded":"2024-04-22T00:00:27.369850+0000","last_fullsized":"2024-04-22T00:00:27.369850+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:42:13.794809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490505+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-22T00:00:27.490505+0000","last_peered":"2024-04-22T00:00:27.490505+0000","last_clean":"2024-04-22T00:00:27.490505+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-22T00:00:27.490505+0000","last_undegraded":"2024-04-22T00:00:27.490505+0000","last_fullsized":"2024-04-22T00:00:27.490505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357610+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-22T00:00:27.357610+0000","last_peered":"2024-04-22T00:00:27.357610+0000","last_clean":"2024-04-22T00:00:27.357610+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-22T00:00:27.357610+0000","last_undegraded":"2024-04-22T00:00:27.357610+0000","last_fullsized":"2024-04-22T00:00:27.357610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355819+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-22T00:00:27.355819+0000","last_peered":"2024-04-22T00:00:27.355819+0000","last_clean":"2024-04-22T00:00:27.355819+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-22T00:00:27.355819+0000","last_undegraded":"2024-04-22T00:00:27.355819+0000","last_fullsized":"2024-04-22T00:00:27.355819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359733+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-22T00:00:27.359733+0000","last_peered":"2024-04-22T00:00:27.359733+0000","last_clean":"2024-04-22T00:00:27.359733+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-22T00:00:27.359733+0000","last_undegraded":"2024-04-22T00:00:27.359733+0000","last_fullsized":"2024-04-22T00:00:27.359733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359209+0000","last_change":"2024-04-22T00:00:27.359209+0000","last_active":"2024-04-22T00:00:27.359209+0000","last_peered":"2024-04-22T00:00:27.359209+0000","last_clean":"2024-04-22T00:00:27.359209+0000","last_became_active":"2024-04-22T00:00:27.358750+0000","last_became_peered":"2024-04-22T00:00:27.358750+0000","last_unstale":"2024-04-22T00:00:27.359209+0000","last_undegraded":"2024-04-22T00:00:27.359209+0000","last_fullsized":"2024-04-22T00:00:27.359209+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:05.053039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355880+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:00:27.355880+0000","last_peered":"2024-04-22T00:00:27.355880+0000","last_clean":"2024-04-22T00:00:27.355880+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:00:27.355880+0000","last_undegraded":"2024-04-22T00:00:27.355880+0000","last_fullsized":"2024-04-22T00:00:27.355880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354859+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-22T00:00:27.354859+0000","last_peered":"2024-04-22T00:00:27.354859+0000","last_clean":"2024-04-22T00:00:27.354859+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-22T00:00:27.354859+0000","last_undegraded":"2024-04-22T00:00:27.354859+0000","last_fullsized":"2024-04-22T00:00:27.354859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.369109+0000","last_change":"2024-04-22T00:00:27.368767+0000","last_active":"2024-04-22T00:00:27.369109+0000","last_peered":"2024-04-22T00:00:27.369109+0000","last_clean":"2024-04-22T00:00:27.369109+0000","last_became_active":"2024-04-22T00:00:27.362160+0000","last_became_peered":"2024-04-22T00:00:27.362160+0000","last_unstale":"2024-04-22T00:00:27.369109+0000","last_undegraded":"2024-04-22T00:00:27.369109+0000","last_fullsized":"2024-04-22T00:00:27.369109+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:38.606596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356383+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-22T00:00:27.356383+0000","last_peered":"2024-04-22T00:00:27.356383+0000","last_clean":"2024-04-22T00:00:27.356383+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-22T00:00:27.356383+0000","last_undegraded":"2024-04-22T00:00:27.356383+0000","last_fullsized":"2024-04-22T00:00:27.356383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490300+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-22T00:00:27.490300+0000","last_peered":"2024-04-22T00:00:27.490300+0000","last_clean":"2024-04-22T00:00:27.490300+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-22T00:00:27.490300+0000","last_undegraded":"2024-04-22T00:00:27.490300+0000","last_fullsized":"2024-04-22T00:00:27.490300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.353337+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:00:27.353337+0000","last_peered":"2024-04-22T00:00:27.353337+0000","last_clean":"2024-04-22T00:00:27.353337+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:00:27.353337+0000","last_undegraded":"2024-04-22T00:00:27.353337+0000","last_fullsized":"2024-04-22T00:00:27.353337+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490676+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-22T00:00:27.490676+0000","last_peered":"2024-04-22T00:00:27.490676+0000","last_clean":"2024-04-22T00:00:27.490676+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-22T00:00:27.490676+0000","last_undegraded":"2024-04-22T00:00:27.490676+0000","last_fullsized":"2024-04-22T00:00:27.490676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352065+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-22T00:00:27.352065+0000","last_peered":"2024-04-22T00:00:27.352065+0000","last_clean":"2024-04-22T00:00:27.352065+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-22T00:00:27.352065+0000","last_undegraded":"2024-04-22T00:00:27.352065+0000","last_fullsized":"2024-04-22T00:00:27.352065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.359020+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-22T00:00:27.359020+0000","last_peered":"2024-04-22T00:00:27.359020+0000","last_clean":"2024-04-22T00:00:27.359020+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-22T00:00:27.359020+0000","last_undegraded":"2024-04-22T00:00:27.359020+0000","last_fullsized":"2024-04-22T00:00:27.359020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357957+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-22T00:00:27.357957+0000","last_peered":"2024-04-22T00:00:27.357957+0000","last_clean":"2024-04-22T00:00:27.357957+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-22T00:00:27.357957+0000","last_undegraded":"2024-04-22T00:00:27.357957+0000","last_fullsized":"2024-04-22T00:00:27.357957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358936+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-22T00:00:27.358936+0000","last_peered":"2024-04-22T00:00:27.358936+0000","last_clean":"2024-04-22T00:00:27.358936+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-22T00:00:27.358936+0000","last_undegraded":"2024-04-22T00:00:27.358936+0000","last_fullsized":"2024-04-22T00:00:27.358936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354792+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:00:27.354792+0000","last_peered":"2024-04-22T00:00:27.354792+0000","last_clean":"2024-04-22T00:00:27.354792+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:00:27.354792+0000","last_undegraded":"2024-04-22T00:00:27.354792+0000","last_fullsized":"2024-04-22T00:00:27.354792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354661+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-22T00:00:27.354661+0000","last_peered":"2024-04-22T00:00:27.354661+0000","last_clean":"2024-04-22T00:00:27.354661+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-22T00:00:27.354661+0000","last_undegraded":"2024-04-22T00:00:27.354661+0000","last_fullsized":"2024-04-22T00:00:27.354661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.492722+0000","last_change":"2024-04-22T00:00:27.492100+0000","last_active":"2024-04-22T00:00:27.492722+0000","last_peered":"2024-04-22T00:00:27.492722+0000","last_clean":"2024-04-22T00:00:27.492722+0000","last_became_active":"2024-04-22T00:00:27.491414+0000","last_became_peered":"2024-04-22T00:00:27.491414+0000","last_unstale":"2024-04-22T00:00:27.492722+0000","last_undegraded":"2024-04-22T00:00:27.492722+0000","last_fullsized":"2024-04-22T00:00:27.492722+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:16:32.280602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490378+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:00:27.490378+0000","last_peered":"2024-04-22T00:00:27.490378+0000","last_clean":"2024-04-22T00:00:27.490378+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:00:27.490378+0000","last_undegraded":"2024-04-22T00:00:27.490378+0000","last_fullsized":"2024-04-22T00:00:27.490378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354802+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-22T00:00:27.354802+0000","last_peered":"2024-04-22T00:00:27.354802+0000","last_clean":"2024-04-22T00:00:27.354802+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-22T00:00:27.354802+0000","last_undegraded":"2024-04-22T00:00:27.354802+0000","last_fullsized":"2024-04-22T00:00:27.354802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.497233+0000","last_change":"2024-04-22T00:00:27.497233+0000","last_active":"2024-04-22T00:00:27.497233+0000","last_peered":"2024-04-22T00:00:27.497233+0000","last_clean":"2024-04-22T00:00:27.497233+0000","last_became_active":"2024-04-22T00:00:27.492847+0000","last_became_peered":"2024-04-22T00:00:27.492847+0000","last_unstale":"2024-04-22T00:00:27.497233+0000","last_undegraded":"2024-04-22T00:00:27.497233+0000","last_fullsized":"2024-04-22T00:00:27.497233+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:25.742746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352162+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:00:27.352162+0000","last_peered":"2024-04-22T00:00:27.352162+0000","last_clean":"2024-04-22T00:00:27.352162+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:00:27.352162+0000","last_undegraded":"2024-04-22T00:00:27.352162+0000","last_fullsized":"2024-04-22T00:00:27.352162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490781+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-22T00:00:27.490781+0000","last_peered":"2024-04-22T00:00:27.490781+0000","last_clean":"2024-04-22T00:00:27.490781+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-22T00:00:27.490781+0000","last_undegraded":"2024-04-22T00:00:27.490781+0000","last_fullsized":"2024-04-22T00:00:27.490781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358857+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-22T00:00:27.358857+0000","last_peered":"2024-04-22T00:00:27.358857+0000","last_clean":"2024-04-22T00:00:27.358857+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-22T00:00:27.358857+0000","last_undegraded":"2024-04-22T00:00:27.358857+0000","last_fullsized":"2024-04-22T00:00:27.358857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354505+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:00:27.354505+0000","last_peered":"2024-04-22T00:00:27.354505+0000","last_clean":"2024-04-22T00:00:27.354505+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:00:27.354505+0000","last_undegraded":"2024-04-22T00:00:27.354505+0000","last_fullsized":"2024-04-22T00:00:27.354505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.357045+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-22T00:00:27.357045+0000","last_peered":"2024-04-22T00:00:27.357045+0000","last_clean":"2024-04-22T00:00:27.357045+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-22T00:00:27.357045+0000","last_undegraded":"2024-04-22T00:00:27.357045+0000","last_fullsized":"2024-04-22T00:00:27.357045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354396+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-22T00:00:27.354396+0000","last_peered":"2024-04-22T00:00:27.354396+0000","last_clean":"2024-04-22T00:00:27.354396+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-22T00:00:27.354396+0000","last_undegraded":"2024-04-22T00:00:27.354396+0000","last_fullsized":"2024-04-22T00:00:27.354396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:01:17.410021+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:01:17.410021+0000","last_peered":"2024-04-22T00:01:17.410021+0000","last_clean":"2024-04-22T00:01:17.410021+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:01:17.410021+0000","last_undegraded":"2024-04-22T00:01:17.410021+0000","last_fullsized":"2024-04-22T00:01:17.410021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358151+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-22T00:00:27.358151+0000","last_peered":"2024-04-22T00:00:27.358151+0000","last_clean":"2024-04-22T00:00:27.358151+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-22T00:00:27.358151+0000","last_undegraded":"2024-04-22T00:00:27.358151+0000","last_fullsized":"2024-04-22T00:00:27.358151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.358746+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-22T00:00:27.358746+0000","last_peered":"2024-04-22T00:00:27.358746+0000","last_clean":"2024-04-22T00:00:27.358746+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-22T00:00:27.358746+0000","last_undegraded":"2024-04-22T00:00:27.358746+0000","last_fullsized":"2024-04-22T00:00:27.358746+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360489+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:00:27.360489+0000","last_peered":"2024-04-22T00:00:27.360489+0000","last_clean":"2024-04-22T00:00:27.360489+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:00:27.360489+0000","last_undegraded":"2024-04-22T00:00:27.360489+0000","last_fullsized":"2024-04-22T00:00:27.360489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356513+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-22T00:00:27.356513+0000","last_peered":"2024-04-22T00:00:27.356513+0000","last_clean":"2024-04-22T00:00:27.356513+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-22T00:00:27.356513+0000","last_undegraded":"2024-04-22T00:00:27.356513+0000","last_fullsized":"2024-04-22T00:00:27.356513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.360458+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-22T00:00:27.360458+0000","last_peered":"2024-04-22T00:00:27.360458+0000","last_clean":"2024-04-22T00:00:27.360458+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-22T00:00:27.360458+0000","last_undegraded":"2024-04-22T00:00:27.360458+0000","last_fullsized":"2024-04-22T00:00:27.360458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356119+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-22T00:00:27.356119+0000","last_peered":"2024-04-22T00:00:27.356119+0000","last_clean":"2024-04-22T00:00:27.356119+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-22T00:00:27.356119+0000","last_undegraded":"2024-04-22T00:00:27.356119+0000","last_fullsized":"2024-04-22T00:00:27.356119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356817+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-22T00:00:27.356817+0000","last_peered":"2024-04-22T00:00:27.356817+0000","last_clean":"2024-04-22T00:00:27.356817+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-22T00:00:27.356817+0000","last_undegraded":"2024-04-22T00:00:27.356817+0000","last_fullsized":"2024-04-22T00:00:27.356817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354922+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-22T00:00:27.354922+0000","last_peered":"2024-04-22T00:00:27.354922+0000","last_clean":"2024-04-22T00:00:27.354922+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-22T00:00:27.354922+0000","last_undegraded":"2024-04-22T00:00:27.354922+0000","last_fullsized":"2024-04-22T00:00:27.354922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352219+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-22T00:00:27.352219+0000","last_peered":"2024-04-22T00:00:27.352219+0000","last_clean":"2024-04-22T00:00:27.352219+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-22T00:00:27.352219+0000","last_undegraded":"2024-04-22T00:00:27.352219+0000","last_fullsized":"2024-04-22T00:00:27.352219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352156+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-22T00:00:27.352156+0000","last_peered":"2024-04-22T00:00:27.352156+0000","last_clean":"2024-04-22T00:00:27.352156+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-22T00:00:27.352156+0000","last_undegraded":"2024-04-22T00:00:27.352156+0000","last_fullsized":"2024-04-22T00:00:27.352156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.362810+0000","last_change":"2024-04-22T00:00:27.362810+0000","last_active":"2024-04-22T00:00:27.362810+0000","last_peered":"2024-04-22T00:00:27.362810+0000","last_clean":"2024-04-22T00:00:27.362810+0000","last_became_active":"2024-04-22T00:00:27.362380+0000","last_became_peered":"2024-04-22T00:00:27.362380+0000","last_unstale":"2024-04-22T00:00:27.362810+0000","last_undegraded":"2024-04-22T00:00:27.362810+0000","last_fullsized":"2024-04-22T00:00:27.362810+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:09:30.391961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356294+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-22T00:00:27.356294+0000","last_peered":"2024-04-22T00:00:27.356294+0000","last_clean":"2024-04-22T00:00:27.356294+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-22T00:00:27.356294+0000","last_undegraded":"2024-04-22T00:00:27.356294+0000","last_fullsized":"2024-04-22T00:00:27.356294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.356715+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-22T00:00:27.356715+0000","last_peered":"2024-04-22T00:00:27.356715+0000","last_clean":"2024-04-22T00:00:27.356715+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-22T00:00:27.356715+0000","last_undegraded":"2024-04-22T00:00:27.356715+0000","last_fullsized":"2024-04-22T00:00:27.356715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.352827+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-22T00:00:27.352827+0000","last_peered":"2024-04-22T00:00:27.352827+0000","last_clean":"2024-04-22T00:00:27.352827+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-22T00:00:27.352827+0000","last_undegraded":"2024-04-22T00:00:27.352827+0000","last_fullsized":"2024-04-22T00:00:27.352827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490869+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-22T00:00:27.490869+0000","last_peered":"2024-04-22T00:00:27.490869+0000","last_clean":"2024-04-22T00:00:27.490869+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-22T00:00:27.490869+0000","last_undegraded":"2024-04-22T00:00:27.490869+0000","last_fullsized":"2024-04-22T00:00:27.490869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.361417+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-22T00:00:27.361417+0000","last_peered":"2024-04-22T00:00:27.361417+0000","last_clean":"2024-04-22T00:00:27.361417+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-22T00:00:27.361417+0000","last_undegraded":"2024-04-22T00:00:27.361417+0000","last_fullsized":"2024-04-22T00:00:27.361417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.490327+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-22T00:00:27.490327+0000","last_peered":"2024-04-22T00:00:27.490327+0000","last_clean":"2024-04-22T00:00:27.490327+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-22T00:00:27.490327+0000","last_undegraded":"2024-04-22T00:00:27.490327+0000","last_fullsized":"2024-04-22T00:00:27.490327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.354899+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-22T00:00:27.354899+0000","last_peered":"2024-04-22T00:00:27.354899+0000","last_clean":"2024-04-22T00:00:27.354899+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-22T00:00:27.354899+0000","last_undegraded":"2024-04-22T00:00:27.354899+0000","last_fullsized":"2024-04-22T00:00:27.354899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355985+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-22T00:00:27.355985+0000","last_peered":"2024-04-22T00:00:27.355985+0000","last_clean":"2024-04-22T00:00:27.355985+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-22T00:00:27.355985+0000","last_undegraded":"2024-04-22T00:00:27.355985+0000","last_fullsized":"2024-04-22T00:00:27.355985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T00:00:27.355867+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-22T00:00:27.355867+0000","last_peered":"2024-04-22T00:00:27.355867+0000","last_clean":"2024-04-22T00:00:27.355867+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-22T00:00:27.355867+0000","last_undegraded":"2024-04-22T00:00:27.355867+0000","last_fullsized":"2024-04-22T00:00:27.355867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69537,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":755,"num_read_kb":453,"num_write":243,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":501936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1155069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":501363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":505725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1650688,"data_stored":1098238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":560230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":498714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":560841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":501332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1654784,"data_stored":1089333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:01:22.022 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T00:01:22.022 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T00:01:22.022 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T00:01:22.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T00:01:22.276 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:22.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:22 smithi038 ceph-mon[27972]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:22.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:22 smithi203 ceph-mon[31431]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:22.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:22 smithi111 ceph-mon[31543]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:01:23.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:23 smithi038 ceph-mon[27972]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:23.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:23 smithi203 ceph-mon[31431]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:23.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:23 smithi111 ceph-mon[31543]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 -- 172.21.15.38:0/3048512888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c102040 msgr2=0x7ff48c105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 --2- 172.21.15.38:0/3048512888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c102040 0x7ff48c105e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0099c0 tx=0x7ff47c009cd0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 -- 172.21.15.38:0/3048512888 shutdown_connections 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 --2- 172.21.15.38:0/3048512888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c1064f0 0x7ff48c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 --2- 172.21.15.38:0/3048512888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 --2- 172.21.15.38:0/3048512888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c102040 0x7ff48c105e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 -- 172.21.15.38:0/3048512888 >> 172.21.15.38:0/3048512888 conn(0x7ff48c077f30 msgr2=0x7ff48c078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 -- 172.21.15.38:0/3048512888 shutdown_connections 2024-04-22T00:01:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.081+0000 7ff49081a700 1 -- 172.21.15.38:0/3048512888 wait complete. 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.082+0000 7ff49081a700 1 Processor -- start 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.082+0000 7ff49081a700 1 -- start start 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c102040 0x7ff48c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1114b0 con 0x7ff48c1064f0 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c111330 con 0x7ff48c101690 2024-04-22T00:01:24.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff49081a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c111630 con 0x7ff48c102040 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff48ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff48ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43342/0 (socket says 172.21.15.38:43342) 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.083+0000 7ff48ad9d700 1 -- 172.21.15.38:0/2951666316 learned_addr learned my addr 172.21.15.38:0/2951666316 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff489d9b700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c102040 0x7ff48c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48a59c700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 -- 172.21.15.38:0/2951666316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c102040 msgr2=0x7ff48c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c102040 0x7ff48c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 -- 172.21.15.38:0/2951666316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 msgr2=0x7ff48c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 -- 172.21.15.38:0/2951666316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c199020 con 0x7ff48c1064f0 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48a59c700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff48ad9d700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff484007f10 tx=0x7ff48400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:24.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484009e20 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c199310 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48400dbd0 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484005a60 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.084+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c1a55d0 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.085+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48c068f20 con 0x7ff48c1064f0 2024-04-22T00:01:24.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.087+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff48401a020 con 0x7ff48c1064f0 2024-04-22T00:01:24.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.088+0000 7ff47b7fe700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 0x7ff4740a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:24.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.088+0000 7ff48a59c700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 0x7ff4740a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:24.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.088+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff4840cf310 con 0x7ff48c1064f0 2024-04-22T00:01:24.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.088+0000 7ff48a59c700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 0x7ff4740a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff480005950 tx=0x7ff480005f50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:24.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.092+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff484010070 con 0x7ff48c1064f0 2024-04-22T00:01:24.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.344+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff48c02db30 con 0x7ff48c1064f0 2024-04-22T00:01:24.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.345+0000 7ff47b7fe700 1 -- 172.21.15.38:0/2951666316 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7ff4840192c0 con 0x7ff48c1064f0 2024-04-22T00:01:24.338 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:24.338 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T00:01:24.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 msgr2=0x7ff4740a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:24.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 0x7ff4740a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff480005950 tx=0x7ff480005f50 comp rx=0 tx=0).stop 2024-04-22T00:01:24.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 msgr2=0x7ff48c1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:24.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff484007f10 tx=0x7ff48400d3b0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 shutdown_connections 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4740a2cb0 0x7ff4740a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff48c1064f0 0x7ff48c1988c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff48c101690 0x7ff48c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 --2- 172.21.15.38:0/2951666316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c102040 0x7ff48c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 >> 172.21.15.38:0/2951666316 conn(0x7ff48c077f30 msgr2=0x7ff48c06acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 shutdown_connections 2024-04-22T00:01:24.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:24.347+0000 7ff49081a700 1 -- 172.21.15.38:0/2951666316 wait complete. 2024-04-22T00:01:25.120 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T00:01:25.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 -- 172.21.15.38:0/4017077500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 msgr2=0x7f7f60100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/4017077500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f60100e50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009900 tx=0x7f7f5002bcd0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 -- 172.21.15.38:0/4017077500 shutdown_connections 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/4017077500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60101390 0x7f7f60101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/4017077500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f60100e50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/4017077500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60100090 0x7f7f601004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 -- 172.21.15.38:0/4017077500 >> 172.21.15.38:0/4017077500 conn(0x7f7f600fbc30 msgr2=0x7f7f600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 -- 172.21.15.38:0/4017077500 shutdown_connections 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 -- 172.21.15.38:0/4017077500 wait complete. 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.241+0000 7f7f67d5c640 1 Processor -- start 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 -- start start 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60100090 0x7f7f60199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 0x7f7f601a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6010f620 con 0x7f7f601009f0 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6010f4a0 con 0x7f7f60100090 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f67d5c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6010f7a0 con 0x7f7f60101390 2024-04-22T00:01:25.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43350/0 (socket says 172.21.15.38:43350) 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 -- 172.21.15.38:0/1246041824 learned_addr learned my addr 172.21.15.38:0/1246041824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f65af8640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60100090 0x7f7f60199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f662f9640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 0x7f7f601a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 -- 172.21.15.38:0/1246041824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 msgr2=0x7f7f601a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 0x7f7f601a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 -- 172.21.15.38:0/1246041824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60100090 msgr2=0x7f7f60199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60100090 0x7f7f60199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 -- 172.21.15.38:0/1246041824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f601a1900 con 0x7f7f601009f0 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f662f9640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 0x7f7f601a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:25.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.242+0000 7f7f652f7640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005ab0 tx=0x7f7f50005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.243+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50041070 con 0x7f7f601009f0 2024-04-22T00:01:25.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.243+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f500054e0 con 0x7f7f601009f0 2024-04-22T00:01:25.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.243+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f601a1b90 con 0x7f7f601009f0 2024-04-22T00:01:25.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.243+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50034440 con 0x7f7f601009f0 2024-04-22T00:01:25.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.243+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f601a2080 con 0x7f7f601009f0 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.244+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f6010ade0 con 0x7f7f601009f0 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.245+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7f50044070 con 0x7f7f601009f0 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.245+0000 7f7f56ffd640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 0x7f7f4c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.245+0000 7f7f65af8640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 0x7f7f4c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.245+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7f500f1340 con 0x7f7f601009f0 2024-04-22T00:01:25.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.246+0000 7f7f65af8640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 0x7f7f4c0a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7f6019ab90 tx=0x7f7f5c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.250+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7f500bdf90 con 0x7f7f601009f0 2024-04-22T00:01:25.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.439+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f6002d3c0 con 0x7f7f601009f0 2024-04-22T00:01:25.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.441+0000 7f7f56ffd640 1 -- 172.21.15.38:0/1246041824 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7f7f500a6e20 con 0x7f7f601009f0 2024-04-22T00:01:25.434 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:25.434 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:25.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 msgr2=0x7f7f4c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 0x7f7f4c0a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7f6019ab90 tx=0x7f7f5c00b390 comp rx=0 tx=0).stop 2024-04-22T00:01:25.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 msgr2=0x7f7f6019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005ab0 tx=0x7f7f50005de0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 shutdown_connections 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7f4c0a2c20 0x7f7f4c0a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f60101390 0x7f7f601a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f601009f0 0x7f7f6019a200 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 --2- 172.21.15.38:0/1246041824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f60100090 0x7f7f60199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 >> 172.21.15.38:0/1246041824 conn(0x7f7f600fbc30 msgr2=0x7f7f600fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.443+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 shutdown_connections 2024-04-22T00:01:25.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.444+0000 7f7f67d5c640 1 -- 172.21.15.38:0/1246041824 wait complete. 2024-04-22T00:01:25.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:25 smithi038 ceph-mon[27972]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:25.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:25 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2951666316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T00:01:25.448 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:01:25.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:01:25.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:25 smithi203 ceph-mon[31431]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:25.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:25 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2951666316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T00:01:25.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 -- 172.21.15.38:0/4074324824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 msgr2=0x7f74f8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/4074324824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8100ac0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009900 tx=0x7f74f402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 -- 172.21.15.38:0/4074324824 shutdown_connections 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/4074324824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/4074324824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 0x7f74f8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/4074324824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8100ac0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 -- 172.21.15.38:0/4074324824 >> 172.21.15.38:0/4074324824 conn(0x7f74f80fbc30 msgr2=0x7f74f80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 -- 172.21.15.38:0/4074324824 shutdown_connections 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.561+0000 7f74ffdd7640 1 -- 172.21.15.38:0/4074324824 wait complete. 2024-04-22T00:01:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 Processor -- start 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 -- start start 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 0x7f74f8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810d440 con 0x7f74f81006a0 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810d2c0 con 0x7f74f8102250 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74ffdd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810d5c0 con 0x7f74f8101890 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43370/0 (socket says 172.21.15.38:43370) 2024-04-22T00:01:25.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 -- 172.21.15.38:0/2391648969 learned_addr learned my addr 172.21.15.38:0/2391648969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fd372640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 0x7f74f8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fe374640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 -- 172.21.15.38:0/2391648969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 msgr2=0x7f74f8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 0x7f74f8198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 -- 172.21.15.38:0/2391648969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 msgr2=0x7f74f819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fdb73640 1 -- 172.21.15.38:0/2391648969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f819f760 con 0x7f74f81006a0 2024-04-22T00:01:25.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.562+0000 7f74fe374640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74fdb73640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f74f4000c00 tx=0x7f74f4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4041070 con 0x7f74f81006a0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f40044d0 con 0x7f74f81006a0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f819f9f0 con 0x7f74f81006a0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4034440 con 0x7f74f81006a0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.563+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f819fee0 con 0x7f74f81006a0 2024-04-22T00:01:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.564+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f8107a60 con 0x7f74f81006a0 2024-04-22T00:01:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.565+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f74f4044070 con 0x7f74f81006a0 2024-04-22T00:01:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.566+0000 7f74eeffd640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 0x7f74e40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.566+0000 7f74fd372640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 0x7f74e40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.566+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74f40f1250 con 0x7f74f81006a0 2024-04-22T00:01:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.566+0000 7f74fd372640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 0x7f74e40a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f74f8199050 tx=0x7f74e8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.569+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f74f40bde00 con 0x7f74f81006a0 2024-04-22T00:01:25.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:25 smithi111 ceph-mon[31543]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:25.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:25 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2951666316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T00:01:25.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.783+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74f802d300 con 0x7f74f81006a0 2024-04-22T00:01:25.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.783+0000 7f74eeffd640 1 -- 172.21.15.38:0/2391648969 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f74f40bd700 con 0x7f74f81006a0 2024-04-22T00:01:25.776 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:01:25.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.785+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 msgr2=0x7f74e40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.785+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 0x7f74e40a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f74f8199050 tx=0x7f74e8005f90 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.785+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 msgr2=0x7f74f8197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.785+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f74f4000c00 tx=0x7f74f4005d50 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 shutdown_connections 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74e40a2c20 0x7f74e40a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74f8102250 0x7f74f819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f8101890 0x7f74f8198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 --2- 172.21.15.38:0/2391648969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74f81006a0 0x7f74f8197b40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 >> 172.21.15.38:0/2391648969 conn(0x7f74f80fbc30 msgr2=0x7f74f8104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 shutdown_connections 2024-04-22T00:01:25.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.786+0000 7f74ffdd7640 1 -- 172.21.15.38:0/2391648969 wait complete. 2024-04-22T00:01:25.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- 172.21.15.38:0/802619592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec104b90 msgr2=0x7f02ec106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/802619592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec104b90 0x7f02ec106f80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009900 tx=0x7f02e002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- 172.21.15.38:0/802619592 shutdown_connections 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/802619592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec1074c0 0x7f02ec1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/802619592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec104b90 0x7f02ec106f80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/802619592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- 172.21.15.38:0/802619592 >> 172.21.15.38:0/802619592 conn(0x7f02ec0fbbf0 msgr2=0x7f02ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- 172.21.15.38:0/802619592 shutdown_connections 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- 172.21.15.38:0/802619592 wait complete. 2024-04-22T00:01:25.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 Processor -- start 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- start start 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec104b90 0x7f02ec19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec10d3d0 con 0x7f02ec1074c0 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec10d250 con 0x7f02ec102260 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.902+0000 7f02f2dd6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02ec10d550 con 0x7f02ec104b90 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43398/0 (socket says 172.21.15.38:43398) 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 -- 172.21.15.38:0/3511521252 learned_addr learned my addr 172.21.15.38:0/3511521252 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:25.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02ebfff640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec104b90 0x7f02ec19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f0b72640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 -- 172.21.15.38:0/3511521252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec104b90 msgr2=0x7f02ec19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec104b90 0x7f02ec19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 -- 172.21.15.38:0/3511521252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 msgr2=0x7f02ec199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 -- 172.21.15.38:0/3511521252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ec1a1920 con 0x7f02ec1074c0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f0b72640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f1373640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00c970 tx=0x7f02dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc007bb0 con 0x7f02ec1074c0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02dc007d50 con 0x7f02ec1074c0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ec1a1c10 con 0x7f02ec1074c0 2024-04-22T00:01:25.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc018800 con 0x7f02ec1074c0 2024-04-22T00:01:25.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.903+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ec1a2160 con 0x7f02ec1074c0 2024-04-22T00:01:25.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.904+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ec108fb0 con 0x7f02ec1074c0 2024-04-22T00:01:25.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.905+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f02dc00fa50 con 0x7f02ec1074c0 2024-04-22T00:01:25.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.906+0000 7f02e9ffb640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 0x7f02d40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:25.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.906+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f02dc0cf630 con 0x7f02ec1074c0 2024-04-22T00:01:25.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.906+0000 7f02f0b72640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 0x7f02d40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:25.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.906+0000 7f02f0b72640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 0x7f02d40a5160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02ec19b1e0 tx=0x7f02e4009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:25.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:25.910+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f02dc014030 con 0x7f02ec1074c0 2024-04-22T00:01:26.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.094+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f02ec02d340 con 0x7f02ec1074c0 2024-04-22T00:01:26.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.094+0000 7f02e9ffb640 1 -- 172.21.15.38:0/3511521252 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f02dc09c0d0 con 0x7f02ec1074c0 2024-04-22T00:01:26.087 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:26.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 msgr2=0x7f02d40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 0x7f02d40a5160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f02ec19b1e0 tx=0x7f02e4009500 comp rx=0 tx=0).stop 2024-04-22T00:01:26.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 msgr2=0x7f02ec1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00c970 tx=0x7f02dc00ce30 comp rx=0 tx=0).stop 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 shutdown_connections 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f02d40a2ca0 0x7f02d40a5160 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02ec1074c0 0x7f02ec1a1280 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02ec104b90 0x7f02ec19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 --2- 172.21.15.38:0/3511521252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02ec102260 0x7f02ec199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 >> 172.21.15.38:0/3511521252 conn(0x7f02ec0fbbf0 msgr2=0x7f02ec1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 shutdown_connections 2024-04-22T00:01:26.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.096+0000 7f02f2dd6640 1 -- 172.21.15.38:0/3511521252 wait complete. 2024-04-22T00:01:26.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:01:26.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.212+0000 7f0ab0d37640 1 -- 172.21.15.38:0/3479838546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac1006a0 msgr2=0x7f0aac100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.212+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/3479838546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac1006a0 0x7f0aac100ac0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009900 tx=0x7f0a9c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:26.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 -- 172.21.15.38:0/3479838546 shutdown_connections 2024-04-22T00:01:26.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/3479838546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 0x7f0aac108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/3479838546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac101890 0x7f0aac101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/3479838546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac1006a0 0x7f0aac100ac0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 -- 172.21.15.38:0/3479838546 >> 172.21.15.38:0/3479838546 conn(0x7f0aac0fbc30 msgr2=0x7f0aac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 -- 172.21.15.38:0/3479838546 shutdown_connections 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 -- 172.21.15.38:0/3479838546 wait complete. 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 Processor -- start 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 -- start start 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac1006a0 0x7f0aac06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.213+0000 7f0ab0d37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 0x7f0aac19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac10c360 con 0x7f0aac101890 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac10c1e0 con 0x7f0aac102250 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0aac10c4e0 con 0x7f0aac1006a0 2024-04-22T00:01:26.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43414/0 (socket says 172.21.15.38:43414) 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 -- 172.21.15.38:0/2487970284 learned_addr learned my addr 172.21.15.38:0/2487970284 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aaad9d640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 0x7f0aac19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aaa59c640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac1006a0 0x7f0aac06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 -- 172.21.15.38:0/2487970284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac1006a0 msgr2=0x7f0aac06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac1006a0 0x7f0aac06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 -- 172.21.15.38:0/2487970284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 msgr2=0x7f0aac19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 0x7f0aac19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 -- 172.21.15.38:0/2487970284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aac19f760 con 0x7f0aac101890 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0aa9d9b640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d770 tx=0x7f0aa000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0017070 con 0x7f0aac101890 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aa0005500 con 0x7f0aac101890 2024-04-22T00:01:26.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aac19fa50 con 0x7f0aac101890 2024-04-22T00:01:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0004e00 con 0x7f0aac101890 2024-04-22T00:01:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.214+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aac19ffa0 con 0x7f0aac101890 2024-04-22T00:01:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.215+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aac1079e0 con 0x7f0aac101890 2024-04-22T00:01:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.217+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0aa0004fa0 con 0x7f0aac101890 2024-04-22T00:01:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.217+0000 7f0a9b7fe640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 0x7f0a940a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.218+0000 7f0aaa59c640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 0x7f0a940a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.218+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0aa00cf390 con 0x7f0aac101890 2024-04-22T00:01:26.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.218+0000 7f0aaa59c640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 0x7f0a940a5160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c02c170 tx=0x7f0a9c02df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.221+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0aa009be50 con 0x7f0aac101890 2024-04-22T00:01:26.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.286+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0aa0097dc0 con 0x7f0aac101890 2024-04-22T00:01:26.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.410+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0aac02d3c0 con 0x7f0aac101890 2024-04-22T00:01:26.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.411+0000 7f0a9b7fe640 1 -- 172.21.15.38:0/2487970284 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f0aa00c91e0 con 0x7f0aac101890 2024-04-22T00:01:26.404 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:01:26.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 msgr2=0x7f0a940a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 0x7f0a940a5160 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c02c170 tx=0x7f0a9c02df90 comp rx=0 tx=0).stop 2024-04-22T00:01:26.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 msgr2=0x7f0aac19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d770 tx=0x7f0aa000dc30 comp rx=0 tx=0).stop 2024-04-22T00:01:26.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 shutdown_connections 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0a940a2ca0 0x7f0a940a5160 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0aac102250 0x7f0aac19eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0aac101890 0x7f0aac19c860 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 --2- 172.21.15.38:0/2487970284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0aac1006a0 0x7f0aac06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 >> 172.21.15.38:0/2487970284 conn(0x7f0aac0fbc30 msgr2=0x7f0aac104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 shutdown_connections 2024-04-22T00:01:26.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.413+0000 7f0ab0d37640 1 -- 172.21.15.38:0/2487970284 wait complete. 2024-04-22T00:01:26.415 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:26 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1246041824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:26.415 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:26 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2391648969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.416 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:26 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3511521252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:01:26.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.528+0000 7fc075f37640 1 -- 172.21.15.38:0/2838444830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070101790 msgr2=0x7fc070101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.528+0000 7fc075f37640 1 --2- 172.21.15.38:0/2838444830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070101790 0x7fc070101bb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc060009ca0 tx=0x7fc06002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 -- 172.21.15.38:0/2838444830 shutdown_connections 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 --2- 172.21.15.38:0/2838444830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 0x7fc070109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 --2- 172.21.15.38:0/2838444830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070102980 0x7fc070102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 --2- 172.21.15.38:0/2838444830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070101790 0x7fc070101bb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 -- 172.21.15.38:0/2838444830 >> 172.21.15.38:0/2838444830 conn(0x7fc0700fcd20 msgr2=0x7fc0700ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 -- 172.21.15.38:0/2838444830 shutdown_connections 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 -- 172.21.15.38:0/2838444830 wait complete. 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 Processor -- start 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.529+0000 7fc075f37640 1 -- start start 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070101790 0x7fc0700754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 0x7fc070074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc07010d460 con 0x7fc070102980 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc07010d2e0 con 0x7fc070101790 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc07010d5e0 con 0x7fc070103340 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43430/0 (socket says 172.21.15.38:43430) 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 -- 172.21.15.38:0/4232467981 learned_addr learned my addr 172.21.15.38:0/4232467981 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06f7fe640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070101790 0x7fc0700754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06ffff640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 0x7fc070074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 -- 172.21.15.38:0/4232467981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 msgr2=0x7fc070074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 0x7fc070074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 -- 172.21.15.38:0/4232467981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070101790 msgr2=0x7fc0700754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070101790 0x7fc0700754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 -- 172.21.15.38:0/4232467981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc070074900 con 0x7fc070102980 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06effd640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c920 tx=0x7fc05800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058007c70 con 0x7fc070102980 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc058007e10 con 0x7fc070102980 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0701a1bf0 con 0x7fc070102980 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058018890 con 0x7fc070102980 2024-04-22T00:01:26.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.530+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0701a2190 con 0x7fc070102980 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.531+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc070108ab0 con 0x7fc070102980 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.532+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc058018a70 con 0x7fc070102980 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.533+0000 7fc06cff9640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 0x7fc05c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.533+0000 7fc06f7fe640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 0x7fc05c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.533+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc0580cf4d0 con 0x7fc070102980 2024-04-22T00:01:26.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.534+0000 7fc06f7fe640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 0x7fc05c0a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc06002c700 tx=0x7fc060032590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:26 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1246041824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:26.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:26 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2391648969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:26 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3511521252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.537+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc058014030 con 0x7fc070102980 2024-04-22T00:01:26.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:26 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1246041824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:26.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:26 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2391648969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:26 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3511521252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:26.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.722+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc070062340 con 0x7fc070102980 2024-04-22T00:01:26.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.723+0000 7fc06cff9640 1 -- 172.21.15.38:0/4232467981 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fc05809c0d0 con 0x7fc070102980 2024-04-22T00:01:26.716 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:26.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 msgr2=0x7fc05c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 0x7fc05c0a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc06002c700 tx=0x7fc060032590 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 msgr2=0x7fc070073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c920 tx=0x7fc05800cde0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 shutdown_connections 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc05c0a2c10 0x7fc05c0a50d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc070103340 0x7fc070074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc070102980 0x7fc070073b30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 --2- 172.21.15.38:0/4232467981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc070101790 0x7fc0700754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 >> 172.21.15.38:0/4232467981 conn(0x7fc0700fcd20 msgr2=0x7fc070105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 shutdown_connections 2024-04-22T00:01:26.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.725+0000 7fc075f37640 1 -- 172.21.15.38:0/4232467981 wait complete. 2024-04-22T00:01:26.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:01:26.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 -- 172.21.15.38:0/2831816294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 msgr2=0x7f04bc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 --2- 172.21.15.38:0/2831816294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc101cf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009900 tx=0x7f04b002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 -- 172.21.15.38:0/2831816294 shutdown_connections 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 --2- 172.21.15.38:0/2831816294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 --2- 172.21.15.38:0/2831816294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc101cf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 --2- 172.21.15.38:0/2831816294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 -- 172.21.15.38:0/2831816294 >> 172.21.15.38:0/2831816294 conn(0x7f04bc0fbbf0 msgr2=0x7f04bc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 -- 172.21.15.38:0/2831816294 shutdown_connections 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.838+0000 7f04c1427640 1 -- 172.21.15.38:0/2831816294 wait complete. 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 Processor -- start 2024-04-22T00:01:26.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 -- start start 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04bc10c300 con 0x7f04bc101870 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04bc10c180 con 0x7f04bc102230 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04c1427640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04bc10c480 con 0x7f04bc100680 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04ba7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04ba7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43452/0 (socket says 172.21.15.38:43452) 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.839+0000 7f04ba7fc640 1 -- 172.21.15.38:0/1913268581 learned_addr learned my addr 172.21.15.38:0/1913268581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04bb7fe640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04baffd640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 -- 172.21.15.38:0/1913268581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 msgr2=0x7f04bc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 -- 172.21.15.38:0/1913268581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 msgr2=0x7f04bc19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 -- 172.21.15.38:0/1913268581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04bc19f750 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04bb7fe640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04baffd640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04ba7fc640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f04b002ce70 tx=0x7f04b0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0041070 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04bc19f9e0 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b00054e0 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04bc19fed0 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.840+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b004b060 con 0x7f04bc101870 2024-04-22T00:01:26.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.841+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04bc1a0100 con 0x7f04bc101870 2024-04-22T00:01:26.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.842+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f04b0044070 con 0x7f04bc101870 2024-04-22T00:01:26.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.843+0000 7f04a3fff640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 0x7f04a40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:26.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.843+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f04b00f14b0 con 0x7f04bc101870 2024-04-22T00:01:26.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.843+0000 7f04baffd640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 0x7f04a40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:26.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.843+0000 7f04baffd640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 0x7f04a40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0058c0 tx=0x7f04ac005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:26.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:26.847+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f04b00be0b0 con 0x7f04bc101870 2024-04-22T00:01:27.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.070+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04bc1a02d0 con 0x7f04bc101870 2024-04-22T00:01:27.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.071+0000 7f04a3fff640 1 -- 172.21.15.38:0/1913268581 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f04b00bd9b0 con 0x7f04bc101870 2024-04-22T00:01:27.064 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:27.064 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:01:27.065 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:27.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 msgr2=0x7f04a40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 0x7f04a40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0058c0 tx=0x7f04ac005850 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 msgr2=0x7f04bc198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f04b002ce70 tx=0x7f04b0005e40 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 shutdown_connections 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f04a40a2b90 0x7f04a40a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04bc102230 0x7f04bc19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04bc101870 0x7f04bc198070 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 --2- 172.21.15.38:0/1913268581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04bc100680 0x7f04bc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 >> 172.21.15.38:0/1913268581 conn(0x7f04bc0fbbf0 msgr2=0x7f04bc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 shutdown_connections 2024-04-22T00:01:27.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.073+0000 7f04c1427640 1 -- 172.21.15.38:0/1913268581 wait complete. 2024-04-22T00:01:27.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:27.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 -- 172.21.15.38:0/2254183920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100107500 msgr2=0x7f61001098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 --2- 172.21.15.38:0/2254183920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100107500 0x7f61001098f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009ca0 tx=0x7f60f802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 -- 172.21.15.38:0/2254183920 shutdown_connections 2024-04-22T00:01:27.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 --2- 172.21.15.38:0/2254183920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100107500 0x7f61001098f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 --2- 172.21.15.38:0/2254183920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100104bd0 0x7f6100106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 --2- 172.21.15.38:0/2254183920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 -- 172.21.15.38:0/2254183920 >> 172.21.15.38:0/2254183920 conn(0x7f61000fbbf0 msgr2=0x7f61000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.189+0000 7f6107101640 1 -- 172.21.15.38:0/2254183920 shutdown_connections 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 -- 172.21.15.38:0/2254183920 wait complete. 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 Processor -- start 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 -- start start 2024-04-22T00:01:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 0x7f61001a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f610010d400 con 0x7f6100104bd0 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f610010d280 con 0x7f6100107500 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f6107101640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f610010d580 con 0x7f61001022a0 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f60fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f60fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43472/0 (socket says 172.21.15.38:43472) 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.190+0000 7f60fffff640 1 -- 172.21.15.38:0/1032361641 learned_addr learned my addr 172.21.15.38:0/1032361641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f6104e9d640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 -- 172.21.15.38:0/1032361641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 msgr2=0x7f6100199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 -- 172.21.15.38:0/1032361641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 msgr2=0x7f61001a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f610569e640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 0x7f61001a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 0x7f61001a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 -- 172.21.15.38:0/1032361641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61001a18d0 con 0x7f6100104bd0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f6104e9d640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f610569e640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 0x7f61001a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fffff640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d770 tx=0x7f60ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec017070 con 0x7f6100104bd0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61001a1bc0 con 0x7f6100104bd0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ec009c30 con 0x7f6100104bd0 2024-04-22T00:01:27.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec00f620 con 0x7f6100104bd0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.191+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61001a2160 con 0x7f6100104bd0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.192+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6100108ff0 con 0x7f6100104bd0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.194+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f60ec01a070 con 0x7f6100104bd0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.195+0000 7f60fdffb640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 0x7f60f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.195+0000 7f6104e9d640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 0x7f60f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.195+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f60ec0cef60 con 0x7f6100104bd0 2024-04-22T00:01:27.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.195+0000 7f6104e9d640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 0x7f60f00a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f610019ab60 tx=0x7f60f4005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.200+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f60ec009de0 con 0x7f6100104bd0 2024-04-22T00:01:27.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:27 smithi038 ceph-mon[27972]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:27.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:27 smithi038 ceph-mon[27972]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-04-22T00:01:27.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:27 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2487970284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:27 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4232467981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:27 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1913268581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:27.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.389+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6100062340 con 0x7f6100104bd0 2024-04-22T00:01:27.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.390+0000 7f60fdffb640 1 -- 172.21.15.38:0/1032361641 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7f60ec09bb60 con 0x7f6100104bd0 2024-04-22T00:01:27.383 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:27.383 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 msgr2=0x7f60f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 0x7f60f00a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f610019ab60 tx=0x7f60f4005e70 comp rx=0 tx=0).stop 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 msgr2=0x7f610019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d770 tx=0x7f60ec00dc30 comp rx=0 tx=0).stop 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 shutdown_connections 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f60f00a2b90 0x7f60f00a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6100107500 0x7f61001a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6100104bd0 0x7f610019a1d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 --2- 172.21.15.38:0/1032361641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61001022a0 0x7f6100199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 >> 172.21.15.38:0/1032361641 conn(0x7f61000fbbf0 msgr2=0x7f6100101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 shutdown_connections 2024-04-22T00:01:27.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.393+0000 7f6107101640 1 -- 172.21.15.38:0/1032361641 wait complete. 2024-04-22T00:01:27.397 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:01:27.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:01:27.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 -- 172.21.15.38:0/1579367327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2040102250 msgr2=0x7f2040108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 --2- 172.21.15.38:0/1579367327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2040102250 0x7f2040108ae0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2038009be0 tx=0x7f203802bef0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 -- 172.21.15.38:0/1579367327 shutdown_connections 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 --2- 172.21.15.38:0/1579367327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2040102250 0x7f2040108ae0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 --2- 172.21.15.38:0/1579367327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 0x7f2040101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 --2- 172.21.15.38:0/1579367327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20401006a0 0x7f2040100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 -- 172.21.15.38:0/1579367327 >> 172.21.15.38:0/1579367327 conn(0x7f20400fbc30 msgr2=0x7f20400fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 -- 172.21.15.38:0/1579367327 shutdown_connections 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.513+0000 7f20463bf640 1 -- 172.21.15.38:0/1579367327 wait complete. 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 Processor -- start 2024-04-22T00:01:27.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 -- start start 2024-04-22T00:01:27.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 0x7f2040197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040102250 0x7f204019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f204010c350 con 0x7f20401006a0 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f204010c1d0 con 0x7f2040102250 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f20463bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f204010c4d0 con 0x7f2040101890 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f203ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f203ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43504/0 (socket says 172.21.15.38:43504) 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.514+0000 7f203ffff640 1 -- 172.21.15.38:0/3158804043 learned_addr learned my addr 172.21.15.38:0/3158804043 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:27.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203f7fe640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 0x7f2040197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f204495c640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040102250 0x7f204019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 -- 172.21.15.38:0/3158804043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 msgr2=0x7f2040197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 0x7f2040197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 -- 172.21.15.38:0/3158804043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040102250 msgr2=0x7f204019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040102250 0x7f204019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 -- 172.21.15.38:0/3158804043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204019f6a0 con 0x7f20401006a0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203ffff640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f203000e9a0 tx=0x7f203000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203000cd80 con 0x7f20401006a0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204019f990 con 0x7f20401006a0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2030010910 con 0x7f20401006a0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030018a40 con 0x7f20401006a0 2024-04-22T00:01:27.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.515+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204019ff30 con 0x7f20401006a0 2024-04-22T00:01:27.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.516+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20401a0160 con 0x7f20401006a0 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.518+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2030010ac0 con 0x7f20401006a0 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.519+0000 7f203d7fa640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 0x7f20280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.519+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f20300cfdf0 con 0x7f20401006a0 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.519+0000 7f203f7fe640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 0x7f20280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.520+0000 7f203f7fe640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 0x7f20280a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2040198f90 tx=0x7f2034005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.522+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f203009c990 con 0x7f20401006a0 2024-04-22T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:27 smithi203 ceph-mon[31431]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:27 smithi203 ceph-mon[31431]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-04-22T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:27 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2487970284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:27 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4232467981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:27 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1913268581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:27.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:27 smithi111 ceph-mon[31543]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:27.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:27 smithi111 ceph-mon[31543]: mgrmap e18: x(active, since 9m), standbys: y, z 2024-04-22T00:01:27.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:27 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2487970284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:27 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4232467981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:27.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:27 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1913268581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:27.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.709+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20401a0330 con 0x7f20401006a0 2024-04-22T00:01:27.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.710+0000 7f203d7fa640 1 -- 172.21.15.38:0/3158804043 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f203009c290 con 0x7f20401006a0 2024-04-22T00:01:27.703 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 msgr2=0x7f20280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 0x7f20280a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2040198f90 tx=0x7f2034005f90 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 msgr2=0x7f2040197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f203000e9a0 tx=0x7f203000ee60 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 shutdown_connections 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f20280a2b90 0x7f20280a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2040102250 0x7f204019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2040101890 0x7f2040197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 --2- 172.21.15.38:0/3158804043 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20401006a0 0x7f2040197a80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 >> 172.21.15.38:0/3158804043 conn(0x7f20400fbc30 msgr2=0x7f20400fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 shutdown_connections 2024-04-22T00:01:27.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.712+0000 7f20463bf640 1 -- 172.21.15.38:0/3158804043 wait complete. 2024-04-22T00:01:27.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:01:27.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 -- 172.21.15.38:0/3430826401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c101870 msgr2=0x7fea6c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 --2- 172.21.15.38:0/3430826401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c101870 0x7fea6c101cf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fea54009900 tx=0x7fea5402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:27.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 -- 172.21.15.38:0/3430826401 shutdown_connections 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 --2- 172.21.15.38:0/3430826401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c102230 0x7fea6c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 --2- 172.21.15.38:0/3430826401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c101870 0x7fea6c101cf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 --2- 172.21.15.38:0/3430826401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c100680 0x7fea6c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 -- 172.21.15.38:0/3430826401 >> 172.21.15.38:0/3430826401 conn(0x7fea6c0fbbf0 msgr2=0x7fea6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 -- 172.21.15.38:0/3430826401 shutdown_connections 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.830+0000 7fea71d12640 1 -- 172.21.15.38:0/3430826401 wait complete. 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 Processor -- start 2024-04-22T00:01:27.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 -- start start 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c100680 0x7fea6c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c101870 0x7fea6c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c10c300 con 0x7fea6c102230 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c10c180 con 0x7fea6c101870 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea71d12640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c10c480 con 0x7fea6c100680 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43514/0 (socket says 172.21.15.38:43514) 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea6bfff640 1 -- 172.21.15.38:0/221627552 learned_addr learned my addr 172.21.15.38:0/221627552 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea6b7fe640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c100680 0x7fea6c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.831+0000 7fea62ffd640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c101870 0x7fea6c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 -- 172.21.15.38:0/221627552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c100680 msgr2=0x7fea6c104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c100680 0x7fea6c104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 -- 172.21.15.38:0/221627552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c101870 msgr2=0x7fea6c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c101870 0x7fea6c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 -- 172.21.15.38:0/221627552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6c103a20 con 0x7fea6c102230 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea6bfff640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea6400cad0 tx=0x7fea6400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6400ae20 con 0x7fea6c102230 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6c1a1c50 con 0x7fea6c102230 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea64005020 con 0x7fea6c102230 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6400f6b0 con 0x7fea6c102230 2024-04-22T00:01:27.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.832+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6c1a2170 con 0x7fea6c102230 2024-04-22T00:01:27.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.833+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6c068560 con 0x7fea6c102230 2024-04-22T00:01:27.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.834+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fea640051d0 con 0x7fea6c102230 2024-04-22T00:01:27.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.835+0000 7fea697fa640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 0x7fea580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:27.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.835+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fea640cf6c0 con 0x7fea6c102230 2024-04-22T00:01:27.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.835+0000 7fea6b7fe640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 0x7fea580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:27.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.836+0000 7fea6b7fe640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 0x7fea580a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea6c1016d0 tx=0x7fea5c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:27.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:27.839+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fea64014030 con 0x7fea6c102230 2024-04-22T00:01:28.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.024+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea6c1a2b30 con 0x7fea6c102230 2024-04-22T00:01:28.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.024+0000 7fea697fa640 1 -- 172.21.15.38:0/221627552 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fea6409bc70 con 0x7fea6c102230 2024-04-22T00:01:28.017 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 msgr2=0x7fea580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 0x7fea580a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fea6c1016d0 tx=0x7fea5c005f90 comp rx=0 tx=0).stop 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 msgr2=0x7fea6c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea6400cad0 tx=0x7fea6400cf90 comp rx=0 tx=0).stop 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 shutdown_connections 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fea580a2b90 0x7fea580a5050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea6c102230 0x7fea6c103190 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c101870 0x7fea6c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 --2- 172.21.15.38:0/221627552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea6c100680 0x7fea6c104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 >> 172.21.15.38:0/221627552 conn(0x7fea6c0fbbf0 msgr2=0x7fea6c068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 shutdown_connections 2024-04-22T00:01:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.026+0000 7fea71d12640 1 -- 172.21.15.38:0/221627552 wait complete. 2024-04-22T00:01:28.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 -- 172.21.15.38:0/1736212150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4100680 msgr2=0x7f81d4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 --2- 172.21.15.38:0/1736212150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4100680 0x7f81d4100aa0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009900 tx=0x7f81c402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 -- 172.21.15.38:0/1736212150 shutdown_connections 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 --2- 172.21.15.38:0/1736212150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4102230 0x7f81d4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 --2- 172.21.15.38:0/1736212150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 0x7f81d4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 --2- 172.21.15.38:0/1736212150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4100680 0x7f81d4100aa0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.142+0000 7f81d934f640 1 -- 172.21.15.38:0/1736212150 >> 172.21.15.38:0/1736212150 conn(0x7f81d40fbbf0 msgr2=0x7f81d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- 172.21.15.38:0/1736212150 shutdown_connections 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- 172.21.15.38:0/1736212150 wait complete. 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 Processor -- start 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- start start 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4100680 0x7f81d40694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 0x7f81d419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f81d410c300 con 0x7f81d4102230 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f81d410c180 con 0x7f81d4100680 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.143+0000 7f81d934f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81d410c480 con 0x7f81d4101870 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43530/0 (socket says 172.21.15.38:43530) 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 -- 172.21.15.38:0/904534824 learned_addr learned my addr 172.21.15.38:0/904534824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d2ffd640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4100680 0x7f81d40694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d27fc640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 0x7f81d419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 -- 172.21.15.38:0/904534824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 msgr2=0x7f81d419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 0x7f81d419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 -- 172.21.15.38:0/904534824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4100680 msgr2=0x7f81d40694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4100680 0x7f81d40694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 -- 172.21.15.38:0/904534824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d419f750 con 0x7f81d4102230 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d37fe640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009fd0 tx=0x7f81cc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc017070 con 0x7f81d4102230 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d419fa40 con 0x7f81d4102230 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81cc003e50 con 0x7f81d4102230 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.144+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0106c0 con 0x7f81d4102230 2024-04-22T00:01:28.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.145+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d419ff40 con 0x7f81d4102230 2024-04-22T00:01:28.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.146+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d4107a10 con 0x7f81d4102230 2024-04-22T00:01:28.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.147+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f81cc003970 con 0x7f81d4102230 2024-04-22T00:01:28.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.147+0000 7f81cbfff640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 0x7f81c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.147+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f81cc0cf0b0 con 0x7f81d4102230 2024-04-22T00:01:28.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.148+0000 7f81d2ffd640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 0x7f81c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.148+0000 7f81d2ffd640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 0x7f81c00a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.152+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f81cc09b660 con 0x7f81d4102230 2024-04-22T00:01:28.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.337+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f81d4062340 con 0x7f81d4102230 2024-04-22T00:01:28.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.337+0000 7f81cbfff640 1 -- 172.21.15.38:0/904534824 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f81cc09f960 con 0x7f81d4102230 2024-04-22T00:01:28.330 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:01:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 msgr2=0x7f81c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 0x7f81c00a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 msgr2=0x7f81d419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009fd0 tx=0x7f81cc00b260 comp rx=0 tx=0).stop 2024-04-22T00:01:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 shutdown_connections 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81c00a2b90 0x7f81c00a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81d4102230 0x7f81d419ee90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4101870 0x7f81d419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 --2- 172.21.15.38:0/904534824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81d4100680 0x7f81d40694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.339+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 >> 172.21.15.38:0/904534824 conn(0x7f81d40fbbf0 msgr2=0x7f81d4104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.340+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 shutdown_connections 2024-04-22T00:01:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.340+0000 7f81d934f640 1 -- 172.21.15.38:0/904534824 wait complete. 2024-04-22T00:01:28.345 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1032361641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:28.345 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3158804043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:28 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/221627552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 -- 172.21.15.38:0/3564971403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 msgr2=0x7fb884102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 --2- 172.21.15.38:0/3564971403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb884102e00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb874009900 tx=0x7fb87402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 -- 172.21.15.38:0/3564971403 shutdown_connections 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 --2- 172.21.15.38:0/3564971403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884103340 0x7fb884109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 --2- 172.21.15.38:0/3564971403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb884102e00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 --2- 172.21.15.38:0/3564971403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884101790 0x7fb884101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 -- 172.21.15.38:0/3564971403 >> 172.21.15.38:0/3564971403 conn(0x7fb8840fcd20 msgr2=0x7fb8840ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 -- 172.21.15.38:0/3564971403 shutdown_connections 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.459+0000 7fb889adf640 1 -- 172.21.15.38:0/3564971403 wait complete. 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 Processor -- start 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 -- start start 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 0x7fb884197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884103340 0x7fb88419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb88410d480 con 0x7fb884102980 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb88410d300 con 0x7fb884101790 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb889adf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb88410d600 con 0x7fb884103340 2024-04-22T00:01:28.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43540/0 (socket says 172.21.15.38:43540) 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 -- 172.21.15.38:0/3200200522 learned_addr learned my addr 172.21.15.38:0/3200200522 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb883fff640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884103340 0x7fb88419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb8837fe640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 0x7fb884197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 -- 172.21.15.38:0/3200200522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884103340 msgr2=0x7fb88419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884103340 0x7fb88419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 -- 172.21.15.38:0/3200200522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 msgr2=0x7fb884197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 0x7fb884197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb882ffd640 1 -- 172.21.15.38:0/3200200522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88419f790 con 0x7fb884102980 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.460+0000 7fb8837fe640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 0x7fb884197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb882ffd640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb874009230 tx=0x7fb874004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874041070 con 0x7fb884102980 2024-04-22T00:01:28.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88419fa20 con 0x7fb884102980 2024-04-22T00:01:28.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8740054e0 con 0x7fb884102980 2024-04-22T00:01:28.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8740346a0 con 0x7fb884102980 2024-04-22T00:01:28.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.461+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88419ff10 con 0x7fb884102980 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.462+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb884108bd0 con 0x7fb884102980 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.464+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb874044070 con 0x7fb884102980 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.464+0000 7fb880ff9640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 0x7fb86c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.464+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8740f1590 con 0x7fb884102980 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.464+0000 7fb8837fe640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 0x7fb86c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.465+0000 7fb8837fe640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 0x7fb86c0a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb878006fd0 tx=0x7fb878008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.468+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb8740be190 con 0x7fb884102980 2024-04-22T00:01:28.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1032361641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:28.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3158804043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:28 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/221627552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1032361641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:28.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3158804043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:28 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/221627552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.651+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb88402d170 con 0x7fb884102980 2024-04-22T00:01:28.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.652+0000 7fb880ff9640 1 -- 172.21.15.38:0/3200200522 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fb8740bda90 con 0x7fb884102980 2024-04-22T00:01:28.645 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 msgr2=0x7fb86c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 0x7fb86c0a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb878006fd0 tx=0x7fb878008040 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 msgr2=0x7fb8841980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb874009230 tx=0x7fb874004040 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 shutdown_connections 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb86c0a2b90 0x7fb86c0a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb884103340 0x7fb88419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb884102980 0x7fb8841980b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 --2- 172.21.15.38:0/3200200522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb884101790 0x7fb884197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 >> 172.21.15.38:0/3200200522 conn(0x7fb8840fcd20 msgr2=0x7fb884105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 shutdown_connections 2024-04-22T00:01:28.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.654+0000 7fb889adf640 1 -- 172.21.15.38:0/3200200522 wait complete. 2024-04-22T00:01:28.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 -- 172.21.15.38:0/833899535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c103340 msgr2=0x7f194c109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 --2- 172.21.15.38:0/833899535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c103340 0x7f194c109bd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1944009ca0 tx=0x7f194402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 -- 172.21.15.38:0/833899535 shutdown_connections 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 --2- 172.21.15.38:0/833899535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c103340 0x7f194c109bd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 --2- 172.21.15.38:0/833899535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c102980 0x7f194c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 --2- 172.21.15.38:0/833899535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c101790 0x7f194c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 -- 172.21.15.38:0/833899535 >> 172.21.15.38:0/833899535 conn(0x7f194c0fcd20 msgr2=0x7f194c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:28.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 -- 172.21.15.38:0/833899535 shutdown_connections 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.767+0000 7f195295c640 1 -- 172.21.15.38:0/833899535 wait complete. 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 Processor -- start 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 -- start start 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 0x7f194c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c103340 0x7f194c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d3f0 con 0x7f194c102980 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d270 con 0x7f194c101790 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f195295c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f194c10d570 con 0x7f194c103340 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f194b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f194b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43558/0 (socket says 172.21.15.38:43558) 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f194b7fe640 1 -- 172.21.15.38:0/2165684180 learned_addr learned my addr 172.21.15.38:0/2165684180 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f194bfff640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 0x7f194c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.768+0000 7f1950ef9640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c103340 0x7f194c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 -- 172.21.15.38:0/2165684180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c103340 msgr2=0x7f194c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c103340 0x7f194c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 -- 172.21.15.38:0/2165684180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 msgr2=0x7f194c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 0x7f194c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 -- 172.21.15.38:0/2165684180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c19f5a0 con 0x7f194c102980 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194bfff640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 0x7f194c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f194b7fe640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f194000c920 tx=0x7f194000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940007c70 con 0x7f194c102980 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c19f890 con 0x7f194c102980 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1940007e10 con 0x7f194c102980 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940018890 con 0x7f194c102980 2024-04-22T00:01:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.769+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c19fde0 con 0x7f194c102980 2024-04-22T00:01:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.770+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c1089e0 con 0x7f194c102980 2024-04-22T00:01:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.771+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1940018a50 con 0x7f194c102980 2024-04-22T00:01:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.772+0000 7f19497fa640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:28.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.772+0000 7f194bfff640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:28.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.772+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19400cf7c0 con 0x7f194c102980 2024-04-22T00:01:28.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.773+0000 7f194bfff640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 0x7f19340a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f193c005d00 tx=0x7f193c005c90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:28.776+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1940014030 con 0x7f194c102980 2024-04-22T00:01:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.011+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f194c02d170 con 0x7f194c102980 2024-04-22T00:01:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.011+0000 7f19497fa640 1 -- 172.21.15.38:0/2165684180 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f194000de20 con 0x7f194c102980 2024-04-22T00:01:29.004 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:29.004 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:01:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.013+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 msgr2=0x7f19340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.013+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 0x7f19340a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f193c005d00 tx=0x7f193c005c90 comp rx=0 tx=0).stop 2024-04-22T00:01:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.013+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 msgr2=0x7f194c197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.013+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f194000c920 tx=0x7f194000cde0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 shutdown_connections 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19340a2b90 0x7f19340a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f194c103340 0x7f194c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f194c102980 0x7f194c197ec0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 --2- 172.21.15.38:0/2165684180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f194c101790 0x7f194c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 >> 172.21.15.38:0/2165684180 conn(0x7f194c0fcd20 msgr2=0x7f194c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 shutdown_connections 2024-04-22T00:01:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.014+0000 7f195295c640 1 -- 172.21.15.38:0/2165684180 wait complete. 2024-04-22T00:01:29.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:29.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 -- 172.21.15.38:0/27058022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 msgr2=0x7fecd4106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 --2- 172.21.15.38:0/27058022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd4106f20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009ca0 tx=0x7fecc402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 -- 172.21.15.38:0/27058022 shutdown_connections 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 --2- 172.21.15.38:0/27058022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 0x7fecd41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 --2- 172.21.15.38:0/27058022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd4106f20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 --2- 172.21.15.38:0/27058022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 0x7fecd41045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.126+0000 7fecdaac2640 1 -- 172.21.15.38:0/27058022 >> 172.21.15.38:0/27058022 conn(0x7fecd40fbbf0 msgr2=0x7fecd40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- 172.21.15.38:0/27058022 shutdown_connections 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- 172.21.15.38:0/27058022 wait complete. 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 Processor -- start 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- start start 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 0x7fecd4101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 0x7fecd41006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fecd410d3f0 con 0x7fecd4104b30 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fecd410d270 con 0x7fecd4102200 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecdaac2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fecd410d570 con 0x7fecd4107460 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43570/0 (socket says 172.21.15.38:43570) 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.127+0000 7fecd3fff640 1 -- 172.21.15.38:0/1354680792 learned_addr learned my addr 172.21.15.38:0/1354680792 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd905f640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 0x7fecd41006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd885e640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 0x7fecd4101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 -- 172.21.15.38:0/1354680792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 msgr2=0x7fecd41006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 0x7fecd41006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 -- 172.21.15.38:0/1354680792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 msgr2=0x7fecd4101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 0x7fecd4101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 -- 172.21.15.38:0/1354680792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd4100f80 con 0x7fecd4104b30 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd3fff640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fecc40324c0 tx=0x7fecc4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4039040 con 0x7fecd4104b30 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc4044070 con 0x7fecd4104b30 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd41a1b50 con 0x7fecd4104b30 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40042a0 con 0x7fecd4104b30 2024-04-22T00:01:29.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.128+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd41a20a0 con 0x7fecd4104b30 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.129+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd40fd6e0 con 0x7fecd4104b30 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.130+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fecc4003970 con 0x7fecd4104b30 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.131+0000 7fecd1ffb640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 0x7fecc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.131+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fecc40f1950 con 0x7fecd4104b30 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.131+0000 7fecd885e640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 0x7fecc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.131+0000 7fecd885e640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 0x7fecc00a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fecbc005fd0 tx=0x7fecbc005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.135+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fecc40be580 con 0x7fecd4104b30 2024-04-22T00:01:29.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.321+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecd4062340 con 0x7fecd4104b30 2024-04-22T00:01:29.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.323+0000 7fecd1ffb640 1 -- 172.21.15.38:0/1354680792 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7fecc40bde80 con 0x7fecd4104b30 2024-04-22T00:01:29.316 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:29.316 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 msgr2=0x7fecc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 0x7fecc00a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fecbc005fd0 tx=0x7fecbc005850 comp rx=0 tx=0).stop 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 msgr2=0x7fecd41001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fecc40324c0 tx=0x7fecc4032f20 comp rx=0 tx=0).stop 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 shutdown_connections 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fecc00a2b90 0x7fecc00a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecd4107460 0x7fecd41006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecd4104b30 0x7fecd41001b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 --2- 172.21.15.38:0/1354680792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd4102200 0x7fecd4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 >> 172.21.15.38:0/1354680792 conn(0x7fecd40fbbf0 msgr2=0x7fecd4102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 shutdown_connections 2024-04-22T00:01:29.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.325+0000 7fecdaac2640 1 -- 172.21.15.38:0/1354680792 wait complete. 2024-04-22T00:01:29.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:01:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 -- 172.21.15.38:0/3403243793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c104bb0 msgr2=0x7f220c106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 --2- 172.21.15.38:0/3403243793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c104bb0 0x7f220c106fa0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2200009900 tx=0x7f220002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 -- 172.21.15.38:0/3403243793 shutdown_connections 2024-04-22T00:01:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 --2- 172.21.15.38:0/3403243793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c1074e0 0x7f220c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 --2- 172.21.15.38:0/3403243793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c104bb0 0x7f220c106fa0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 --2- 172.21.15.38:0/3403243793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c102280 0x7f220c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 -- 172.21.15.38:0/3403243793 >> 172.21.15.38:0/3403243793 conn(0x7f220c0fbc30 msgr2=0x7f220c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 -- 172.21.15.38:0/3403243793 shutdown_connections 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.439+0000 7f22147af640 1 -- 172.21.15.38:0/3403243793 wait complete. 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 Processor -- start 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 -- start start 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 0x7f220c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c104bb0 0x7f220c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d3f0 con 0x7f220c1074e0 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d270 con 0x7f220c102280 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f22147af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d570 con 0x7f220c104bb0 2024-04-22T00:01:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43584/0 (socket says 172.21.15.38:43584) 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 -- 172.21.15.38:0/968936045 learned_addr learned my addr 172.21.15.38:0/968936045 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f221254b640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 0x7f220c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2211d4a640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c104bb0 0x7f220c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 -- 172.21.15.38:0/968936045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c104bb0 msgr2=0x7f220c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c104bb0 0x7f220c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 -- 172.21.15.38:0/968936045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 msgr2=0x7f220c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 0x7f220c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 -- 172.21.15.38:0/968936045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c19f720 con 0x7f220c1074e0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f221254b640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 0x7f220c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.440+0000 7f2212d4c640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f220400eaa0 tx=0x7f220400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.441+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220400cdb0 con 0x7f220c1074e0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.441+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2204005020 con 0x7f220c1074e0 2024-04-22T00:01:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.441+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c19fa10 con 0x7f220c1074e0 2024-04-22T00:01:29.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.441+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22040116b0 con 0x7f220c1074e0 2024-04-22T00:01:29.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.441+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c19fee0 con 0x7f220c1074e0 2024-04-22T00:01:29.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.442+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f220c0689b0 con 0x7f220c1074e0 2024-04-22T00:01:29.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.443+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2204005560 con 0x7f220c1074e0 2024-04-22T00:01:29.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.443+0000 7f21ff7fe640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 0x7f21f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.443+0000 7f221254b640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 0x7f21f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.443+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f22040cf6b0 con 0x7f220c1074e0 2024-04-22T00:01:29.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.444+0000 7f221254b640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 0x7f21f80a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f22080058c0 tx=0x7f2208005ec0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.448+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2204016030 con 0x7f220c1074e0 2024-04-22T00:01:29.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:29 smithi203 ceph-mon[31431]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T00:01:29.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:29 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/904534824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:29 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3200200522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:29 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2165684180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:29.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:29 smithi111 ceph-mon[31543]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T00:01:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:29 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/904534824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:29 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3200200522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:29 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2165684180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:29.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.671+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f220c062340 con 0x7f220c1074e0 2024-04-22T00:01:29.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.672+0000 7f21ff7fe640 1 -- 172.21.15.38:0/968936045 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f220409c2b0 con 0x7f220c1074e0 2024-04-22T00:01:29.665 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:29.665 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:01:29.665 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 msgr2=0x7f21f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 0x7f21f80a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f22080058c0 tx=0x7f2208005ec0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 msgr2=0x7f220c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f220400eaa0 tx=0x7f220400ef60 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 shutdown_connections 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f21f80a2b90 0x7f21f80a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f220c1074e0 0x7f220c19f0b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f220c104bb0 0x7f220c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 --2- 172.21.15.38:0/968936045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f220c102280 0x7f220c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 >> 172.21.15.38:0/968936045 conn(0x7f220c0fbc30 msgr2=0x7f220c102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 shutdown_connections 2024-04-22T00:01:29.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.674+0000 7f22147af640 1 -- 172.21.15.38:0/968936045 wait complete. 2024-04-22T00:01:29.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:29 smithi038 ceph-mon[27972]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-22T00:01:29.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:29 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/904534824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:29 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3200200522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:29.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:29 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2165684180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:29.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:59:36.029248+0000', 'modified': '2024-04-22T00:00:05.266438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24487, 'mds_2': 24460, 'mds_3': 24455, 'mds_4': 24448}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/3894275312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 3894275312}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 3894275312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14634': {'gid': 14634, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6841/3401657425', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6840', 'nonce': 3401657425}, {'type': 'v1', 'addr': '172.21.15.38:6841', 'nonce': 3401657425}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6833/618180230', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6832', 'nonce': 618180230}, {'type': 'v1', 'addr': '172.21.15.111:6833', 'nonce': 618180230}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24409': {'gid': 24409, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/1279747281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 1279747281}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 1279747281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/3450283861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 3450283861}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 3450283861}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.203:6835/521403621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 521403621}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 521403621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/782972998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 782972998}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 782972998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/1809489419', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 1809489419}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1809489419}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.203:6839/3602809941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3602809941}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3602809941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3890542559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3890542559}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3890542559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24487, 24500, 24460, 24455, 24448]} 2024-04-22T00:01:29.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T00:01:29.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 -- 172.21.15.38:0/995365255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 msgr2=0x7f6228100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 --2- 172.21.15.38:0/995365255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f6228100ac0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f621c009900 tx=0x7f621c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 -- 172.21.15.38:0/995365255 shutdown_connections 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 --2- 172.21.15.38:0/995365255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 0x7f6228108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 --2- 172.21.15.38:0/995365255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 0x7f6228101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 --2- 172.21.15.38:0/995365255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f6228100ac0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 -- 172.21.15.38:0/995365255 >> 172.21.15.38:0/995365255 conn(0x7f62280fbc30 msgr2=0x7f62280fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 -- 172.21.15.38:0/995365255 shutdown_connections 2024-04-22T00:01:29.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.792+0000 7f622ec7d640 1 -- 172.21.15.38:0/995365255 wait complete. 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 Processor -- start 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 -- start start 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 0x7f622819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 0x7f622819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f622810c320 con 0x7f62281006a0 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f622810c1a0 con 0x7f6228101890 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ec7d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f622810c4a0 con 0x7f6228102250 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.793+0000 7f622ca19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43596/0 (socket says 172.21.15.38:43596) 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 -- 172.21.15.38:0/4116251892 learned_addr learned my addr 172.21.15.38:0/4116251892 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f6227fff640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 0x7f622819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622d21a640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 0x7f622819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 -- 172.21.15.38:0/4116251892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 msgr2=0x7f622819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 0x7f622819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 -- 172.21.15.38:0/4116251892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 msgr2=0x7f622819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 0x7f622819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 -- 172.21.15.38:0/4116251892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622819b4f0 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ca19640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f621c005ab0 tx=0x7f621c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f6225ffb640 1 -- 172.21.15.38:0/4116251892 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c041070 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622819b780 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f6225ffb640 1 -- 172.21.15.38:0/4116251892 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621c0054e0 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f6225ffb640 1 -- 172.21.15.38:0/4116251892 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c034440 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.794+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622819bc70 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.796+0000 7f6225ffb640 1 -- 172.21.15.38:0/4116251892 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f621c044070 con 0x7f62281006a0 2024-04-22T00:01:29.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 Processor -- start 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 -- start start 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00070 0x563598b00490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b05560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563598a64df0 con 0x563598b05bd0 2024-04-22T00:01:29.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563598a0d640 con 0x563598b00a60 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6231867740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563598a60db0 con 0x563598b00070 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6225ffb640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 0x7f62180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6227fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f622d21a640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 0x7f62180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6227fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43598/0 (socket says 172.21.15.38:43598) 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6227fff640 1 -- 172.21.15.38:0/2770305590 learned_addr learned my addr 172.21.15.38:0/2770305590 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f6225ffb640 1 -- 172.21.15.38:0/4116251892 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f621c0f0f10 con 0x7f62281006a0 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f622d21a640 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00070 0x563598b00490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.797+0000 7f622ca19640 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b05560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 -- 172.21.15.38:0/2770305590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00070 msgr2=0x563598b00490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00070 0x563598b00490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 -- 172.21.15.38:0/2770305590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 msgr2=0x563598b05560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b05560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 -- 172.21.15.38:0/2770305590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635987792d0 con 0x563598b05bd0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f622d21a640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 0x7f62180a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6220005cc0 tx=0x7f622000ad50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6227fff640 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f621400bd10 tx=0x7f62140107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30ce86c5cae804f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6212ffd640 1 -- 172.21.15.38:0/2770305590 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214005220 con 0x563598b05bd0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6212ffd640 1 -- 172.21.15.38:0/2770305590 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6214004c80 con 0x563598b05bd0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.798+0000 7f6212ffd640 1 -- 172.21.15.38:0/2770305590 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214011820 con 0x563598b05bd0 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- 172.21.15.38:0/2770305590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 msgr2=0x563598b08060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f621400bd10 tx=0x7f62140107a0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- 172.21.15.38:0/2770305590 shutdown_connections 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b08060 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b05560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- 172.21.15.38:0/2770305590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00070 0x563598b00490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- 172.21.15.38:0/2770305590 >> 172.21.15.38:0/2770305590 conn(0x563598afb5e0 msgr2=0x563598afb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- 172.21.15.38:0/2770305590 shutdown_connections 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- 172.21.15.38:0/2770305590 wait complete. 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 -- start start 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 Processor -- start 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b0aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.799+0000 7f6231867740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b11a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563598b08b80 con 0x563598b00070 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563598a20be0 con 0x563598b00a60 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563598b08d00 con 0x563598b05bd0 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 10 client.0 _ll_register_callbacks cb 0x563598a205d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43610/0 (socket says 172.21.15.38:43610) 2024-04-22T00:01:29.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 -- 172.21.15.38:0/3389980498 learned_addr learned my addr 172.21.15.38:0/3389980498 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6227fff640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b0aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622d21a640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b11a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 -- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 msgr2=0x563598b11a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b11a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 -- 172.21.15.38:0/3389980498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 msgr2=0x563598b0aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b0aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 -- 172.21.15.38:0/3389980498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635988436b0 con 0x563598b00070 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622d21a640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b11a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f622ca19640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f621c0bd1c0 tx=0x7f621c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddae9d086ee9e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6207fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c04b040 con 0x563598b00070 2024-04-22T00:01:29.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f621c0c5bf0 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6214004c80 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f621c0bce20 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 10 client.15216 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.800+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563598b149d0 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6214004c80 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f621c0d6120 con 0x563598b00070 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 10 client.15216 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 10 client.15216 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 20 client.15216 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b170e0 0x563598b18550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 4 client.15216 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563598a72330 con 0x563598b170e0 2024-04-22T00:01:29.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.801+0000 7f6227fff640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b170e0 0x563598b18550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.802+0000 7f6227fff640 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b170e0 0x563598b18550 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ffea83ea7b4ef9ed server_cookie=8f637b0e5abc5c51 in_seq=0 out_seq=0 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.802+0000 7f6207fff640 10 client.15216 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.802+0000 7f6207fff640 1 -- 172.21.15.38:0/3389980498 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563598a72330 con 0x563598b170e0 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.802+0000 7f6207fff640 10 client.15216 handle_command_reply: tid=0 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.802+0000 7f6231867740 1 client.15216 shutdown 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 msgr2=0x563598b0a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f621c0bd1c0 tx=0x7f621c009680 comp rx=0 tx=0).stop 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 shutdown_connections 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b170e0 0x563598b18550 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b11a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00a60 0x563598b0aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 --2- 172.21.15.38:0/3389980498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00070 0x563598b0a500 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6207fff640 0 client.15216 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:29.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 >> 172.21.15.38:0/3389980498 conn(0x563598afb5e0 msgr2=0x563598afa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 shutdown_connections 2024-04-22T00:01:29.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 1 -- 172.21.15.38:0/3389980498 wait complete. 2024-04-22T00:01:29.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.803+0000 7f6231867740 20 client.15216 trim_cache size 0 max 16384 2024-04-22T00:01:29.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 Processor -- start 2024-04-22T00:01:29.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 -- start start 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00a60 0x563598b11a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563598b16890 con 0x563598b05bd0 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563598b16f60 con 0x563598b00070 2024-04-22T00:01:29.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f6231867740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563598b17630 con 0x563598b00a60 2024-04-22T00:01:29.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f622d21a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f622d21a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43616/0 (socket says 172.21.15.38:43616) 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f622d21a640 1 -- 172.21.15.38:0/1820410468 learned_addr learned my addr 172.21.15.38:0/1820410468 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.821+0000 7f622ca19640 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f6227fff640 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00a60 0x563598b11a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 -- 172.21.15.38:0/1820410468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00a60 msgr2=0x563598b11a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00a60 0x563598b11a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 -- 172.21.15.38:0/1820410468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 msgr2=0x563598b0a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 -- 172.21.15.38:0/1820410468 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563598b00e40 con 0x563598b05bd0 2024-04-22T00:01:29.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f622d21a640 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f62200141e0 tx=0x7f6220004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6105513d441d5e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f62127fc640 1 -- 172.21.15.38:0/1820410468 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220025070 con 0x563598b05bd0 2024-04-22T00:01:29.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f62127fc640 1 -- 172.21.15.38:0/1820410468 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62200142d0 con 0x563598b05bd0 2024-04-22T00:01:29.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.822+0000 7f62127fc640 1 -- 172.21.15.38:0/1820410468 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622002ae60 con 0x563598b05bd0 2024-04-22T00:01:29.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- 172.21.15.38:0/1820410468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 msgr2=0x563598b0a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f62200141e0 tx=0x7f6220004ab0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- 172.21.15.38:0/1820410468 shutdown_connections 2024-04-22T00:01:29.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b05bd0 0x563598b0a650 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b00a60 0x563598b11a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- 172.21.15.38:0/1820410468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- 172.21.15.38:0/1820410468 >> 172.21.15.38:0/1820410468 conn(0x563598afb5e0 msgr2=0x563598b15d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- 172.21.15.38:0/1820410468 shutdown_connections 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- 172.21.15.38:0/1820410468 wait complete. 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- start start 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 Processor -- start 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b10d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563598b183d0 con 0x563598b00a60 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563598b17d00 con 0x563598b00070 2024-04-22T00:01:29.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563598b18aa0 con 0x563598b05bd0 2024-04-22T00:01:29.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6231867740 10 client.0 _ll_register_callbacks cb 0x7f621c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:29.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.823+0000 7f6227fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43622/0 (socket says 172.21.15.38:43622) 2024-04-22T00:01:29.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 -- 172.21.15.38:0/3102568202 learned_addr learned my addr 172.21.15.38:0/3102568202 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f622ca19640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 -- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 msgr2=0x563598b10d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b10d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 -- 172.21.15.38:0/3102568202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 msgr2=0x563598b0dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 -- 172.21.15.38:0/3102568202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563598b11360 con 0x563598b00a60 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6227fff640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6214011010 tx=0x7f621400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbdd64f338b6316 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6207fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214005510 con 0x563598b00a60 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6214004020 con 0x563598b00a60 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62200142d0 con 0x563598b00a60 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6214033050 con 0x563598b00a60 2024-04-22T00:01:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6231867740 10 client.15222 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563598b7d7a0 con 0x563598b00a60 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.824+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62200142d0 con 0x563598b00a60 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f62140035f0 con 0x563598b00a60 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 10 client.15222 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 10 client.15222 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 20 client.15222 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b822c0 0x563598b7e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 4 client.15222 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "status"}) -- 0x563598a6c680 con 0x563598b822c0 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f622ca19640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b822c0 0x563598b7e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f622ca19640 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b822c0 0x563598b7e700 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2feb5706c790717a server_cookie=8839c8667242b868 in_seq=0 out_seq=0 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.825+0000 7f6207fff640 10 client.15222 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6207fff640 1 -- 172.21.15.38:0/3102568202 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563598a6c680 con 0x563598b822c0 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6207fff640 10 client.15222 handle_command_reply: tid=0 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 client.15222 shutdown 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 msgr2=0x563598b0e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6214011010 tx=0x7f621400be00 comp rx=0 tx=0).stop 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 shutdown_connections 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x563598b822c0 0x563598b7e700 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563598b05bd0 0x563598b10d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563598b00a60 0x563598b0e1b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6207fff640 0 client.15222 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 --2- 172.21.15.38:0/3102568202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563598b00070 0x563598b0dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 >> 172.21.15.38:0/3102568202 conn(0x563598afb5e0 msgr2=0x563598af9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 shutdown_connections 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 1 -- 172.21.15.38:0/3102568202 wait complete. 2024-04-22T00:01:29.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f6231867740 20 client.15222 trim_cache size 0 max 16384 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 4, 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24448, 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 85.562421667999999, 2024-04-22T00:01:29.852 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:29.853 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:29.853 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:29.853 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 136.154164826 2024-04-22T00:01:29.853 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:29.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 msgr2=0x7f62180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 0x7f62180a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6220005cc0 tx=0x7f622000ad50 comp rx=0 tx=0).stop 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 msgr2=0x7f622806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f621c005ab0 tx=0x7f621c005dc0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 shutdown_connections 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f62180a2b90 0x7f62180a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6228102250 0x7f622819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6228101890 0x7f622819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 --2- 172.21.15.38:0/4116251892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62281006a0 0x7f622806a910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.826+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 >> 172.21.15.38:0/4116251892 conn(0x7f62280fbc30 msgr2=0x7f6228104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.827+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 shutdown_connections 2024-04-22T00:01:29.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.827+0000 7f622ec7d640 1 -- 172.21.15.38:0/4116251892 wait complete. 2024-04-22T00:01:29.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T00:01:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 -- 172.21.15.38:0/862012791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0102990 msgr2=0x7f3ac0102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/862012791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0102990 0x7f3ac0102e10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009900 tx=0x7f3aa802bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 -- 172.21.15.38:0/862012791 shutdown_connections 2024-04-22T00:01:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/862012791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0103350 0x7f3ac0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/862012791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0102990 0x7f3ac0102e10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/862012791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 -- 172.21.15.38:0/862012791 >> 172.21.15.38:0/862012791 conn(0x7f3ac00fcd10 msgr2=0x7f3ac00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 -- 172.21.15.38:0/862012791 shutdown_connections 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.984+0000 7f3ac4f13640 1 -- 172.21.15.38:0/862012791 wait complete. 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 Processor -- start 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 -- start start 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 0x7f3ac0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac010d400 con 0x7f3ac0103350 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac010d280 con 0x7f3ac01017a0 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3ac4f13640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac010d580 con 0x7f3ac0102990 2024-04-22T00:01:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43640/0 (socket says 172.21.15.38:43640) 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 -- 172.21.15.38:0/3900436517 learned_addr learned my addr 172.21.15.38:0/3900436517 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abe59c640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 0x7f3ac0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 -- 172.21.15.38:0/3900436517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 msgr2=0x7f3ac0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 0x7f3ac0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 -- 172.21.15.38:0/3900436517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 msgr2=0x7f3ac0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abed9d640 1 -- 172.21.15.38:0/3900436517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac019f6e0 con 0x7f3ac0103350 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 0x7f3ac0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.985+0000 7f3abe59c640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3abed9d640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000cad0 tx=0x7f3ab000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3ab77fe640 1 -- 172.21.15.38:0/3900436517 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000ae20 con 0x7f3ac0103350 2024-04-22T00:01:29.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3ab77fe640 1 -- 172.21.15.38:0/3900436517 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab0005020 con 0x7f3ac0103350 2024-04-22T00:01:29.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac019f9d0 con 0x7f3ac0103350 2024-04-22T00:01:29.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3ab77fe640 1 -- 172.21.15.38:0/3900436517 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000f6f0 con 0x7f3ac0103350 2024-04-22T00:01:29.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.986+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac019fed0 con 0x7f3ac0103350 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.988+0000 7f3ab77fe640 1 -- 172.21.15.38:0/3900436517 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3ab00051d0 con 0x7f3ac0103350 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.988+0000 7f3ac7bda740 1 Processor -- start 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 -- start start 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50ed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557db504cdf0 con 0x557db50e8070 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557db4ff5640 con 0x557db50e8a60 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557db5048db0 con 0x557db50edbd0 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ab77fe640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 0x7f3aac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43642/0 (socket says 172.21.15.38:43642) 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2181493027 learned_addr learned my addr 172.21.15.38:0/2181493027 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abed9d640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 0x7f3aac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2181493027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 msgr2=0x557db50f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2181493027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 msgr2=0x557db50ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50ed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2181493027 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557db4d612d0 con 0x557db50e8070 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3ab77fe640 1 -- 172.21.15.38:0/3900436517 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3ab00cf4e0 con 0x7f3ac0103350 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.989+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005ab0 tx=0x7f3aa8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1c38171cddb0bc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3abed9d640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 0x7f3aac0a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0097de0 tx=0x7f3ab006a900 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ab4ff9640 1 -- 172.21.15.38:0/2181493027 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa802cc00 con 0x557db50e8070 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ab4ff9640 1 -- 172.21.15.38:0/2181493027 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa8005040 con 0x557db50e8070 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ab4ff9640 1 -- 172.21.15.38:0/2181493027 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8004a60 con 0x557db50e8070 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2181493027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 msgr2=0x557db50e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005ab0 tx=0x7f3aa8005d50 comp rx=0 tx=0).stop 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2181493027 shutdown_connections 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2181493027 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50e8490 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.990+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2181493027 >> 172.21.15.38:0/2181493027 conn(0x557db50e35e0 msgr2=0x557db50e39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2181493027 shutdown_connections 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2181493027 wait complete. 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- start start 2024-04-22T00:01:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 Processor -- start 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8070 0x557db50f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557db50f0b20 con 0x557db50e8a60 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557db5008be0 con 0x557db50e8070 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557db50f0ca0 con 0x557db50edbd0 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3ac7bda740 10 client.0 _ll_register_callbacks cb 0x557db50085d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43652/0 (socket says 172.21.15.38:43652) 2024-04-22T00:01:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abe59c640 1 -- 172.21.15.38:0/2621851579 learned_addr learned my addr 172.21.15.38:0/2621851579 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8070 0x557db50f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.991+0000 7f3abe59c640 1 -- 172.21.15.38:0/2621851579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 msgr2=0x557db50f9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abe59c640 1 -- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8070 msgr2=0x557db50f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8070 0x557db50f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abe59c640 1 -- 172.21.15.38:0/2621851579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557db4e2b6b0 con 0x557db50e8a60 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3ac01027f0 tx=0x7f3ab80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3170667c4ef00b80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8005220 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3ab80053c0 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa8005040 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ab8025050 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 10 client.15234 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557db50fcbf0 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa8005040 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3ab80129b0 con 0x557db50e8a60 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 10 client.15234 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 10 client.15234 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 20 client.15234 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50ff2f0 0x557db5100760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 4 client.15234 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.992+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557db505a330 con 0x557db50ff2f0 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.993+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50ff2f0 0x557db5100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.993+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50ff2f0 0x557db5100760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1fa665e14c3c39cd server_cookie=a121794d34c229ba in_seq=0 out_seq=0 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.993+0000 7f3aa5ffb640 10 client.15234 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2621851579 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557db505a330 con 0x557db50ff2f0 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3aa5ffb640 10 client.15234 handle_command_reply: tid=0 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 client.15234 shutdown 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 msgr2=0x557db50f2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3ac01027f0 tx=0x7f3ab80117b0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 shutdown_connections 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50edbd0 0x557db50f9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50ff2f0 0x557db5100760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8a60 0x557db50f2af0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2621851579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8070 0x557db50f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3aa5ffb640 0 client.15234 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 >> 172.21.15.38:0/2621851579 conn(0x557db50e35e0 msgr2=0x557db50e2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 shutdown_connections 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2621851579 wait complete. 2024-04-22T00:01:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:29.994+0000 7f3ac7bda740 20 client.15234 trim_cache size 0 max 16384 2024-04-22T00:01:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 Processor -- start 2024-04-22T00:01:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 -- start start 2024-04-22T00:01:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e35e0 0x557db50f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557db50feaa0 con 0x557db50e8070 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557db50ff170 con 0x557db50e8a60 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.012+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557db50ff840 con 0x557db50e35e0 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43656/0 (socket says 172.21.15.38:43656) 2024-04-22T00:01:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 -- 172.21.15.38:0/3787671066 learned_addr learned my addr 172.21.15.38:0/3787671066 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 -- 172.21.15.38:0/3787671066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e35e0 msgr2=0x557db50f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e35e0 0x557db50f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 -- 172.21.15.38:0/3787671066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 msgr2=0x557db50f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 -- 172.21.15.38:0/3787671066 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557db50e4350 con 0x557db50e8070 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3abe59c640 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8012e50 tx=0x7f3ab8005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db2d3c7b814d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3aa7fff640 1 -- 172.21.15.38:0/3787671066 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8022040 con 0x557db50e8070 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3aa7fff640 1 -- 172.21.15.38:0/3787671066 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab801c480 con 0x557db50e8070 2024-04-22T00:01:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.013+0000 7f3aa7fff640 1 -- 172.21.15.38:0/3787671066 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8031660 con 0x557db50e8070 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- 172.21.15.38:0/3787671066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 msgr2=0x557db50f9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8012e50 tx=0x7f3ab8005690 comp rx=0 tx=0).stop 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- 172.21.15.38:0/3787671066 shutdown_connections 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e8070 0x557db50f9af0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/3787671066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e35e0 0x557db50f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- 172.21.15.38:0/3787671066 >> 172.21.15.38:0/3787671066 conn(0x557db50e9840 msgr2=0x557db50f21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- 172.21.15.38:0/3787671066 shutdown_connections 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- 172.21.15.38:0/3787671066 wait complete. 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- start start 2024-04-22T00:01:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 Processor -- start 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e8070 0x557db50f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557db51005e0 con 0x557db50e35e0 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557db50fff10 con 0x557db50e8a60 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557db5100cb0 con 0x557db50e8070 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3ac7bda740 10 client.0 _ll_register_callbacks cb 0x7f3ab800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3abdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3abdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43658/0 (socket says 172.21.15.38:43658) 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.014+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2419078274 learned_addr learned my addr 172.21.15.38:0/2419078274 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e8070 0x557db50f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2419078274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e8070 msgr2=0x557db50f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e8070 0x557db50f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 msgr2=0x557db50f89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 -- 172.21.15.38:0/2419078274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557db50f8fc0 con 0x557db50e35e0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3abdd9b640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3aa802cfd0 tx=0x7f3aa802caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efc24ba5bb06b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3aa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8050070 con 0x557db50e35e0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3aa8005040 con 0x557db50e35e0 2024-04-22T00:01:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ab801c480 con 0x557db50e35e0 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3aa80044c0 con 0x557db50e35e0 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3ac7bda740 10 client.15240 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557db5165530 con 0x557db50e35e0 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.015+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab801c480 con 0x557db50e35e0 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3aa80525a0 con 0x557db50e35e0 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 10 client.15240 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 20 client.15240 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50edbd0 0x557db5166140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 4 client.15240 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "status"}) -- 0x557db5054680 con 0x557db50edbd0 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.016+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50edbd0 0x557db5166140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3abe59c640 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50edbd0 0x557db5166140 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7ed61ffe3c46456a server_cookie=65899415408d3d77 in_seq=0 out_seq=0 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3aa5ffb640 10 client.15240 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3aa5ffb640 1 -- 172.21.15.38:0/2419078274 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557db5054680 con 0x557db50edbd0 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3aa5ffb640 10 client.15240 handle_command_reply: tid=0 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 client.15240 shutdown 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 msgr2=0x557db50f5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3aa802cfd0 tx=0x7f3aa802caf0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 shutdown_connections 2024-04-22T00:01:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557db50e8a60 0x557db50f89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557db50e8070 0x557db50f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x557db50edbd0 0x557db5166140 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 --2- 172.21.15.38:0/2419078274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557db50e35e0 0x557db50f5470 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3aa5ffb640 0 client.15240 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 >> 172.21.15.38:0/2419078274 conn(0x557db50e9840 msgr2=0x557db50e6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 shutdown_connections 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 1 -- 172.21.15.38:0/2419078274 wait complete. 2024-04-22T00:01:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac7bda740 20 client.15240 trim_cache size 0 max 16384 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 3, 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24455, 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:30.025 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 86.765419508999997, 2024-04-22T00:01:30.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:30.026 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:30.026 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:30.026 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 130.48803914600001 2024-04-22T00:01:30.026 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:30.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 msgr2=0x7f3aac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 0x7f3aac0a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0097de0 tx=0x7f3ab006a900 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 msgr2=0x7f3ac019f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000cad0 tx=0x7f3ab000cf90 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.017+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 shutdown_connections 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3aac0a2b90 0x7f3aac0a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ac0103350 0x7f3ac019f070 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac0102990 0x7f3ac0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 --2- 172.21.15.38:0/3900436517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ac01017a0 0x7f3ac0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 >> 172.21.15.38:0/3900436517 conn(0x7f3ac00fcd10 msgr2=0x7f3ac0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 shutdown_connections 2024-04-22T00:01:30.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.018+0000 7f3ac4f13640 1 -- 172.21.15.38:0/3900436517 wait complete. 2024-04-22T00:01:30.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T00:01:30.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 -- 172.21.15.38:0/2095908179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa600103350 msgr2=0x7fa600109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 --2- 172.21.15.38:0/2095908179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa600103350 0x7fa600109be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009ca0 tx=0x7fa5f802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 -- 172.21.15.38:0/2095908179 shutdown_connections 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 --2- 172.21.15.38:0/2095908179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa600103350 0x7fa600109be0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 --2- 172.21.15.38:0/2095908179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa600102990 0x7fa600102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 --2- 172.21.15.38:0/2095908179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6001017a0 0x7fa600101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 -- 172.21.15.38:0/2095908179 >> 172.21.15.38:0/2095908179 conn(0x7fa6000fcd10 msgr2=0x7fa6000ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 -- 172.21.15.38:0/2095908179 shutdown_connections 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.144+0000 7fa6078c2640 1 -- 172.21.15.38:0/2095908179 wait complete. 2024-04-22T00:01:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 Processor -- start 2024-04-22T00:01:30.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 -- start start 2024-04-22T00:01:30.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102990 0x7fa600197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa600103350 0x7fa60019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d420 con 0x7fa6001017a0 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d2a0 con 0x7fa600102990 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa60010d5a0 con 0x7fa600103350 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43676/0 (socket says 172.21.15.38:43676) 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 -- 172.21.15.38:0/1326467174 learned_addr learned my addr 172.21.15.38:0/1326467174 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 -- 172.21.15.38:0/1326467174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa600103350 msgr2=0x7fa60019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa600103350 0x7fa60019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 -- 172.21.15.38:0/1326467174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102990 msgr2=0x7fa600197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102990 0x7fa600197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 -- 172.21.15.38:0/1326467174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60019f660 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa60565e640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00c970 tx=0x7fa5fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa5f67fc640 1 -- 172.21.15.38:0/1326467174 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc007c10 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60019f950 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa5f67fc640 1 -- 172.21.15.38:0/1326467174 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5fc007db0 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.145+0000 7fa5f67fc640 1 -- 172.21.15.38:0/1326467174 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc00f8a0 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.146+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60019fef0 con 0x7fa6001017a0 2024-04-22T00:01:30.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa5f67fc640 1 -- 172.21.15.38:0/1326467174 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa5fc00fa60 con 0x7fa6001017a0 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 Processor -- start 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 -- start start 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7222490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c722a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d3c7186df0 con 0x55d3c7222a60 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d3c712f640 con 0x55d3c7222070 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60a582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3c7182db0 con 0x55d3c7227bd0 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa5f67fc640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 0x7fa5ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60565e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60565e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43684/0 (socket says 172.21.15.38:43684) 2024-04-22T00:01:30.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.148+0000 7fa60565e640 1 -- 172.21.15.38:0/3396857705 learned_addr learned my addr 172.21.15.38:0/3396857705 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa604e5d640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c722a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa605e5f640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 0x7fa5ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa605e5f640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7222490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa5f67fc640 1 -- 172.21.15.38:0/1326467174 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa5fc0cf7a0 con 0x7fa6001017a0 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 -- 172.21.15.38:0/3396857705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 msgr2=0x55d3c722a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c722a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 -- 172.21.15.38:0/3396857705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 msgr2=0x55d3c7222490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7222490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 -- 172.21.15.38:0/3396857705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c6e9b2d0 con 0x55d3c7222a60 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa604e5d640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c722a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa605e5f640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 0x7fa5ec0a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006010 tx=0x7fa5f80093f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa60565e640 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0a43e0 tx=0x7fa5fc018ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c598dadbd630f11f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa5eb7fe640 1 -- 172.21.15.38:0/3396857705 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0a00e0 con 0x55d3c7222a60 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa5eb7fe640 1 -- 172.21.15.38:0/3396857705 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5fc0a4910 con 0x55d3c7222a60 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.149+0000 7fa5eb7fe640 1 -- 172.21.15.38:0/3396857705 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0a4480 con 0x55d3c7222a60 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- 172.21.15.38:0/3396857705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 msgr2=0x55d3c7227560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0a43e0 tx=0x7fa5fc018ef0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- 172.21.15.38:0/3396857705 shutdown_connections 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c722a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7227560 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 --2- 172.21.15.38:0/3396857705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7222490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- 172.21.15.38:0/3396857705 >> 172.21.15.38:0/3396857705 conn(0x55d3c721d5e0 msgr2=0x55d3c721d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- 172.21.15.38:0/3396857705 shutdown_connections 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- 172.21.15.38:0/3396857705 wait complete. 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 -- start start 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.150+0000 7fa60a582740 1 Processor -- start 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c722c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 0x55d3c7233af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d3c722ab20 con 0x55d3c7222a60 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d3c7142be0 con 0x55d3c7227bd0 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3c722aca0 con 0x55d3c7222070 2024-04-22T00:01:30.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa60a582740 10 client.0 _ll_register_callbacks cb 0x55d3c71425d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43686/0 (socket says 172.21.15.38:43686) 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 -- 172.21.15.38:0/1223941707 learned_addr learned my addr 172.21.15.38:0/1223941707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa605e5f640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 0x55d3c7233af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 -- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 msgr2=0x55d3c722c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c722c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 -- 172.21.15.38:0/1223941707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 msgr2=0x55d3c7233af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 0x55d3c7233af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 -- 172.21.15.38:0/1223941707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c6f656b0 con 0x55d3c7222a60 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa605e5f640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 0x55d3c7233af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa604e5d640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000afd0 tx=0x7fa5f000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74a9c9fa9baf60c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa5e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.151+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0004220 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa5f00043c0 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5fc0a4910 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa5f00047e0 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 10 client.15252 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3c7236bd0 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5fc0a4910 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa5f000d7f0 con 0x55d3c7222a60 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 10 client.15252 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 10 client.15252 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T00:01:30.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 20 client.15252 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c7239360 0x55d3c723a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 4 client.15252 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3c7194330 con 0x55d3c7239360 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.152+0000 7fa60565e640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c7239360 0x55d3c723a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.153+0000 7fa60565e640 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c7239360 0x55d3c723a7d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d6a0dd1e5bb2696 server_cookie=e7495b4564ef428 in_seq=0 out_seq=0 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.153+0000 7fa5e8ff9640 10 client.15252 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.153+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/1223941707 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3c7194330 con 0x55d3c7239360 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.153+0000 7fa5e8ff9640 10 client.15252 handle_command_reply: tid=0 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 client.15252 shutdown 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 msgr2=0x55d3c722cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000afd0 tx=0x7fa5f000fad0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 shutdown_connections 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c7239360 0x55d3c723a7d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7227bd0 0x55d3c7233af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa5e8ff9640 0 client.15252 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c722cab0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 --2- 172.21.15.38:0/1223941707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c722c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 >> 172.21.15.38:0/1223941707 conn(0x55d3c721d5e0 msgr2=0x55d3c721c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 shutdown_connections 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 1 -- 172.21.15.38:0/1223941707 wait complete. 2024-04-22T00:01:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.154+0000 7fa60a582740 20 client.15252 trim_cache size 0 max 16384 2024-04-22T00:01:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 Processor -- start 2024-04-22T00:01:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 -- start start 2024-04-22T00:01:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7218340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c721d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d3c7238b10 con 0x55d3c7222a60 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d3c72391e0 con 0x55d3c7222070 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa60a582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3c72398b0 con 0x55d3c7227bd0 2024-04-22T00:01:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa604e5d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa604e5d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43700/0 (socket says 172.21.15.38:43700) 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.172+0000 7fa604e5d640 1 -- 172.21.15.38:0/3289073341 learned_addr learned my addr 172.21.15.38:0/3289073341 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 -- 172.21.15.38:0/3289073341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 msgr2=0x55d3c721d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c721d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 -- 172.21.15.38:0/3289073341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 msgr2=0x55d3c7218340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa60565e640 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7218340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7218340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 -- 172.21.15.38:0/3289073341 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3c7238940 con 0x55d3c7222a60 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa60565e640 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7218340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa604e5d640 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000dcb0 tx=0x7fa5f000a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b5e0b7ac93e6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa5eaffd640 1 -- 172.21.15.38:0/3289073341 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f002c070 con 0x55d3c7222a60 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa5eaffd640 1 -- 172.21.15.38:0/3289073341 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f000d9f0 con 0x55d3c7222a60 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.173+0000 7fa5eaffd640 1 -- 172.21.15.38:0/3289073341 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0018440 con 0x55d3c7222a60 2024-04-22T00:01:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- 172.21.15.38:0/3289073341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 msgr2=0x55d3c7233ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000dcb0 tx=0x7fa5f000a6b0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- 172.21.15.38:0/3289073341 shutdown_connections 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7227bd0 0x55d3c721d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7222a60 0x55d3c7233ab0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- 172.21.15.38:0/3289073341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222070 0x55d3c7218340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- 172.21.15.38:0/3289073341 >> 172.21.15.38:0/3289073341 conn(0x55d3c721d5e0 msgr2=0x55d3c7223500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- 172.21.15.38:0/3289073341 shutdown_connections 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- 172.21.15.38:0/3289073341 wait complete. 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- start start 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 Processor -- start 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c7231a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 0x55d3c7231f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d3c723a650 con 0x55d3c7227bd0 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d3c7239f80 con 0x55d3c7222a60 2024-04-22T00:01:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3c723ad20 con 0x55d3c7222070 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa60a582740 10 client.0 _ll_register_callbacks cb 0x7fa5f000ae90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa605e5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa605e5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43710/0 (socket says 172.21.15.38:43710) 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.174+0000 7fa605e5f640 1 -- 172.21.15.38:0/2133194654 learned_addr learned my addr 172.21.15.38:0/2133194654 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa60565e640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c7231a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 -- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 msgr2=0x55d3c7231a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa604e5d640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 0x55d3c7231f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c7231a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 -- 172.21.15.38:0/2133194654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 msgr2=0x55d3c7231f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 0x55d3c7231f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 -- 172.21.15.38:0/2133194654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3c72329c0 con 0x55d3c7227bd0 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa604e5d640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 0x55d3c7231f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa605e5f640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8036000 tx=0x7fa5f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e338f8abd5bb5f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa5e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8039040 con 0x55d3c7227bd0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa5f8042890 con 0x55d3c7227bd0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5f000d9f0 con 0x55d3c7227bd0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa5f805c050 con 0x55d3c7227bd0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa60a582740 10 client.15258 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3c72a7b20 con 0x55d3c7227bd0 2024-04-22T00:01:30.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.175+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f000d9f0 con 0x55d3c7227bd0 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa5f8057030 con 0x55d3c7227bd0 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 10 client.15258 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 10 client.15258 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 20 client.15258 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c72ac5a0 0x55d3c72ab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 4 client.15258 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "status"}) -- 0x55d3c718e680 con 0x55d3c72ac5a0 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60565e640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c72ac5a0 0x55d3c72ab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa5e8ff9640 10 client.15258 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60565e640 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c72ac5a0 0x55d3c72ab2b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2455312b98ca21a4 server_cookie=889fce67c4d2f81a in_seq=0 out_seq=0 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa5e8ff9640 1 -- 172.21.15.38:0/2133194654 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d3c718e680 con 0x55d3c72ac5a0 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa5e8ff9640 10 client.15258 handle_command_reply: tid=0 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.176+0000 7fa60a582740 1 client.15258 shutdown 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 msgr2=0x55d3c7232480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8036000 tx=0x7fa5f80095c0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 shutdown_connections 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d3c72ac5a0 0x55d3c72ab2b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d3c7227bd0 0x55d3c7232480 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c7222a60 0x55d3c7231f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa5e8ff9640 0 client.15258 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 --2- 172.21.15.38:0/2133194654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3c7222070 0x55d3c7231a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 >> 172.21.15.38:0/2133194654 conn(0x55d3c721d5e0 msgr2=0x55d3c7218340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 shutdown_connections 2024-04-22T00:01:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 1 -- 172.21.15.38:0/2133194654 wait complete. 2024-04-22T00:01:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa60a582740 20 client.15258 trim_cache size 0 max 16384 2024-04-22T00:01:30.191 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24460, 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:30.192 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 87.932400401999999, 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 133.63055196799999 2024-04-22T00:01:30.193 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 msgr2=0x7fa5ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 0x7fa5ec0a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006010 tx=0x7fa5f80093f0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 msgr2=0x7fa600197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc00c970 tx=0x7fa5fc00ce30 comp rx=0 tx=0).stop 2024-04-22T00:01:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 shutdown_connections 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fa5ec0a2b90 0x7fa5ec0a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa600103350 0x7fa60019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa600102990 0x7fa600197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 --2- 172.21.15.38:0/1326467174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6001017a0 0x7fa600197a40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 >> 172.21.15.38:0/1326467174 conn(0x7fa6000fcd10 msgr2=0x7fa600105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.177+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 shutdown_connections 2024-04-22T00:01:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.178+0000 7fa6078c2640 1 -- 172.21.15.38:0/1326467174 wait complete. 2024-04-22T00:01:30.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T00:01:30.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 -- 172.21.15.38:0/762537059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 msgr2=0x7fd738100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/762537059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd738100aa0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd728009900 tx=0x7fd72802bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 -- 172.21.15.38:0/762537059 shutdown_connections 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/762537059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/762537059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 0x7fd738101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/762537059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd738100aa0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 -- 172.21.15.38:0/762537059 >> 172.21.15.38:0/762537059 conn(0x7fd7380fbbf0 msgr2=0x7fd7380fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 -- 172.21.15.38:0/762537059 shutdown_connections 2024-04-22T00:01:30.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.314+0000 7fd73e7b8640 1 -- 172.21.15.38:0/762537059 wait complete. 2024-04-22T00:01:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 Processor -- start 2024-04-22T00:01:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 -- start start 2024-04-22T00:01:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 0x7fd738102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd73810c300 con 0x7fd738100680 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd73810c180 con 0x7fd738101870 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73e7b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd73810c480 con 0x7fd738102230 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43722/0 (socket says 172.21.15.38:43722) 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd737fff640 1 -- 172.21.15.38:0/1851368865 learned_addr learned my addr 172.21.15.38:0/1851368865 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.315+0000 7fd73cd55640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd7377fe640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 0x7fd738102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 -- 172.21.15.38:0/1851368865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 msgr2=0x7fd738103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 -- 172.21.15.38:0/1851368865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 msgr2=0x7fd738102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 0x7fd738102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 -- 172.21.15.38:0/1851368865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd738103a50 con 0x7fd738100680 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd73cd55640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:30.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd737fff640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd728000c00 tx=0x7fd728005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd7357fa640 1 -- 172.21.15.38:0/1851368865 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728041070 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7381a1b40 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd7357fa640 1 -- 172.21.15.38:0/1851368865 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7280044d0 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd7357fa640 1 -- 172.21.15.38:0/1851368865 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728034440 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.316+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7381a2050 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd7357fa640 1 -- 172.21.15.38:0/1851368865 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd728044070 con 0x7fd738100680 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 Processor -- start 2024-04-22T00:01:30.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 -- start start 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d102490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d10a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e80d066df0 con 0x55e80d102a60 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e80d00f640 con 0x55e80d107bd0 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd74139d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e80d062db0 con 0x55e80d102070 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43734/0 (socket says 172.21.15.38:43734) 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.319+0000 7fd737fff640 1 -- 172.21.15.38:0/852910927 learned_addr learned my addr 172.21.15.38:0/852910927 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd73cd55640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd7357fa640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 0x7fd7200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd7377fe640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d10a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 -- 172.21.15.38:0/852910927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 msgr2=0x55e80d102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd73cd55640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 0x7fd7200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 -- 172.21.15.38:0/852910927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 msgr2=0x55e80d10a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d10a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 -- 172.21.15.38:0/852910927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e80cd7b2d0 con 0x55e80d102a60 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd7377fe640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d10a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd7357fa640 1 -- 172.21.15.38:0/1851368865 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd7280f0ff0 con 0x7fd738100680 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd737fff640 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd7280b9950 tx=0x7fd72804d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e50bbe7f3bade7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd71e7fc640 1 -- 172.21.15.38:0/852910927 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72804b070 con 0x55e80d102a60 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd73cd55640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 0x7fd7200a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55e80d1032c0 tx=0x7fd730004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd71e7fc640 1 -- 172.21.15.38:0/852910927 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7280bd4f0 con 0x55e80d102a60 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.320+0000 7fd71e7fc640 1 -- 172.21.15.38:0/852910927 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280bdbf0 con 0x55e80d102a60 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- 172.21.15.38:0/852910927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 msgr2=0x55e80d107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd7280b9950 tx=0x7fd72804d000 comp rx=0 tx=0).stop 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- 172.21.15.38:0/852910927 shutdown_connections 2024-04-22T00:01:30.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d10a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d107560 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 --2- 172.21.15.38:0/852910927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- 172.21.15.38:0/852910927 >> 172.21.15.38:0/852910927 conn(0x55e80d0fd5e0 msgr2=0x55e80d0fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- 172.21.15.38:0/852910927 shutdown_connections 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- 172.21.15.38:0/852910927 wait complete. 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 -- start start 2024-04-22T00:01:30.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.321+0000 7fd74139d740 1 Processor -- start 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d10c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d113820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e80d10a940 con 0x55e80d102a60 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e80d022be0 con 0x55e80d107bd0 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e80d10aac0 con 0x55e80d102070 2024-04-22T00:01:30.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd74139d740 10 client.0 _ll_register_callbacks cb 0x55e80d0225d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43740/0 (socket says 172.21.15.38:43740) 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 -- 172.21.15.38:0/474647483 learned_addr learned my addr 172.21.15.38:0/474647483 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd73cd55640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd737fff640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d10c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 -- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 msgr2=0x55e80d10c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d10c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 -- 172.21.15.38:0/474647483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 msgr2=0x55e80d113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 -- 172.21.15.38:0/474647483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e80ce456b0 con 0x55e80d102a60 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7377fe640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0106d0 tx=0x7fd72c010ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0ddb6e114e6ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7177fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c011700 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.322+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd72c0118a0 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e80d074330 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd72c011aa0 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 10 client.15270 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e80d1168d0 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e80d074330 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd72c0193d0 con 0x55e80d102a60 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 10 client.15270 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 20 client.15270 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d119010 0x55e80d11a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 4 client.15270 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e80d06e680 con 0x55e80d119010 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.323+0000 7fd737fff640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d119010 0x55e80d11a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.324+0000 7fd737fff640 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d119010 0x55e80d11a480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c35dc1d373724ce server_cookie=4be3d4d785c7213c in_seq=0 out_seq=0 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.324+0000 7fd7177fe640 10 client.15270 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd7177fe640 1 -- 172.21.15.38:0/474647483 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e80d06e680 con 0x55e80d119010 2024-04-22T00:01:30.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd7177fe640 10 client.15270 handle_command_reply: tid=0 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 client.15270 shutdown 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 msgr2=0x55e80d10c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0106d0 tx=0x7fd72c010ad0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 shutdown_connections 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d119010 0x55e80d11a480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d107bd0 0x55e80d113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10c7e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 --2- 172.21.15.38:0/474647483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d10c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd7177fe640 0 client.15270 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 >> 172.21.15.38:0/474647483 conn(0x55e80d0fd5e0 msgr2=0x55e80d0fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 shutdown_connections 2024-04-22T00:01:30.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 1 -- 172.21.15.38:0/474647483 wait complete. 2024-04-22T00:01:30.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.325+0000 7fd74139d740 20 client.15270 trim_cache size 0 max 16384 2024-04-22T00:01:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.343+0000 7fd74139d740 1 Processor -- start 2024-04-22T00:01:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.343+0000 7fd74139d740 1 -- start start 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d0fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 0x55e80d1137e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e80d1187c0 con 0x55e80d107bd0 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e80d118e90 con 0x55e80d102a60 2024-04-22T00:01:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd74139d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e80d119560 con 0x55e80d102070 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43752/0 (socket says 172.21.15.38:43752) 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 -- 172.21.15.38:0/3265954677 learned_addr learned my addr 172.21.15.38:0/3265954677 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd7377fe640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 0x55e80d1137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd737fff640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d0fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 -- 172.21.15.38:0/3265954677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 msgr2=0x55e80d0fbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d0fbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 -- 172.21.15.38:0/3265954677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 msgr2=0x55e80d1137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 0x55e80d1137e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 -- 172.21.15.38:0/3265954677 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e80d1032f0 con 0x55e80d107bd0 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd7377fe640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 0x55e80d1137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd737fff640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d0fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd73cd55640 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e950 tx=0x7fd730009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b865d36d6a0930 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd71dffb640 1 -- 172.21.15.38:0/3265954677 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730010aa0 con 0x55e80d107bd0 2024-04-22T00:01:30.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd71dffb640 1 -- 172.21.15.38:0/3265954677 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd730025070 con 0x55e80d107bd0 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.344+0000 7fd71dffb640 1 -- 172.21.15.38:0/3265954677 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7300206b0 con 0x55e80d107bd0 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- 172.21.15.38:0/3265954677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 msgr2=0x55e80d0fc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e950 tx=0x7fd730009b20 comp rx=0 tx=0).stop 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- 172.21.15.38:0/3265954677 shutdown_connections 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d107bd0 0x55e80d0fc3d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102a60 0x55e80d1137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 --2- 172.21.15.38:0/3265954677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d102070 0x55e80d0fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- 172.21.15.38:0/3265954677 >> 172.21.15.38:0/3265954677 conn(0x55e80d0fd5e0 msgr2=0x55e80d0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- 172.21.15.38:0/3265954677 shutdown_connections 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- 172.21.15.38:0/3265954677 wait complete. 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.345+0000 7fd74139d740 1 -- start start 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 Processor -- start 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102070 0x55e80d10f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d107bd0 0x55e80d112ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e80d11a300 con 0x55e80d102a60 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e80d119c30 con 0x55e80d102070 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e80d11a9d0 con 0x55e80d107bd0 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd74139d740 10 client.0 _ll_register_callbacks cb 0x7fd72c003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43768/0 (socket says 172.21.15.38:43768) 2024-04-22T00:01:30.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 -- 172.21.15.38:0/3650128250 learned_addr learned my addr 172.21.15.38:0/3650128250 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd737fff640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102070 0x55e80d10f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd73cd55640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d107bd0 0x55e80d112ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 -- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d107bd0 msgr2=0x55e80d112ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d107bd0 0x55e80d112ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 -- 172.21.15.38:0/3650128250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102070 msgr2=0x55e80d10f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102070 0x55e80d10f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 -- 172.21.15.38:0/3650128250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e80d1130c0 con 0x55e80d102a60 2024-04-22T00:01:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7377fe640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd72c000c00 tx=0x7fd72c019dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473fc721bb5703dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.346+0000 7fd7177fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c02d070 con 0x55e80d102a60 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd730025070 con 0x55e80d102a60 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd72c0193d0 con 0x55e80d102a60 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd72c039050 con 0x55e80d102a60 2024-04-22T00:01:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 10 client.15276 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e80d17f620 con 0x55e80d102a60 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd730025070 con 0x55e80d102a60 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd72c032690 con 0x55e80d102a60 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 10 client.15276 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 20 client.15276 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d184260 0x55e80d182f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 4 client.15276 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "status"}) -- 0x55e80cfeca70 con 0x55e80d184260 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.347+0000 7fd737fff640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d184260 0x55e80d182f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd737fff640 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d184260 0x55e80d182f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e823972d50bcb5f server_cookie=97f3d43132131238 in_seq=0 out_seq=0 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd7177fe640 10 client.15276 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd7177fe640 1 -- 172.21.15.38:0/3650128250 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55e80cfeca70 con 0x55e80d184260 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd7177fe640 10 client.15276 handle_command_reply: tid=0 2024-04-22T00:01:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 client.15276 shutdown 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 msgr2=0x55e80d10fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd72c000c00 tx=0x7fd72c019dc0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 shutdown_connections 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e80d184260 0x55e80d182f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e80d107bd0 0x55e80d112ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e80d102a60 0x55e80d10fab0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd7177fe640 0 client.15276 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 --2- 172.21.15.38:0/3650128250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e80d102070 0x55e80d10f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 >> 172.21.15.38:0/3650128250 conn(0x55e80d0fd5e0 msgr2=0x55e80d0fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 shutdown_connections 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 1 -- 172.21.15.38:0/3650128250 wait complete. 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.348+0000 7fd74139d740 20 client.15276 trim_cache size 0 max 16384 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:30.357 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24487, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 89.111283318000005, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:30.358 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 127.98829261 2024-04-22T00:01:30.359 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 msgr2=0x7fd7200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 0x7fd7200a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55e80d1032c0 tx=0x7fd730004ab0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 msgr2=0x7fd7381045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd728000c00 tx=0x7fd728005d50 comp rx=0 tx=0).stop 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 shutdown_connections 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd7200a2b90 0x7fd7200a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd738102230 0x7fd738103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd738101870 0x7fd738102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 --2- 172.21.15.38:0/1851368865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd738100680 0x7fd7381045c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 >> 172.21.15.38:0/1851368865 conn(0x7fd7380fbbf0 msgr2=0x7fd7380fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 shutdown_connections 2024-04-22T00:01:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.349+0000 7fd73e7b8640 1 -- 172.21.15.38:0/1851368865 wait complete. 2024-04-22T00:01:30.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 -- 172.21.15.38:0/375876411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a701017a0 msgr2=0x7f5a70101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 --2- 172.21.15.38:0/375876411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a701017a0 0x7f5a70101bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009900 tx=0x7f5a6002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 -- 172.21.15.38:0/375876411 shutdown_connections 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 --2- 172.21.15.38:0/375876411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a70103350 0x7f5a70109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 --2- 172.21.15.38:0/375876411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70102990 0x7f5a70102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 --2- 172.21.15.38:0/375876411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a701017a0 0x7f5a70101bc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 -- 172.21.15.38:0/375876411 >> 172.21.15.38:0/375876411 conn(0x7f5a700fcd10 msgr2=0x7f5a700ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 -- 172.21.15.38:0/375876411 shutdown_connections 2024-04-22T00:01:30.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 -- 172.21.15.38:0/375876411 wait complete. 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.479+0000 7f5a75c78640 1 Processor -- start 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 -- start start 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a701017a0 0x7f5a70197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70103350 0x7f5a7019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010d410 con 0x7f5a70102990 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010d290 con 0x7f5a701017a0 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a75c78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010d590 con 0x7f5a70103350 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43790/0 (socket says 172.21.15.38:43790) 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 -- 172.21.15.38:0/1752987778 learned_addr learned my addr 172.21.15.38:0/1752987778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a701017a0 0x7f5a70197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70103350 0x7f5a7019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 -- 172.21.15.38:0/1752987778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70103350 msgr2=0x7f5a7019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70103350 0x7f5a7019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 -- 172.21.15.38:0/1752987778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a701017a0 msgr2=0x7f5a70197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a701017a0 0x7f5a70197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 -- 172.21.15.38:0/1752987778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7019f700 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.480+0000 7f5a6effd640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c970 tx=0x7f5a6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.481+0000 7f5a6cff9640 1 -- 172.21.15.38:0/1752987778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64007bb0 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.481+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7019f9f0 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.481+0000 7f5a6cff9640 1 -- 172.21.15.38:0/1752987778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a64007d50 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.481+0000 7f5a6cff9640 1 -- 172.21.15.38:0/1752987778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400f450 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.481+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7019fef0 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a6cff9640 1 -- 172.21.15.38:0/1752987778 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5a64005020 con 0x7f5a70102990 2024-04-22T00:01:30.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 Processor -- start 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 -- start start 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cb480 0x5560b68cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 0x5560b68d3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560b682bdf0 con 0x5560b68cbe70 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560b67d4640 con 0x5560b68cb480 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a78859740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560b6827db0 con 0x5560b68d0fe0 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43804/0 (socket says 172.21.15.38:43804) 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/1378282405 learned_addr learned my addr 172.21.15.38:0/1378282405 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.483+0000 7f5a6cff9640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 0x7f5a580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6effd640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 0x5560b68d3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cb480 0x5560b68cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 0x7f5a580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/1378282405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 msgr2=0x5560b68d3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 0x5560b68d3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/1378282405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cb480 msgr2=0x5560b68cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cb480 0x5560b68cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/1378282405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b65402d0 con 0x5560b68cbe70 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6effd640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 0x5560b68d3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6cff9640 1 -- 172.21.15.38:0/1752987778 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a640cf180 con 0x7f5a70102990 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009900 tx=0x7f5a60005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b48155b25f89c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a55ffb640 1 -- 172.21.15.38:0/1378282405 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6002cc80 con 0x5560b68cbe70 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a55ffb640 1 -- 172.21.15.38:0/1378282405 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a60005040 con 0x5560b68cbe70 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.484+0000 7f5a55ffb640 1 -- 172.21.15.38:0/1378282405 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60004a40 con 0x5560b68cbe70 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- 172.21.15.38:0/1378282405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 msgr2=0x5560b68d0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009900 tx=0x7f5a60005dd0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 0x7f5a580a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009fd0 tx=0x7f5a68005bd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- 172.21.15.38:0/1378282405 shutdown_connections 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68d0fe0 0x5560b68d3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cbe70 0x5560b68d0970 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 --2- 172.21.15.38:0/1378282405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cb480 0x5560b68cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- 172.21.15.38:0/1378282405 >> 172.21.15.38:0/1378282405 conn(0x5560b68c25e0 msgr2=0x5560b68c29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- 172.21.15.38:0/1378282405 shutdown_connections 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- 172.21.15.38:0/1378282405 wait complete. 2024-04-22T00:01:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 -- start start 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.485+0000 7f5a78859740 1 Processor -- start 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68d5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560b68d3f30 con 0x5560b68d0fe0 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560b67e7be0 con 0x5560b68cbe70 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560b68d40b0 con 0x5560b68cb480 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 10 client.0 _ll_register_callbacks cb 0x5560b67e75d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43820/0 (socket says 172.21.15.38:43820) 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6effd640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 -- 172.21.15.38:0/942647202 learned_addr learned my addr 172.21.15.38:0/942647202 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68d5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 -- 172.21.15.38:0/942647202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 msgr2=0x5560b68d5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68d5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 -- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 msgr2=0x5560b68d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68d5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 -- 172.21.15.38:0/942647202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b660a6b0 con 0x5560b68d0fe0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6effd640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5a680109d0 tx=0x7f5a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5793107f9220ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a4effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800fb50 con 0x5560b68d0fe0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a60005040 con 0x5560b68d0fe0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.486+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a6800fcf0 con 0x5560b68d0fe0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a68027400 con 0x5560b68d0fe0 2024-04-22T00:01:30.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 10 client.15288 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560b68dffd0 con 0x5560b68d0fe0 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a60005040 con 0x5560b68d0fe0 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5a6801f6f0 con 0x5560b68d0fe0 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 10 client.15288 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 10 client.15288 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 20 client.15288 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b68e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 4 client.15288 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560b6839330 con 0x5560b68e2680 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.487+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b68e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.488+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b68e3af0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=354ed09f3d8aa696 server_cookie=2cfc276e8911a2e2 in_seq=0 out_seq=0 2024-04-22T00:01:30.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.488+0000 7f5a4effd640 10 client.15288 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a4effd640 1 -- 172.21.15.38:0/942647202 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560b6839330 con 0x5560b68e2680 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a4effd640 10 client.15288 handle_command_reply: tid=0 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 client.15288 shutdown 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 msgr2=0x5560b68dced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5a680109d0 tx=0x7f5a68004a40 comp rx=0 tx=0).stop 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 shutdown_connections 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b68e3af0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dced0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a4effd640 0 client.15288 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 --2- 172.21.15.38:0/942647202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68d5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 >> 172.21.15.38:0/942647202 conn(0x5560b68c25e0 msgr2=0x5560b68c1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 shutdown_connections 2024-04-22T00:01:30.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 1 -- 172.21.15.38:0/942647202 wait complete. 2024-04-22T00:01:30.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.489+0000 7f5a78859740 20 client.15288 trim_cache size 0 max 16384 2024-04-22T00:01:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.507+0000 7f5a78859740 1 Processor -- start 2024-04-22T00:01:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 -- start start 2024-04-22T00:01:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e1e30 con 0x5560b68d0fe0 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e2500 con 0x5560b68cbe70 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a78859740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e2bd0 con 0x5560b68cb480 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43834/0 (socket says 172.21.15.38:43834) 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 -- 172.21.15.38:0/584643683 learned_addr learned my addr 172.21.15.38:0/584643683 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6effd640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 -- 172.21.15.38:0/584643683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 msgr2=0x5560b68cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 -- 172.21.15.38:0/584643683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 msgr2=0x5560b68cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 -- 172.21.15.38:0/584643683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b68c0c20 con 0x5560b68d0fe0 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5a68006010 tx=0x7f5a6800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e63bd3d12d8d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a557fa640 1 -- 172.21.15.38:0/584643683 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68039070 con 0x5560b68d0fe0 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a557fa640 1 -- 172.21.15.38:0/584643683 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a680120b0 con 0x5560b68d0fe0 2024-04-22T00:01:30.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.508+0000 7f5a557fa640 1 -- 172.21.15.38:0/584643683 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6801f650 con 0x5560b68d0fe0 2024-04-22T00:01:30.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- 172.21.15.38:0/584643683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 msgr2=0x5560b68dce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5a68006010 tx=0x7f5a6800f790 comp rx=0 tx=0).stop 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- 172.21.15.38:0/584643683 shutdown_connections 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68d0fe0 0x5560b68dce90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68cbe70 0x5560b68cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 --2- 172.21.15.38:0/584643683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cb480 0x5560b68cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- 172.21.15.38:0/584643683 >> 172.21.15.38:0/584643683 conn(0x5560b68c25e0 msgr2=0x5560b68c29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- 172.21.15.38:0/584643683 shutdown_connections 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- 172.21.15.38:0/584643683 wait complete. 2024-04-22T00:01:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.509+0000 7f5a78859740 1 -- start start 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 Processor -- start 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cbe70 0x5560b68d9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 0x5560b68dc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e3970 con 0x5560b68cb480 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e32a0 con 0x5560b68d0fe0 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560b68e4040 con 0x5560b68cbe70 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a78859740 10 client.0 _ll_register_callbacks cb 0x7f5a68010580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43848/0 (socket says 172.21.15.38:43848) 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/3295772434 learned_addr learned my addr 172.21.15.38:0/3295772434 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6effd640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cbe70 0x5560b68d9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 0x5560b68dc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/3295772434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cbe70 msgr2=0x5560b68d9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cbe70 0x5560b68d9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 msgr2=0x5560b68dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 0x5560b68dc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 -- 172.21.15.38:0/3295772434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b68dc740 con 0x5560b68cb480 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6ffff640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 0x5560b68dc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a6f7fe640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009850 tx=0x7f5a60005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184b7b14a5cdada5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a4effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60004230 con 0x5560b68cb480 2024-04-22T00:01:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.510+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a600043d0 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a680120b0 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a60005040 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 10 client.15294 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560b6948b70 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a680120b0 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5a6004e070 con 0x5560b68cb480 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 10 client.15294 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 10 client.15294 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 20 client.15294 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b694c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 4 client.15294 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "status"}) -- 0x5560b6833680 con 0x5560b68e2680 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.511+0000 7f5a6effd640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b694c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a4effd640 10 client.15294 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a6effd640 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b694c090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7bd8a85fab05c94 server_cookie=eeee824f9da420b3 in_seq=0 out_seq=0 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a4effd640 1 -- 172.21.15.38:0/3295772434 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5560b6833680 con 0x5560b68e2680 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a4effd640 10 client.15294 handle_command_reply: tid=0 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 client.15294 shutdown 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 msgr2=0x5560b68d90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009850 tx=0x7f5a60005a20 comp rx=0 tx=0).stop 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 shutdown_connections 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5560b68d0fe0 0x5560b68dc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560b68cbe70 0x5560b68d9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5560b68e2680 0x5560b694c090 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 --2- 172.21.15.38:0/3295772434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5560b68cb480 0x5560b68d90e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a4effd640 0 client.15294 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 >> 172.21.15.38:0/3295772434 conn(0x5560b68c25e0 msgr2=0x5560b68cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 shutdown_connections 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 1 -- 172.21.15.38:0/3295772434 wait complete. 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.512+0000 7f5a78859740 20 client.15294 trim_cache size 0 max 16384 2024-04-22T00:01:30.515 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24500, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 113.73781766800001, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 119.715494595 2024-04-22T00:01:30.516 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 msgr2=0x7f5a580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 0x7f5a580a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009fd0 tx=0x7f5a68005bd0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 msgr2=0x7f5a70198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c970 tx=0x7f5a6400ce30 comp rx=0 tx=0).stop 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 shutdown_connections 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5a580a2b90 0x7f5a580a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70103350 0x7f5a7019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a70102990 0x7f5a70198020 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 --2- 172.21.15.38:0/1752987778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a701017a0 0x7f5a70197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 >> 172.21.15.38:0/1752987778 conn(0x7f5a700fcd10 msgr2=0x7f5a70105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 shutdown_connections 2024-04-22T00:01:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:30.513+0000 7f5a75c78640 1 -- 172.21.15.38:0/1752987778 wait complete. 2024-04-22T00:01:30.519 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T00:01:30.520 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T00:01:30.755 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:01:30.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T00:01:30.756 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:30 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1354680792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:30.756 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:30 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/968936045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:30.756 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:30 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1354680792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:30.756 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:30 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/968936045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:30 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1354680792' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:30 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/968936045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:31.005 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:31.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:31 smithi038 ceph-mon[27972]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-22T00:01:31.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:31 smithi111 ceph-mon[31543]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-22T00:01:31.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:31 smithi203 ceph-mon[31431]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-22T00:01:32.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 -- 172.21.15.38:0/2584527169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 msgr2=0x7f340c1027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:32.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 --2- 172.21.15.38:0/2584527169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c1027f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f34000099c0 tx=0x7f3400009cd0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 -- 172.21.15.38:0/2584527169 shutdown_connections 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 --2- 172.21.15.38:0/2584527169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c102d30 0x7f340c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 --2- 172.21.15.38:0/2584527169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c101190 0x7f340c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 --2- 172.21.15.38:0/2584527169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c1027f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.791+0000 7f341194a700 1 -- 172.21.15.38:0/2584527169 >> 172.21.15.38:0/2584527169 conn(0x7f340c0fc730 msgr2=0x7f340c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.792+0000 7f341194a700 1 -- 172.21.15.38:0/2584527169 shutdown_connections 2024-04-22T00:01:32.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.792+0000 7f341194a700 1 -- 172.21.15.38:0/2584527169 wait complete. 2024-04-22T00:01:32.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 Processor -- start 2024-04-22T00:01:32.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 -- start start 2024-04-22T00:01:32.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 0x7f340c06aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:32.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:32.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 0x7f340c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f340c10cd60 con 0x7f340c102390 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f340c10cbe0 con 0x7f340c101190 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f341194a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f340c10cee0 con 0x7f340c102d30 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43876/0 (socket says 172.21.15.38:43876) 2024-04-22T00:01:32.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 -- 172.21.15.38:0/1498962646 learned_addr learned my addr 172.21.15.38:0/1498962646 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340affd700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 0x7f340c06aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340b7fe700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 0x7f340c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 -- 172.21.15.38:0/1498962646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 msgr2=0x7f340c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 0x7f340c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 -- 172.21.15.38:0/1498962646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 msgr2=0x7f340c06aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 0x7f340c06aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340a7fc700 1 -- 172.21.15.38:0/1498962646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340c1a00b0 con 0x7f340c102390 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340b7fe700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 0x7f340c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.793+0000 7f340affd700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 0x7f340c06aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f340a7fc700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f34000098e0 tx=0x7f340000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400009e60 con 0x7f340c102390 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f340000f460 con 0x7f340c102390 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400017650 con 0x7f340c102390 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340c1a0340 con 0x7f340c102390 2024-04-22T00:01:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.794+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f340c1a0790 con 0x7f340c102390 2024-04-22T00:01:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.795+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f340c068f20 con 0x7f340c102390 2024-04-22T00:01:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.796+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f340000f9a0 con 0x7f340c102390 2024-04-22T00:01:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.797+0000 7f3410948700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 0x7f33f40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.797+0000 7f340affd700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 0x7f33f40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.797+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f34000d0420 con 0x7f340c102390 2024-04-22T00:01:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.797+0000 7f340affd700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 0x7f33f40a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f340c1021f0 tx=0x7f33fc00a680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:32.801+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3400013070 con 0x7f340c102390 2024-04-22T00:01:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.026+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f340c062730 con 0x7f33f40a2c20 2024-04-22T00:01:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.027+0000 7f3410948700 1 -- 172.21.15.38:0/1498962646 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f340c062730 con 0x7f33f40a2c20 2024-04-22T00:01:33.020 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2024-04-22T00:01:33.020 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2024-04-22T00:01:33.020 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 msgr2=0x7f33f40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 0x7f33f40a50e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f340c1021f0 tx=0x7f33fc00a680 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 msgr2=0x7f340c19d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f34000098e0 tx=0x7f340000bf90 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 shutdown_connections 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f33f40a2c20 0x7f33f40a50e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f340c102d30 0x7f340c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f340c101190 0x7f340c06aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 --2- 172.21.15.38:0/1498962646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f340c102390 0x7f340c19d150 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:33.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 >> 172.21.15.38:0/1498962646 conn(0x7f340c0fc730 msgr2=0x7f340c105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:33.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 shutdown_connections 2024-04-22T00:01:33.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:33.029+0000 7f341194a700 1 -- 172.21.15.38:0/1498962646 wait complete. 2024-04-22T00:01:33.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:33 smithi111 ceph-mon[31543]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:33.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:33 smithi038 ceph-mon[27972]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:33.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T00:01:33.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:33 smithi203 ceph-mon[31431]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:33.972 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:34.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:34 smithi038 ceph-mon[27972]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:34.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:34 smithi111 ceph-mon[31543]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:34.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:34 smithi203 ceph-mon[31431]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:35.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:35 smithi038 ceph-mon[27972]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-22T00:01:35.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:35 smithi111 ceph-mon[31543]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-22T00:01:35.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 -- 172.21.15.38:0/935025629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0103530 msgr2=0x7f51f0103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:35.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/935025629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0103530 0x7f51f0103990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009990 tx=0x7f51d8009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 -- 172.21.15.38:0/935025629 shutdown_connections 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/935025629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 0x7f51f010a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/935025629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0100550 0x7f51f0100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/935025629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0103530 0x7f51f0103990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.720+0000 7f51f5c2c700 1 -- 172.21.15.38:0/935025629 >> 172.21.15.38:0/935025629 conn(0x7f51f00fbe90 msgr2=0x7f51f00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- 172.21.15.38:0/935025629 shutdown_connections 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- 172.21.15.38:0/935025629 wait complete. 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 Processor -- start 2024-04-22T00:01:35.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- start start 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 0x7f51f019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 0x7f51f01a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010dec0 con 0x7f51f0100550 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010dd40 con 0x7f51f0103530 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51f5c2c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010e040 con 0x7f51f0103ed0 2024-04-22T00:01:35.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.721+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:35.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43564/0 (socket says 172.21.15.38:43564) 2024-04-22T00:01:35.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 -- 172.21.15.38:0/3453303323 learned_addr learned my addr 172.21.15.38:0/3453303323 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:35.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51effff700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 0x7f51f01a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:35.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51e7fff700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 0x7f51f019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:35.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 -- 172.21.15.38:0/3453303323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 msgr2=0x7f51f01a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 0x7f51f01a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 -- 172.21.15.38:0/3453303323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 msgr2=0x7f51f019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 0x7f51f019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 -- 172.21.15.38:0/3453303323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f01a2340 con 0x7f51f0100550 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51e7fff700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 0x7f51f019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:35.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ef7fe700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c9a0 tx=0x7f51e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0004220 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e0007d40 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f01a2630 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0003da0 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.722+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f01a2c40 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.723+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f0068f20 con 0x7f51f0100550 2024-04-22T00:01:35.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.724+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f51e00043c0 con 0x7f51f0100550 2024-04-22T00:01:35.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.725+0000 7f51ed7fa700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 0x7f51dc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:35.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.725+0000 7f51e7fff700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 0x7f51dc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:35.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.726+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51e00cf350 con 0x7f51f0100550 2024-04-22T00:01:35.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.726+0000 7f51e7fff700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 0x7f51dc0a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b5c0 tx=0x7f51d8005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:35.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.729+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f51e0014030 con 0x7f51f0100550 2024-04-22T00:01:35.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:35 smithi203 ceph-mon[31431]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-22T00:01:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.945+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f51f0102a40 con 0x7f51dc0a2ca0 2024-04-22T00:01:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.960+0000 7f51ed7fa700 1 -- 172.21.15.38:0/3453303323 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f51f0102a40 con 0x7f51dc0a2ca0 2024-04-22T00:01:35.953 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T00:01:35.953 INFO:teuthology.orchestra.run.smithi038.stdout:mds.a smithi038 running (2m) 19s ago 2m 17.8M - 19.0.0-2405-g820a2f8c 6702148920ad 0ad9c3a25426 2024-04-22T00:01:35.953 INFO:teuthology.orchestra.run.smithi038.stdout:mds.b smithi111 running (2m) 58s ago 2m 28.5M - 19.0.0-2405-g820a2f8c 6702148920ad bd03537b71aa 2024-04-22T00:01:35.953 INFO:teuthology.orchestra.run.smithi038.stdout:mds.c smithi203 running (2m) 58s ago 2m 16.4M - 19.0.0-2405-g820a2f8c 6702148920ad 03bebce1483c 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.d smithi038 running (2m) 19s ago 2m 16.4M - 19.0.0-2405-g820a2f8c 6702148920ad 1e801e466c4e 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.e smithi111 running (2m) 58s ago 2m 19.6M - 19.0.0-2405-g820a2f8c 6702148920ad e1b5c303d49d 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.f smithi203 running (2m) 58s ago 2m 16.8M - 19.0.0-2405-g820a2f8c 6702148920ad bbb4bd77686c 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.g smithi038 running (2m) 19s ago 2m 16.4M - 19.0.0-2405-g820a2f8c 6702148920ad 7e1a5fbaf296 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.h smithi111 running (2m) 58s ago 2m 17.4M - 19.0.0-2405-g820a2f8c 6702148920ad e40b0aeba82c 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.i smithi203 running (2m) 58s ago 2m 17.7M - 19.0.0-2405-g820a2f8c 6702148920ad 93786033400e 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.j smithi038 running (2m) 19s ago 2m 13.1M - 19.0.0-2405-g820a2f8c 6702148920ad 212f5685aa77 2024-04-22T00:01:35.954 INFO:teuthology.orchestra.run.smithi038.stdout:mds.k smithi111 running (2m) 58s ago 2m 17.4M - 19.0.0-2405-g820a2f8c 6702148920ad 33c4e449aa43 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mds.l smithi203 running (2m) 58s ago 2m 16.1M - 19.0.0-2405-g820a2f8c 6702148920ad ce1211fe9b59 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.x smithi038 *:9283,8765 running (9m) 19s ago 9m 516M - 19.0.0-2405-g820a2f8c 6702148920ad aed10d5afb91 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.y smithi111 *:8443,8765 running (7m) 58s ago 7m 453M - 19.0.0-2405-g820a2f8c 6702148920ad 302af1c9f77c 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.z smithi203 *:8443,8765 running (7m) 58s ago 7m 451M - 19.0.0-2405-g820a2f8c 6702148920ad a76505f5a347 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a smithi038 running (10m) 19s ago 10m 52.3M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 0310a999fd72 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b smithi111 running (8m) 58s ago 8m 42.0M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 7fe567889c33 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c smithi203 running (8m) 58s ago 8m 39.6M 2048M 19.0.0-2405-g820a2f8c 6702148920ad c2c78d773b27 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (7m) 19s ago 7m 56.3M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 5e90c03034cb 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi111 running (7m) 58s ago 7m 45.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 4ce29f80d5d2 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi203 running (6m) 58s ago 6m 57.4M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 39f2a7f5f742 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi038 running (6m) 19s ago 6m 44.4M 4273M 19.0.0-2405-g820a2f8c 6702148920ad e2dae6234715 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi111 running (5m) 58s ago 5m 43.3M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 13c65d2a7080 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi203 running (5m) 58s ago 5m 54.3M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 7cf9b807c309 2024-04-22T00:01:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 smithi038 running (5m) 19s ago 5m 42.6M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 8d0e4445b07a 2024-04-22T00:01:35.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 smithi111 running (4m) 58s ago 4m 44.5M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 05bf97d34aea 2024-04-22T00:01:35.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 smithi203 running (4m) 58s ago 4m 52.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 84de3f49b305 2024-04-22T00:01:35.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 smithi038 running (3m) 19s ago 3m 41.8M 4273M 19.0.0-2405-g820a2f8c 6702148920ad a4c607ca3e09 2024-04-22T00:01:35.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 smithi111 running (3m) 58s ago 3m 44.2M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 8e83acdf498f 2024-04-22T00:01:35.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 smithi203 running (2m) 58s ago 2m 44.1M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 6bebf3c17c4d 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 msgr2=0x7f51dc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 0x7f51dc0a5160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b5c0 tx=0x7f51d8005fb0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 msgr2=0x7f51f019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c9a0 tx=0x7f51e000ccb0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 shutdown_connections 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f51dc0a2ca0 0x7f51dc0a5160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f0103ed0 0x7f51f01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51f0100550 0x7f51f019a6e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 --2- 172.21.15.38:0/3453303323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0103530 0x7f51f019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 >> 172.21.15.38:0/3453303323 conn(0x7f51f00fbe90 msgr2=0x7f51f0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 shutdown_connections 2024-04-22T00:01:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:35.963+0000 7f51f5c2c700 1 -- 172.21.15.38:0/3453303323 wait complete. 2024-04-22T00:01:36.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T00:01:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:37.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:37 smithi111 ceph-mon[31543]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:37.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:37 smithi111 ceph-mon[31543]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:37.581 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:37 smithi203 ceph-mon[31431]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:37.582 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:37 smithi203 ceph-mon[31431]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:37.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:37 smithi038 ceph-mon[27972]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-22T00:01:37.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:37 smithi038 ceph-mon[27972]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 -- 172.21.15.38:0/1529691288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 msgr2=0x7fdf481022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/1529691288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481022b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009f90 tx=0x7fdf4000b330 comp rx=0 tx=0).stop 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 -- 172.21.15.38:0/1529691288 shutdown_connections 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/1529691288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481022b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/1529691288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf48100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/1529691288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf481018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 -- 172.21.15.38:0/1529691288 >> 172.21.15.38:0/1529691288 conn(0x7fdf480fc750 msgr2=0x7fdf480feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 -- 172.21.15.38:0/1529691288 shutdown_connections 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.511+0000 7fdf4f42d700 1 -- 172.21.15.38:0/1529691288 wait complete. 2024-04-22T00:01:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 Processor -- start 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 -- start start 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf4819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48110060 con 0x7fdf48101470 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4810fee0 con 0x7fdf48101e30 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4f42d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481101e0 con 0x7fdf48100b10 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43568/0 (socket says 172.21.15.38:43568) 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 -- 172.21.15.38:0/2087616411 learned_addr learned my addr 172.21.15.38:0/2087616411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4d1c9700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf4819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4d9ca700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 -- 172.21.15.38:0/2087616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 msgr2=0x7fdf4819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf4819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 -- 172.21.15.38:0/2087616411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 msgr2=0x7fdf481a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4c9c8700 1 -- 172.21.15.38:0/2087616411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf481a2280 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4d1c9700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf4819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.512+0000 7fdf4d9ca700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf4c9c8700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d9b0 tx=0x7fdf3800dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf38016070 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf3800de90 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf481a2570 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3800f620 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.513+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481a2ab0 con 0x7fdf48101470 2024-04-22T00:01:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.514+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf48068f20 con 0x7fdf48101470 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.515+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdf38019070 con 0x7fdf48101470 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.516+0000 7fdf3e7fc700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 0x7fdf340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.516+0000 7fdf4d1c9700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 0x7fdf340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.516+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdf380cefe0 con 0x7fdf48101470 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.516+0000 7fdf4d1c9700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 0x7fdf340a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009cb0 tx=0x7fdf44009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.519+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdf3809bba0 con 0x7fdf48101470 2024-04-22T00:01:38.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.734+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fdf48062730 con 0x7fdf340a2c20 2024-04-22T00:01:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.740+0000 7fdf3e7fc700 1 -- 172.21.15.38:0/2087616411 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fdf48062730 con 0x7fdf340a2c20 2024-04-22T00:01:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T00:01:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:mds.all 12/12 61s ago 2m smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi111=b;smithi111=e;smithi111=h;smithi111=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-22T00:01:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 3/3 61s ago 8m smithi038=x;smithi111=y;smithi203=z;count:3 2024-04-22T00:01:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 61s ago 8m smithi038:172.21.15.38=a;smithi111:172.21.15.111=b;smithi203:172.21.15.203=c;count:3 2024-04-22T00:01:38.733 INFO:teuthology.orchestra.run.smithi038.stdout:osd 12 61s ago - 2024-04-22T00:01:38.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 msgr2=0x7fdf340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:38.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 0x7fdf340a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009cb0 tx=0x7fdf44009260 comp rx=0 tx=0).stop 2024-04-22T00:01:38.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 msgr2=0x7fdf4819ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:38.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d9b0 tx=0x7fdf3800dd70 comp rx=0 tx=0).stop 2024-04-22T00:01:38.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 shutdown_connections 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fdf340a2c20 0x7fdf340a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf48101e30 0x7fdf481a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf48100b10 0x7fdf4819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 --2- 172.21.15.38:0/2087616411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf48101470 0x7fdf4819ab60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 >> 172.21.15.38:0/2087616411 conn(0x7fdf480fc750 msgr2=0x7fdf480fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 shutdown_connections 2024-04-22T00:01:38.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:38.742+0000 7fdf4f42d700 1 -- 172.21.15.38:0/2087616411 wait complete. 2024-04-22T00:01:39.436 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:39 smithi038 ceph-mon[27972]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T00:01:39.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T00:01:39.719 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:39.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:39 smithi203 ceph-mon[31431]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T00:01:39.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:39 smithi111 ceph-mon[31543]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T00:01:40.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:40 smithi038 ceph-mon[27972]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:40.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:40 smithi203 ceph-mon[31431]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:40.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:40 smithi111 ceph-mon[31543]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:41.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.431+0000 7f6adfff1700 1 -- 172.21.15.38:0/2820602551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 msgr2=0x7f6ad8105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:41.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.431+0000 7f6adfff1700 1 --2- 172.21.15.38:0/2820602551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad8105170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009990 tx=0x7f6ad4009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 -- 172.21.15.38:0/2820602551 shutdown_connections 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 --2- 172.21.15.38:0/2820602551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 0x7f6ad810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 --2- 172.21.15.38:0/2820602551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad8105170 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 --2- 172.21.15.38:0/2820602551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 0x7f6ad8107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 -- 172.21.15.38:0/2820602551 >> 172.21.15.38:0/2820602551 conn(0x7f6ad80fc730 msgr2=0x7f6ad80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 -- 172.21.15.38:0/2820602551 shutdown_connections 2024-04-22T00:01:41.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 -- 172.21.15.38:0/2820602551 wait complete. 2024-04-22T00:01:41.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:41 smithi038 ceph-mon[27972]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:01:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 Processor -- start 2024-04-22T00:01:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.432+0000 7f6adfff1700 1 -- start start 2024-04-22T00:01:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 0x7f6ad819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 0x7f6ad81a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad810df60 con 0x7f6ad8102d80 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad810dde0 con 0x7f6ad81056b0 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6adfff1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad810e0e0 con 0x7f6ad8107fe0 2024-04-22T00:01:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43578/0 (socket says 172.21.15.38:43578) 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 -- 172.21.15.38:0/1619230874 learned_addr learned my addr 172.21.15.38:0/1619230874 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6add58c700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 0x7f6ad819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6ade58e700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 0x7f6ad81a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 -- 172.21.15.38:0/1619230874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 msgr2=0x7f6ad81a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 0x7f6ad81a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 -- 172.21.15.38:0/1619230874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 msgr2=0x7f6ad819ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 0x7f6ad819ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 -- 172.21.15.38:0/1619230874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad81a2490 con 0x7f6ad8102d80 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.433+0000 7f6addd8d700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40047b0 tx=0x7f6ad4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.434+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4027070 con 0x7f6ad8102d80 2024-04-22T00:01:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.434+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad81a2720 con 0x7f6ad8102d80 2024-04-22T00:01:41.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.434+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad81a2c50 con 0x7f6ad8102d80 2024-04-22T00:01:41.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.434+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ad4022470 con 0x7f6ad8102d80 2024-04-22T00:01:41.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.434+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400f6c0 con 0x7f6ad8102d80 2024-04-22T00:01:41.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.435+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6abc005330 con 0x7f6ad8102d80 2024-04-22T00:01:41.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.436+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6ad402a070 con 0x7f6ad8102d80 2024-04-22T00:01:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.436+0000 7f6aceffd700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 0x7f6ac40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.437+0000 7f6add58c700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 0x7f6ac40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.437+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6ad40d0280 con 0x7f6ad8102d80 2024-04-22T00:01:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.437+0000 7f6add58c700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 0x7f6ac40a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009a10 tx=0x7f6ac8009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:41.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.441+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6ad405e6f0 con 0x7f6ad8102d80 2024-04-22T00:01:41.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.655+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6abc000bc0 con 0x7f6ac40a2ca0 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.656+0000 7f6aceffd700 1 -- 172.21.15.38:0/1619230874 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6abc000bc0 con 0x7f6ac40a2ca0 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:smithi111 172.21.15.111 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:smithi203 172.21.15.203 2024-04-22T00:01:41.649 INFO:teuthology.orchestra.run.smithi038.stdout:3 hosts in cluster 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 msgr2=0x7f6ac40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 0x7f6ac40a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009a10 tx=0x7f6ac8009190 comp rx=0 tx=0).stop 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 msgr2=0x7f6ad819a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40047b0 tx=0x7f6ad4009f50 comp rx=0 tx=0).stop 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 shutdown_connections 2024-04-22T00:01:41.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f6ac40a2ca0 0x7f6ac40a5160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ad8107fe0 0x7f6ad81a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ad8102d80 0x7f6ad819a830 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 --2- 172.21.15.38:0/1619230874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ad81056b0 0x7f6ad819ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.658+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 >> 172.21.15.38:0/1619230874 conn(0x7f6ad80fc730 msgr2=0x7f6ad81024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.659+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 shutdown_connections 2024-04-22T00:01:41.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:41.659+0000 7f6adfff1700 1 -- 172.21.15.38:0/1619230874 wait complete. 2024-04-22T00:01:41.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:41 smithi203 ceph-mon[31431]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:01:41.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:41 smithi111 ceph-mon[31543]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:01:42.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T00:01:42.619 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:43.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:43 smithi111 ceph-mon[31543]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:43.629 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:43 smithi111 ceph-mon[31543]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T00:01:43.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:43 smithi038 ceph-mon[27972]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:43.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:43 smithi038 ceph-mon[27972]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T00:01:43.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:43 smithi203 ceph-mon[31431]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:43.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:43 smithi203 ceph-mon[31431]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T00:01:44.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 -- 172.21.15.38:0/3250956838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781023d0 msgr2=0x7fb578102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:44.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/3250956838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781023d0 0x7fb578102830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb5680099c0 tx=0x7fb568009cd0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 -- 172.21.15.38:0/3250956838 shutdown_connections 2024-04-22T00:01:44.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/3250956838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 0x7fb578109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/3250956838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781011d0 0x7fb5781015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/3250956838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781023d0 0x7fb578102830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 -- 172.21.15.38:0/3250956838 >> 172.21.15.38:0/3250956838 conn(0x7fb5780fc790 msgr2=0x7fb5780febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 -- 172.21.15.38:0/3250956838 shutdown_connections 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.349+0000 7fb57fbe9700 1 -- 172.21.15.38:0/3250956838 wait complete. 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 Processor -- start 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 -- start start 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 0x7fb5781989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 0x7fb57819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb57810cdb0 con 0x7fb5781011d0 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb57810cc30 con 0x7fb578102d70 2024-04-22T00:01:44.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57fbe9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb57810cf30 con 0x7fb5781023d0 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57092/0 (socket says 172.21.15.38:57092) 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 -- 172.21.15.38:0/1471147874 learned_addr learned my addr 172.21.15.38:0/1471147874 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57e186700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 0x7fb57819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d184700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 0x7fb5781989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 -- 172.21.15.38:0/1471147874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 msgr2=0x7fb5781989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 0x7fb5781989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 -- 172.21.15.38:0/1471147874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 msgr2=0x7fb57819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 0x7fb57819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 -- 172.21.15.38:0/1471147874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5781a00e0 con 0x7fb5781011d0 2024-04-22T00:01:44.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d184700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 0x7fb5781989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.350+0000 7fb57d985700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb57400d9b0 tx=0x7fb57400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.351+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574017070 con 0x7fb5781011d0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.351+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb574009e10 con 0x7fb5781011d0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.351+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5781a03d0 con 0x7fb5781011d0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.351+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57400f620 con 0x7fb5781011d0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.351+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5781a0910 con 0x7fb5781011d0 2024-04-22T00:01:44.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.352+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb578068f20 con 0x7fb5781011d0 2024-04-22T00:01:44.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.353+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb574009930 con 0x7fb5781011d0 2024-04-22T00:01:44.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.353+0000 7fb56effd700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 0x7fb5640a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:44.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.354+0000 7fb57d184700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 0x7fb5640a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:44.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.354+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb5740cef30 con 0x7fb5781011d0 2024-04-22T00:01:44.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.354+0000 7fb57d184700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 0x7fb5640a5160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb568006010 tx=0x7fb568009510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:44.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.358+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb57409bad0 con 0x7fb5781011d0 2024-04-22T00:01:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.574+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb5780fe8d0 con 0x7fb5640a2ca0 2024-04-22T00:01:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.576+0000 7fb56effd700 1 -- 172.21.15.38:0/1471147874 <== mgr.14148 v2:172.21.15.38:6800/2340980806 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb5780fe8d0 con 0x7fb5640a2ca0 2024-04-22T00:01:44.568 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T00:01:44.568 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-22T00:01:44.568 INFO:teuthology.orchestra.run.smithi038.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T00:01:44.569 INFO:teuthology.orchestra.run.smithi038.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 msgr2=0x7fb5640a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 0x7fb5640a5160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb568006010 tx=0x7fb568009510 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 msgr2=0x7fb578198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb57400d9b0 tx=0x7fb57400dcc0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 shutdown_connections 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb5640a2ca0 0x7fb5640a5160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb578102d70 0x7fb57819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5781011d0 0x7fb578198480 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 --2- 172.21.15.38:0/1471147874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5781023d0 0x7fb5781989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:44.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 >> 172.21.15.38:0/1471147874 conn(0x7fb5780fc790 msgr2=0x7fb5780fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:44.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 shutdown_connections 2024-04-22T00:01:44.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:44.578+0000 7fb57fbe9700 1 -- 172.21.15.38:0/1471147874 wait complete. 2024-04-22T00:01:45.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T00:01:45.483 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:01:45.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T00:01:45.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:45 smithi038 ceph-mon[27972]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:45.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:45 smithi203 ceph-mon[31431]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:45.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:45 smithi111 ceph-mon[31543]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:46 smithi038 ceph-mon[27972]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:46.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:46 smithi203 ceph-mon[31431]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:46.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:46 smithi111 ceph-mon[31543]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:01:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 -- 172.21.15.38:0/1728757713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 msgr2=0x7fd518105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 --2- 172.21.15.38:0/1728757713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518105150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009990 tx=0x7fd50c009ca0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 -- 172.21.15.38:0/1728757713 shutdown_connections 2024-04-22T00:01:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 --2- 172.21.15.38:0/1728757713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd51810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 --2- 172.21.15.38:0/1728757713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518105150 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 --2- 172.21.15.38:0/1728757713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 -- 172.21.15.38:0/1728757713 >> 172.21.15.38:0/1728757713 conn(0x7fd5180fc790 msgr2=0x7fd5180febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:47.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 -- 172.21.15.38:0/1728757713 shutdown_connections 2024-04-22T00:01:47.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.454+0000 7fd51f03c700 1 -- 172.21.15.38:0/1728757713 wait complete. 2024-04-22T00:01:47.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:47 smithi038 ceph-mon[27972]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 Processor -- start 2024-04-22T00:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 -- start start 2024-04-22T00:01:47.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd518101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd51810df70 con 0x7fd518102d60 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd51810ddf0 con 0x7fd518107fc0 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51f03c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd51810e0f0 con 0x7fd518105690 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd51cdd8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd517fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd517fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42642/0 (socket says 172.21.15.38:42642) 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.455+0000 7fd517fff700 1 -- 172.21.15.38:0/3090822815 learned_addr learned my addr 172.21.15.38:0/3090822815 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd51d5d9700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd518101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 -- 172.21.15.38:0/3090822815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 msgr2=0x7fd518101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd518101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 -- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 msgr2=0x7fd518102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 -- 172.21.15.38:0/3090822815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd518101b20 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd51cdd8700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd51d5d9700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd518101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd517fff700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c9b0 tx=0x7fd50400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.456+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504007b00 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.457+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5181a2650 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.457+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd504007ca0 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.457+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504018730 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.457+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5181a2c30 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.458+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd518068f20 con 0x7fd518105690 2024-04-22T00:01:47.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.459+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd504043ae0 con 0x7fd518105690 2024-04-22T00:01:47.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.460+0000 7fd515ffb700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 0x7fd5080a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:47.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.460+0000 7fd51cdd8700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 0x7fd5080a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:47.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.460+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5040cf5c0 con 0x7fd518105690 2024-04-22T00:01:47.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.460+0000 7fd51cdd8700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 0x7fd5080a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00b5c0 tx=0x7fd50c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:47.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.464+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd504014030 con 0x7fd518105690 2024-04-22T00:01:47.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.716+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd51802db30 con 0x7fd518105690 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.717+0000 7fd515ffb700 1 -- 172.21.15.38:0/3090822815 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3094 (secure 0 0 0) 0x7fd50409c160 con 0x7fd518105690 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:e27 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2024-04-22T00:01:47.710 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 27 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-21T23:59:36.029248+0000 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T00:00:05.266438+0000 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 300 2024-04-22T00:01:47.711 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:max_xattr_size 65536 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 0 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 5 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:in 0,1,2,3,4 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:up {0=24500,1=24487,2=24460,3=24455,4=24448} 2024-04-22T00:01:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3,4] 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data disabled 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:bal_rank_mask -1 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 0 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:qdb_cluster leader: 24500 members: 24487,24500,24460,24455,24448 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b{0:24500} state up:active seq 3 addr [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.e{0:24425} state up:standby-replay seq 1 addr [v2:172.21.15.111:6834/3450283861,v1:172.21.15.111:6835/3450283861] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.713 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.i{1:24487} state up:active seq 11 addr [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.l{1:24409} state up:standby-replay seq 1 addr [v2:172.21.15.203:6832/1279747281,v1:172.21.15.203:6833/1279747281] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c{2:24460} state up:active seq 13 addr [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.k{2:24395} state up:standby-replay seq 1 addr [v2:172.21.15.111:6832/618180230,v1:172.21.15.111:6833/618180230] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.h{3:24455} state up:active seq 12 addr [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a{3:14634} state up:standby-replay seq 1 addr [v2:172.21.15.38:6840/3401657425,v1:172.21.15.38:6841/3401657425] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.f{4:24448} state up:active seq 14 addr [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.g{4:14622} state up:standby-replay seq 1 addr [v2:172.21.15.38:6838/3894275312,v1:172.21.15.38:6839/3894275312] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:Standby daemons: 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.j{-1:14532} state up:standby seq 1 addr [v2:172.21.15.38:6834/425693237,v1:172.21.15.38:6835/425693237] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.714 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.38:6836/815557617,v1:172.21.15.38:6837/815557617] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T00:01:47.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 msgr2=0x7fd5080a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:47.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 0x7fd5080a50e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00b5c0 tx=0x7fd50c005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 msgr2=0x7fd518100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c9b0 tx=0x7fd50400ccc0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 shutdown_connections 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd5080a2c20 0x7fd5080a50e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd518107fc0 0x7fd518101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102d60 0x7fd518102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 --2- 172.21.15.38:0/3090822815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd518105690 0x7fd518100d10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 >> 172.21.15.38:0/3090822815 conn(0x7fd5180fc790 msgr2=0x7fd5181036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 shutdown_connections 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:47.719+0000 7fd51f03c700 1 -- 172.21.15.38:0/3090822815 wait complete. 2024-04-22T00:01:47.716 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:47.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:47 smithi203 ceph-mon[31431]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:47.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:47 smithi111 ceph-mon[31543]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:48.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T00:01:48.618 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:48.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:48 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3090822815' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T00:01:48.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:48 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3090822815' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T00:01:48.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:48 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3090822815' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T00:01:49.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:49 smithi038 ceph-mon[27972]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:49.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:49 smithi203 ceph-mon[31431]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:49.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:49 smithi111 ceph-mon[31543]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:01:50.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 -- 172.21.15.38:0/865401992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8101190 msgr2=0x7f3de81015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:50.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 --2- 172.21.15.38:0/865401992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8101190 0x7f3de81015b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0099c0 tx=0x7f3ddc009cd0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 -- 172.21.15.38:0/865401992 shutdown_connections 2024-04-22T00:01:50.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 --2- 172.21.15.38:0/865401992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de81095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 --2- 172.21.15.38:0/865401992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8101190 0x7f3de81015b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 --2- 172.21.15.38:0/865401992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8102390 0x7f3de81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 -- 172.21.15.38:0/865401992 >> 172.21.15.38:0/865401992 conn(0x7f3de80fc730 msgr2=0x7f3de80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 -- 172.21.15.38:0/865401992 shutdown_connections 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.321+0000 7f3deee33700 1 -- 172.21.15.38:0/865401992 wait complete. 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 Processor -- start 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 -- start start 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8101190 0x7f3de81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810cdd0 con 0x7f3de8102390 2024-04-22T00:01:50.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810cc50 con 0x7f3de8101190 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.322+0000 7f3deee33700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3de810cf50 con 0x7f3de8102d30 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57144/0 (socket says 172.21.15.38:57144) 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 -- 172.21.15.38:0/3086643419 learned_addr learned my addr 172.21.15.38:0/3086643419 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3decbcf700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8101190 0x7f3de81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3ded3d0700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 -- 172.21.15.38:0/3086643419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 msgr2=0x7f3de819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 -- 172.21.15.38:0/3086643419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8101190 msgr2=0x7f3de81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8101190 0x7f3de81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 -- 172.21.15.38:0/3086643419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de81a0140 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3ded3d0700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c9e0 tx=0x7f3dd400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4007ac0 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de81a0430 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd4007c60 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.323+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4018730 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.324+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de81a0a40 con 0x7f3de8102390 2024-04-22T00:01:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.325+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de8068f20 con 0x7f3de8102390 2024-04-22T00:01:50.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.326+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3dd40188d0 con 0x7f3de8102390 2024-04-22T00:01:50.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.326+0000 7f3de5ffb700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 0x7f3dd80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:50.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.327+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3dd40cf600 con 0x7f3de8102390 2024-04-22T00:01:50.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.327+0000 7f3decbcf700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 0x7f3dd80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:50.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.327+0000 7f3decbcf700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 0x7f3dd80a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:50.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.330+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3dd4014030 con 0x7f3de8102390 2024-04-22T00:01:50.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.539+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f3de81a0d00 con 0x7f3de8102390 2024-04-22T00:01:50.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.540+0000 7f3de5ffb700 1 -- 172.21.15.38:0/3086643419 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5593 (secure 0 0 0) 0x7f3dd409c290 con 0x7f3de8102390 2024-04-22T00:01:50.532 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 91 2024-04-22T00:01:50.532 INFO:teuthology.orchestra.run.smithi038.stdout:fsid eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:01:50.532 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-04-21T23:51:31.159714+0000 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-04-22T00:00:27.347647+0000 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 27 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client luminous 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-04-22T00:01:50.533 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 12 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6802/1038840841,v1:172.21.15.38:6803/1038840841] [v2:172.21.15.38:6804/1038840841,v1:172.21.15.38:6805/1038840841] exists,up 41c96fae-afbf-4345-b838-bfdab30012d1 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/4079345941,v1:172.21.15.111:6801/4079345941] [v2:172.21.15.111:6802/4079345941,v1:172.21.15.111:6803/4079345941] exists,up 87fe4b77-db92-4f21-b74f-76c1aec76f38 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2981966976,v1:172.21.15.203:6801/2981966976] [v2:172.21.15.203:6802/2981966976,v1:172.21.15.203:6803/2981966976] exists,up d2e0ed53-ee4a-432b-9147-1e4df2da6dd5 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6810/1951374002,v1:172.21.15.38:6811/1951374002] [v2:172.21.15.38:6812/1951374002,v1:172.21.15.38:6813/1951374002] exists,up 28f56df4-c518-4ecc-bbfa-51b25a0104d5 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/753751366,v1:172.21.15.111:6809/753751366] [v2:172.21.15.111:6810/753751366,v1:172.21.15.111:6811/753751366] exists,up cab4f9b6-2a83-4faf-87a1-1f5b6baaae91 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/1090281315,v1:172.21.15.203:6809/1090281315] [v2:172.21.15.203:6810/1090281315,v1:172.21.15.203:6811/1090281315] exists,up 89520131-b2cc-4385-a99f-e77fbde2670b 2024-04-22T00:01:50.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6818/1747049129,v1:172.21.15.38:6819/1747049129] [v2:172.21.15.38:6820/1747049129,v1:172.21.15.38:6821/1747049129] exists,up fe082d0b-2acc-446e-ba14-845d68b911e2 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/961343269,v1:172.21.15.111:6817/961343269] [v2:172.21.15.111:6818/961343269,v1:172.21.15.111:6819/961343269] exists,up 8bff1d33-4cb9-45ba-8cf4-efc6051da49a 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/434241518,v1:172.21.15.203:6817/434241518] [v2:172.21.15.203:6818/434241518,v1:172.21.15.203:6819/434241518] exists,up 6d85f1f1-9fed-495f-92b8-cd9f0fad42ce 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6826/654111402,v1:172.21.15.38:6827/654111402] [v2:172.21.15.38:6828/654111402,v1:172.21.15.38:6829/654111402] exists,up 75dc496d-7396-4c25-94c0-4976235724a8 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/4030828796,v1:172.21.15.111:6825/4030828796] [v2:172.21.15.111:6826/4030828796,v1:172.21.15.111:6827/4030828796] exists,up 9465f481-94aa-472a-993a-61d6a1946d0d 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/3896776627,v1:172.21.15.203:6825/3896776627] [v2:172.21.15.203:6826/3896776627,v1:172.21.15.203:6827/3896776627] exists,up 66e605ac-1577-4fcf-861c-cc6278c07699 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.4 [2,11] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.5 [8,11] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.1 [4,9] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.13 [4,9] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.23 [4,8] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.33 [5,9] 2024-04-22T00:01:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.5 [5,8] 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.c [9,10] 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.f [9,8] 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.1b [9,8] 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/4082720653 expires 2024-04-22T23:52:25.561345+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/2824973701 expires 2024-04-22T23:52:25.561345+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1866190732 expires 2024-04-22T23:51:57.369607+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/2824973701 expires 2024-04-22T23:52:25.561345+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1458203717 expires 2024-04-22T23:51:57.369607+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/71250327 expires 2024-04-22T23:51:57.369607+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/71250327 expires 2024-04-22T23:51:57.369607+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/247200712 expires 2024-04-22T23:52:25.561345+0000 2024-04-22T00:01:50.536 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/4200150941 expires 2024-04-22T23:51:57.369607+0000 2024-04-22T00:01:50.537 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2769068060 expires 2024-04-22T23:52:25.561345+0000 2024-04-22T00:01:50.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 msgr2=0x7f3dd80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:50.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 0x7f3dd80a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc005fb0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 msgr2=0x7f3de8198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:50.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400c9e0 tx=0x7f3dd400ccf0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 shutdown_connections 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f3dd80a2c20 0x7f3dd80a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de8102d30 0x7f3de819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3de8101190 0x7f3de81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 --2- 172.21.15.38:0/3086643419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8102390 0x7f3de8198a20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 >> 172.21.15.38:0/3086643419 conn(0x7f3de80fc730 msgr2=0x7f3de80fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 shutdown_connections 2024-04-22T00:01:50.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:50.542+0000 7f3deee33700 1 -- 172.21.15.38:0/3086643419 wait complete. 2024-04-22T00:01:51.189 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T00:01:51.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 -- 172.21.15.38:0/229665359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 msgr2=0x7f7c8c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 --2- 172.21.15.38:0/229665359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c106f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c80009900 tx=0x7f7c8002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 -- 172.21.15.38:0/229665359 shutdown_connections 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 --2- 172.21.15.38:0/229665359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 --2- 172.21.15.38:0/229665359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c106f20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 --2- 172.21.15.38:0/229665359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 0x7f7c8c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 -- 172.21.15.38:0/229665359 >> 172.21.15.38:0/229665359 conn(0x7f7c8c0fbbf0 msgr2=0x7f7c8c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 -- 172.21.15.38:0/229665359 shutdown_connections 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.319+0000 7f7c91d85640 1 -- 172.21.15.38:0/229665359 wait complete. 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 Processor -- start 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 -- start start 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 0x7f7c8c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10d410 con 0x7f7c8c104b30 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10d290 con 0x7f7c8c107460 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c91d85640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c10d590 con 0x7f7c8c102200 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c8affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c8affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57168/0 (socket says 172.21.15.38:57168) 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c8affd640 1 -- 172.21.15.38:0/3255653300 learned_addr learned my addr 172.21.15.38:0/3255653300 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c8b7fe640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 0x7f7c8c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.320+0000 7f7c8bfff640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 -- 172.21.15.38:0/3255653300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 msgr2=0x7f7c8c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 0x7f7c8c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 -- 172.21.15.38:0/3255653300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 msgr2=0x7f7c8c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 -- 172.21.15.38:0/3255653300 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c19f700 con 0x7f7c8c104b30 2024-04-22T00:01:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8bfff640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c8affd640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c80005ab0 tx=0x7f7c80005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c80041070 con 0x7f7c8c104b30 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c800054e0 con 0x7f7c8c104b30 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c19f990 con 0x7f7c8c104b30 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c80034440 con 0x7f7c8c104b30 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.321+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c19fe80 con 0x7f7c8c104b30 2024-04-22T00:01:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.322+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c8c0fd6e0 con 0x7f7c8c104b30 2024-04-22T00:01:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.323+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7c80044070 con 0x7f7c8c104b30 2024-04-22T00:01:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.324+0000 7f7c88ff9640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 0x7f7c740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.324+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c800f0fb0 con 0x7f7c8c104b30 2024-04-22T00:01:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.324+0000 7f7c8b7fe640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 0x7f7c740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.324+0000 7f7c8b7fe640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 0x7f7c740a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c005fd0 tx=0x7f7c7c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.328+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7c800bdbb0 con 0x7f7c8c104b30 2024-04-22T00:01:51.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.516+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c8c02d390 con 0x7f7c8c104b30 2024-04-22T00:01:51.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.517+0000 7f7c88ff9640 1 -- 172.21.15.38:0/3255653300 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7f7c800bd4b0 con 0x7f7c8c104b30 2024-04-22T00:01:51.510 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:51.510 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 msgr2=0x7f7c740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 0x7f7c740a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c005fd0 tx=0x7f7c7c00b390 comp rx=0 tx=0).stop 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 msgr2=0x7f7c8c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c80005ab0 tx=0x7f7c80005dc0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 shutdown_connections 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f7c740a2b90 0x7f7c740a5050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c8c107460 0x7f7c8c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c8c104b30 0x7f7c8c198020 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 --2- 172.21.15.38:0/3255653300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c8c102200 0x7f7c8c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 >> 172.21.15.38:0/3255653300 conn(0x7f7c8c0fbbf0 msgr2=0x7f7c8c102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 shutdown_connections 2024-04-22T00:01:51.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.519+0000 7f7c91d85640 1 -- 172.21.15.38:0/3255653300 wait complete. 2024-04-22T00:01:51.522 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:51 smithi038 ceph-mon[27972]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:51.522 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:51 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3086643419' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T00:01:51.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:01:51.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:01:51.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.636+0000 7f747f5d7640 1 -- 172.21.15.38:0/978025044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7478101890 msgr2=0x7f7478101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.636+0000 7f747f5d7640 1 --2- 172.21.15.38:0/978025044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7478101890 0x7f7478101d10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7468009900 tx=0x7f746802bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- 172.21.15.38:0/978025044 shutdown_connections 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- 172.21.15.38:0/978025044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478102250 0x7f7478108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- 172.21.15.38:0/978025044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7478101890 0x7f7478101d10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- 172.21.15.38:0/978025044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74781006a0 0x7f7478100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- 172.21.15.38:0/978025044 >> 172.21.15.38:0/978025044 conn(0x7f74780fbc10 msgr2=0x7f74780fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- 172.21.15.38:0/978025044 shutdown_connections 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- 172.21.15.38:0/978025044 wait complete. 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 Processor -- start 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- start start 2024-04-22T00:01:51.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478101890 0x7f7478197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7478102250 0x7f747819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f747810d440 con 0x7f74781006a0 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f747810d2c0 con 0x7f7478101890 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747f5d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f747810d5c0 con 0x7f7478102250 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747d373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747d373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57188/0 (socket says 172.21.15.38:57188) 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.637+0000 7f747d373640 1 -- 172.21.15.38:0/3716921029 learned_addr learned my addr 172.21.15.38:0/3716921029 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747cb72640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478101890 0x7f7478197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747db74640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7478102250 0x7f747819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 -- 172.21.15.38:0/3716921029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7478102250 msgr2=0x7f747819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7478102250 0x7f747819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 -- 172.21.15.38:0/3716921029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478101890 msgr2=0x7f7478197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478101890 0x7f7478197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 -- 172.21.15.38:0/3716921029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747819f6c0 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747d373640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f747000d950 tx=0x7f747000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470017020 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7470009c30 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747819f9b0 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747000f740 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.638+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747819fed0 con 0x7f74781006a0 2024-04-22T00:01:51.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.639+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74781079c0 con 0x7f74781006a0 2024-04-22T00:01:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.640+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f747001a020 con 0x7f74781006a0 2024-04-22T00:01:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.641+0000 7f746e7fc640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 0x7f74640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.641+0000 7f747cb72640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 0x7f74640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.641+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74700cedc0 con 0x7f74781006a0 2024-04-22T00:01:51.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.642+0000 7f747cb72640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 0x7f74640a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f746802c7b0 tx=0x7f7468005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.646+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7470021070 con 0x7f74781006a0 2024-04-22T00:01:51.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:51 smithi203 ceph-mon[31431]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:51.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:51 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3086643419' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T00:01:51.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.832+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7478062340 con 0x7f74781006a0 2024-04-22T00:01:51.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.833+0000 7f746e7fc640 1 -- 172.21.15.38:0/3716921029 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f7470009de0 con 0x7f74781006a0 2024-04-22T00:01:51.825 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:01:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:51 smithi111 ceph-mon[31543]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:01:51.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:51 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3086643419' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 msgr2=0x7f74640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 0x7f74640a5050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f746802c7b0 tx=0x7f7468005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 msgr2=0x7f7478197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f747000d950 tx=0x7f747000de10 comp rx=0 tx=0).stop 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 shutdown_connections 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f74640a2b90 0x7f74640a5050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7478102250 0x7f747819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7478101890 0x7f7478197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 --2- 172.21.15.38:0/3716921029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74781006a0 0x7f7478197aa0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.834+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 >> 172.21.15.38:0/3716921029 conn(0x7f74780fbc10 msgr2=0x7f7478104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.835+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 shutdown_connections 2024-04-22T00:01:51.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.835+0000 7f747f5d7640 1 -- 172.21.15.38:0/3716921029 wait complete. 2024-04-22T00:01:51.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 -- 172.21.15.38:0/746639547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 msgr2=0x7f4804109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 --2- 172.21.15.38:0/746639547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804109be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009900 tx=0x7f47f402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 -- 172.21.15.38:0/746639547 shutdown_connections 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 --2- 172.21.15.38:0/746639547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804109be0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 --2- 172.21.15.38:0/746639547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 --2- 172.21.15.38:0/746639547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 0x7f4804101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 -- 172.21.15.38:0/746639547 >> 172.21.15.38:0/746639547 conn(0x7f48040fcd10 msgr2=0x7f48040ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 -- 172.21.15.38:0/746639547 shutdown_connections 2024-04-22T00:01:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.950+0000 7f480bed4640 1 -- 172.21.15.38:0/746639547 wait complete. 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 Processor -- start 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 -- start start 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 0x7f48040754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f480410d4d0 con 0x7f4804103350 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f480410d350 con 0x7f48041017a0 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480bed4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f480410d650 con 0x7f4804102990 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57210/0 (socket says 172.21.15.38:57210) 2024-04-22T00:01:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 -- 172.21.15.38:0/2020474434 learned_addr learned my addr 172.21.15.38:0/2020474434 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f4809c70640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 0x7f48040754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480946f640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 -- 172.21.15.38:0/2020474434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 msgr2=0x7f4804073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 -- 172.21.15.38:0/2020474434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 msgr2=0x7f48040754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 0x7f48040754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 -- 172.21.15.38:0/2020474434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4804074900 con 0x7f4804103350 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480946f640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.951+0000 7f480a471640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005ab0 tx=0x7f47f4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.952+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4043020 con 0x7f4804103350 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.952+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f40054e0 con 0x7f4804103350 2024-04-22T00:01:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.952+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48041a1bd0 con 0x7f4804103350 2024-04-22T00:01:51.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.952+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003e20 con 0x7f4804103350 2024-04-22T00:01:51.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.952+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48041a2120 con 0x7f4804103350 2024-04-22T00:01:51.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.953+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4804108a60 con 0x7f4804103350 2024-04-22T00:01:51.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.954+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f47f4046020 con 0x7f4804103350 2024-04-22T00:01:51.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.954+0000 7f4802ffd640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 0x7f47ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:51.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.955+0000 7f4809c70640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 0x7f47ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:51.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.955+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f47f40f16f0 con 0x7f4804103350 2024-04-22T00:01:51.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.955+0000 7f4809c70640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 0x7f47ec0a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f48041027f0 tx=0x7f47fc009870 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:51.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:51.959+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f47f404d020 con 0x7f4804103350 2024-04-22T00:01:52.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.143+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4804062340 con 0x7f4804103350 2024-04-22T00:01:52.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.144+0000 7f4802ffd640 1 -- 172.21.15.38:0/2020474434 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f47f40be2f0 con 0x7f4804103350 2024-04-22T00:01:52.137 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:52.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 msgr2=0x7f47ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 0x7f47ec0a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f48041027f0 tx=0x7f47fc009870 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 msgr2=0x7f4804074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005ab0 tx=0x7f47f4005dc0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 shutdown_connections 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f47ec0a2b90 0x7f47ec0a5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4804103350 0x7f4804074070 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4804102990 0x7f4804073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 --2- 172.21.15.38:0/2020474434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48041017a0 0x7f48040754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 >> 172.21.15.38:0/2020474434 conn(0x7f48040fcd10 msgr2=0x7f4804105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 shutdown_connections 2024-04-22T00:01:52.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.146+0000 7f480bed4640 1 -- 172.21.15.38:0/2020474434 wait complete. 2024-04-22T00:01:52.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 -- 172.21.15.38:0/451099926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a8102990 msgr2=0x7f67a8102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 --2- 172.21.15.38:0/451099926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a8102990 0x7f67a8102e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6790009900 tx=0x7f679002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 -- 172.21.15.38:0/451099926 shutdown_connections 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 --2- 172.21.15.38:0/451099926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8103350 0x7f67a8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 --2- 172.21.15.38:0/451099926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a8102990 0x7f67a8102e10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 --2- 172.21.15.38:0/451099926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a81017a0 0x7f67a8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 -- 172.21.15.38:0/451099926 >> 172.21.15.38:0/451099926 conn(0x7f67a80fcd10 msgr2=0x7f67a80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 -- 172.21.15.38:0/451099926 shutdown_connections 2024-04-22T00:01:52.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.262+0000 7f67acbb8640 1 -- 172.21.15.38:0/451099926 wait complete. 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 Processor -- start 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 -- start start 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 0x7f67a8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a8103350 0x7f67a819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f67a810d420 con 0x7f67a81017a0 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f67a810d2a0 con 0x7f67a8103350 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67acbb8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67a810d5a0 con 0x7f67a8102990 2024-04-22T00:01:52.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57236/0 (socket says 172.21.15.38:57236) 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 -- 172.21.15.38:0/2371931033 learned_addr learned my addr 172.21.15.38:0/2371931033 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f679ffff640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 0x7f67a8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a6d9d640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a8103350 0x7f67a819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 -- 172.21.15.38:0/2371931033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 msgr2=0x7f67a8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 0x7f67a8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 -- 172.21.15.38:0/2371931033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a8103350 msgr2=0x7f67a819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a8103350 0x7f67a819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.263+0000 7f67a659c640 1 -- 172.21.15.38:0/2371931033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a819f6c0 con 0x7f67a81017a0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f679ffff640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 0x7f67a8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f67a659c640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f679800c970 tx=0x7f679800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798004220 con 0x7f67a81017a0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a819f9b0 con 0x7f67a81017a0 2024-04-22T00:01:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67980043c0 con 0x7f67a81017a0 2024-04-22T00:01:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679800f450 con 0x7f67a81017a0 2024-04-22T00:01:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.264+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a819ff20 con 0x7f67a81017a0 2024-04-22T00:01:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.265+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a8108ae0 con 0x7f67a81017a0 2024-04-22T00:01:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.266+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f67980079b0 con 0x7f67a81017a0 2024-04-22T00:01:52.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.267+0000 7f679f7fe640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 0x7f67940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.267+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f67980cf640 con 0x7f67a81017a0 2024-04-22T00:01:52.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.267+0000 7f679ffff640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 0x7f67940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.267+0000 7f679ffff640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 0x7f67940a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6790006010 tx=0x7f679002c8a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.270+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6798014030 con 0x7f67a81017a0 2024-04-22T00:01:52.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.454+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67a802d170 con 0x7f67a81017a0 2024-04-22T00:01:52.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.455+0000 7f679f7fe640 1 -- 172.21.15.38:0/2371931033 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f679809c1e0 con 0x7f67a81017a0 2024-04-22T00:01:52.449 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:01:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 msgr2=0x7f67940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 0x7f67940a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6790006010 tx=0x7f679002c8a0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 msgr2=0x7f67a8197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f679800c970 tx=0x7f679800ce30 comp rx=0 tx=0).stop 2024-04-22T00:01:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 shutdown_connections 2024-04-22T00:01:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f67940a2b90 0x7f67940a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a8103350 0x7f67a819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a8102990 0x7f67a8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 --2- 172.21.15.38:0/2371931033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f67a81017a0 0x7f67a8197aa0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 >> 172.21.15.38:0/2371931033 conn(0x7f67a80fcd10 msgr2=0x7f67a8105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 shutdown_connections 2024-04-22T00:01:52.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.457+0000 7f67acbb8640 1 -- 172.21.15.38:0/2371931033 wait complete. 2024-04-22T00:01:52.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3255653300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:52.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3716921029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2020474434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- 172.21.15.38:0/2569845420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18104b30 msgr2=0x7efe18106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 --2- 172.21.15.38:0/2569845420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18104b30 0x7efe18106f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009900 tx=0x7efe0c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- 172.21.15.38:0/2569845420 shutdown_connections 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 --2- 172.21.15.38:0/2569845420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 0x7efe181098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 --2- 172.21.15.38:0/2569845420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18104b30 0x7efe18106f20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 --2- 172.21.15.38:0/2569845420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18102200 0x7efe181045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- 172.21.15.38:0/2569845420 >> 172.21.15.38:0/2569845420 conn(0x7efe180fbbf0 msgr2=0x7efe180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- 172.21.15.38:0/2569845420 shutdown_connections 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- 172.21.15.38:0/2569845420 wait complete. 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 Processor -- start 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.574+0000 7efe1d688640 1 -- start start 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18104b30 0x7efe1819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 0x7efe181a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810d410 con 0x7efe18102200 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810d290 con 0x7efe18107460 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810d590 con 0x7efe18104b30 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57246/0 (socket says 172.21.15.38:57246) 2024-04-22T00:01:52.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 -- 172.21.15.38:0/1984664125 learned_addr learned my addr 172.21.15.38:0/1984664125 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe167fc640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18104b30 0x7efe1819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe177fe640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 0x7efe181a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 -- 172.21.15.38:0/1984664125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18104b30 msgr2=0x7efe1819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18104b30 0x7efe1819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 -- 172.21.15.38:0/1984664125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 msgr2=0x7efe181a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 0x7efe181a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 -- 172.21.15.38:0/1984664125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe181a1980 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe16ffd640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efe0800e9f0 tx=0x7efe0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0800cd80 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe181a1c70 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe08005500 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08010690 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.575+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe181a2190 con 0x7efe18102200 2024-04-22T00:01:52.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.576+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe18108fb0 con 0x7efe18102200 2024-04-22T00:01:52.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.578+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efe080046c0 con 0x7efe18102200 2024-04-22T00:01:52.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.578+0000 7efdfffff640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 0x7efe000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.578+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efe080cf990 con 0x7efe18102200 2024-04-22T00:01:52.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.578+0000 7efe167fc640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 0x7efe000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.579+0000 7efe167fc640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 0x7efe000a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe0c000c00 tx=0x7efe0c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.582+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efe080536f0 con 0x7efe18102200 2024-04-22T00:01:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.766+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe1802d170 con 0x7efe18102200 2024-04-22T00:01:52.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.766+0000 7efdfffff640 1 -- 172.21.15.38:0/1984664125 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7efe0809c590 con 0x7efe18102200 2024-04-22T00:01:52.759 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 msgr2=0x7efe000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 0x7efe000a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe0c000c00 tx=0x7efe0c005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 msgr2=0x7efe18199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efe0800e9f0 tx=0x7efe0800edf0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 shutdown_connections 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7efe000a2c10 0x7efe000a50d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe18107460 0x7efe181a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe18104b30 0x7efe1819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 --2- 172.21.15.38:0/1984664125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efe18102200 0x7efe18199d60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 >> 172.21.15.38:0/1984664125 conn(0x7efe180fbbf0 msgr2=0x7efe181019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 shutdown_connections 2024-04-22T00:01:52.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.768+0000 7efe1d688640 1 -- 172.21.15.38:0/1984664125 wait complete. 2024-04-22T00:01:52.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:01:52.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3255653300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:52.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3716921029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2020474434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3255653300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:52.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3716921029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2020474434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.881+0000 7f81ec0fb640 1 -- 172.21.15.38:0/3510425360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 msgr2=0x7f81e4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.881+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/3510425360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4101cf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009900 tx=0x7f81d802bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- 172.21.15.38:0/3510425360 shutdown_connections 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/3510425360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4102230 0x7f81e4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/3510425360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4101cf0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/3510425360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4100680 0x7f81e4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- 172.21.15.38:0/3510425360 >> 172.21.15.38:0/3510425360 conn(0x7f81e40fbbf0 msgr2=0x7f81e40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- 172.21.15.38:0/3510425360 shutdown_connections 2024-04-22T00:01:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- 172.21.15.38:0/3510425360 wait complete. 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 Processor -- start 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- start start 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4100680 0x7f81e4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 0x7f81e419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f81e410d4c0 con 0x7f81e4101870 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f81e410d340 con 0x7f81e4100680 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.882+0000 7f81ec0fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81e410d640 con 0x7f81e4102230 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57272/0 (socket says 172.21.15.38:57272) 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 -- 172.21.15.38:0/2447214232 learned_addr learned my addr 172.21.15.38:0/2447214232 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9e97640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4100680 0x7f81e4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81ea698640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 0x7f81e419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 -- 172.21.15.38:0/2447214232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 msgr2=0x7f81e419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 0x7f81e419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 -- 172.21.15.38:0/2447214232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4100680 msgr2=0x7f81e4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4100680 0x7f81e4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 -- 172.21.15.38:0/2447214232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e419f6f0 con 0x7f81e4101870 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81ea698640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 0x7f81e419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e9696640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8043020 con 0x7f81e4101870 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d8004440 con 0x7f81e4101870 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e419f980 con 0x7f81e4101870 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d803e450 con 0x7f81e4101870 2024-04-22T00:01:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.883+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e419ff20 con 0x7f81e4101870 2024-04-22T00:01:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.884+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e4107940 con 0x7f81e4101870 2024-04-22T00:01:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.885+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f81d8046020 con 0x7f81e4101870 2024-04-22T00:01:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.886+0000 7f81e2ffd640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 0x7f81cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.886+0000 7f81e9e97640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 0x7f81cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.886+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f81d80f1a80 con 0x7f81e4101870 2024-04-22T00:01:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.886+0000 7f81e9e97640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 0x7f81cc0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81e41016d0 tx=0x7f81dc009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:52.890+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f81d804d020 con 0x7f81e4101870 2024-04-22T00:01:53.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.131+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81e402d170 con 0x7f81e4101870 2024-04-22T00:01:53.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.132+0000 7f81e2ffd640 1 -- 172.21.15.38:0/2447214232 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f81d80be680 con 0x7f81e4101870 2024-04-22T00:01:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:53.125 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:01:53.125 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 msgr2=0x7f81cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 0x7f81cc0a50d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81e41016d0 tx=0x7f81dc009110 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 msgr2=0x7f81e4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8004040 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 shutdown_connections 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f81cc0a2c10 0x7f81cc0a50d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81e4102230 0x7f81e419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81e4101870 0x7f81e4198010 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 --2- 172.21.15.38:0/2447214232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81e4100680 0x7f81e4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 >> 172.21.15.38:0/2447214232 conn(0x7f81e40fbbf0 msgr2=0x7f81e4104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 shutdown_connections 2024-04-22T00:01:53.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.134+0000 7f81ec0fb640 1 -- 172.21.15.38:0/2447214232 wait complete. 2024-04-22T00:01:53.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:53.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 -- 172.21.15.38:0/3193842463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541017b0 msgr2=0x7fb354101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 --2- 172.21.15.38:0/3193842463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541017b0 0x7fb354101bd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb344009900 tx=0x7fb34402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 -- 172.21.15.38:0/3193842463 shutdown_connections 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 --2- 172.21.15.38:0/3193842463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb354103360 0x7fb354109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 --2- 172.21.15.38:0/3193842463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3541029a0 0x7fb354102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 --2- 172.21.15.38:0/3193842463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541017b0 0x7fb354101bd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 -- 172.21.15.38:0/3193842463 >> 172.21.15.38:0/3193842463 conn(0x7fb3540fcd40 msgr2=0x7fb3540ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 -- 172.21.15.38:0/3193842463 shutdown_connections 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.248+0000 7fb3596fb640 1 -- 172.21.15.38:0/3193842463 wait complete. 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 Processor -- start 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 -- start start 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3541017b0 0x7fb354197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb354103360 0x7fb35419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb35410d400 con 0x7fb3541029a0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb35410d280 con 0x7fb3541017b0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3596fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb35410d580 con 0x7fb354103360 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57282/0 (socket says 172.21.15.38:57282) 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 -- 172.21.15.38:0/2637173546 learned_addr learned my addr 172.21.15.38:0/2637173546 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb352ffd640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3541017b0 0x7fb354197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3537fe640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb354103360 0x7fb35419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 -- 172.21.15.38:0/2637173546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb354103360 msgr2=0x7fb35419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb354103360 0x7fb35419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 -- 172.21.15.38:0/2637173546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3541017b0 msgr2=0x7fb354197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3541017b0 0x7fb354197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.249+0000 7fb3527fc640 1 -- 172.21.15.38:0/2637173546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35419f630 con 0x7fb3541029a0 2024-04-22T00:01:53.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb3527fc640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c920 tx=0x7fb34800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348007c70 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb348007e10 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35419f920 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348018890 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.250+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35419fe70 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.251+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb354108a70 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.252+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb348018a70 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.253+0000 7fb33bfff640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 0x7fb33c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.253+0000 7fb352ffd640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 0x7fb33c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.253+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb3480cf550 con 0x7fb3541029a0 2024-04-22T00:01:53.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.253+0000 7fb352ffd640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 0x7fb33c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb344000c00 tx=0x7fb344005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.257+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb348014030 con 0x7fb3541029a0 2024-04-22T00:01:53.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:53 smithi038 ceph-mon[27972]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:53.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:53 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2371931033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:53 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1984664125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:53 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2447214232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:53.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.447+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb354062340 con 0x7fb3541029a0 2024-04-22T00:01:53.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.449+0000 7fb33bfff640 1 -- 172.21.15.38:0/2637173546 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7fb34809c0f0 con 0x7fb3541029a0 2024-04-22T00:01:53.441 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:53.442 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:53.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 msgr2=0x7fb33c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 0x7fb33c0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb344000c00 tx=0x7fb344005fd0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 msgr2=0x7fb354197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c920 tx=0x7fb34800cde0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 shutdown_connections 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fb33c0a2b90 0x7fb33c0a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb354103360 0x7fb35419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb3541029a0 0x7fb354197f50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 --2- 172.21.15.38:0/2637173546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3541017b0 0x7fb354197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 >> 172.21.15.38:0/2637173546 conn(0x7fb3540fcd40 msgr2=0x7fb354105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 shutdown_connections 2024-04-22T00:01:53.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.451+0000 7fb3596fb640 1 -- 172.21.15.38:0/2637173546 wait complete. 2024-04-22T00:01:53.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:01:53.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:01:53.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.572+0000 7f0df99e5640 1 -- 172.21.15.38:0/1466381514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df41006a0 msgr2=0x7f0df4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.572+0000 7f0df99e5640 1 --2- 172.21.15.38:0/1466381514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df41006a0 0x7f0df4100ac0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0de4009900 tx=0x7f0de402bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:53.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.572+0000 7f0df99e5640 1 -- 172.21.15.38:0/1466381514 shutdown_connections 2024-04-22T00:01:53.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.572+0000 7f0df99e5640 1 --2- 172.21.15.38:0/1466381514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df4102250 0x7f0df4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.572+0000 7f0df99e5640 1 --2- 172.21.15.38:0/1466381514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4101890 0x7f0df4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 --2- 172.21.15.38:0/1466381514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df41006a0 0x7f0df4100ac0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- 172.21.15.38:0/1466381514 >> 172.21.15.38:0/1466381514 conn(0x7f0df40fbc30 msgr2=0x7f0df40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- 172.21.15.38:0/1466381514 shutdown_connections 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- 172.21.15.38:0/1466381514 wait complete. 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 Processor -- start 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- start start 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df41006a0 0x7f0df4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4102250 0x7f0df419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0df410d4a0 con 0x7f0df4101890 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.573+0000 7f0df99e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0df410d320 con 0x7f0df41006a0 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df99e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0df410d620 con 0x7f0df4102250 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57306/0 (socket says 172.21.15.38:57306) 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 -- 172.21.15.38:0/2222972185 learned_addr learned my addr 172.21.15.38:0/2222972185 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df37fe640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4102250 0x7f0df419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df2ffd640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df41006a0 0x7f0df4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 -- 172.21.15.38:0/2222972185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4102250 msgr2=0x7f0df419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4102250 0x7f0df419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 -- 172.21.15.38:0/2222972185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df41006a0 msgr2=0x7f0df4197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df41006a0 0x7f0df4197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 -- 172.21.15.38:0/2222972185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df419f730 con 0x7f0df4101890 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df27fc640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0de800d770 tx=0x7f0de800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8017070 con 0x7f0df4101890 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0de8005500 con 0x7f0df4101890 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df419fa20 con 0x7f0df4101890 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8004e00 con 0x7f0df4101890 2024-04-22T00:01:53.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.574+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df419ff20 con 0x7f0df4101890 2024-04-22T00:01:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.575+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df4107a40 con 0x7f0df4101890 2024-04-22T00:01:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.576+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0de8004fa0 con 0x7f0df4101890 2024-04-22T00:01:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.577+0000 7f0df89e3640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 0x7f0ddc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.577+0000 7f0df2ffd640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 0x7f0ddc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.577+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0de80cee20 con 0x7f0df4101890 2024-04-22T00:01:53.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.578+0000 7f0df2ffd640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 0x7f0ddc0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0de402c170 tx=0x7f0de402df90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.582+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0de809ba20 con 0x7f0df4101890 2024-04-22T00:01:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.769+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0df4062340 con 0x7f0df4101890 2024-04-22T00:01:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.769+0000 7f0df89e3640 1 -- 172.21.15.38:0/2222972185 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f0de809b320 con 0x7f0df4101890 2024-04-22T00:01:53.762 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:01:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 msgr2=0x7f0ddc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 0x7f0ddc0a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0de402c170 tx=0x7f0de402df90 comp rx=0 tx=0).stop 2024-04-22T00:01:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 msgr2=0x7f0df4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0de800d770 tx=0x7f0de800dc30 comp rx=0 tx=0).stop 2024-04-22T00:01:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 shutdown_connections 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f0ddc0a2b90 0x7f0ddc0a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df4102250 0x7f0df419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0df4101890 0x7f0df4198050 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 --2- 172.21.15.38:0/2222972185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0df41006a0 0x7f0df4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 >> 172.21.15.38:0/2222972185 conn(0x7f0df40fbc30 msgr2=0x7f0df4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 shutdown_connections 2024-04-22T00:01:53.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.771+0000 7f0df99e5640 1 -- 172.21.15.38:0/2222972185 wait complete. 2024-04-22T00:01:53.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:01:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:53 smithi203 ceph-mon[31431]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:53 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2371931033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:53 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1984664125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:53 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2447214232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:53 smithi111 ceph-mon[31543]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:01:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:53 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2371931033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:53 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1984664125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:53.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:53 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2447214232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 -- 172.21.15.38:0/2028105565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 msgr2=0x7f4c90100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2028105565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90100ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009ca0 tx=0x7f4c8002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 -- 172.21.15.38:0/2028105565 shutdown_connections 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2028105565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c90108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2028105565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2028105565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90100ac0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 -- 172.21.15.38:0/2028105565 >> 172.21.15.38:0/2028105565 conn(0x7f4c900fbc30 msgr2=0x7f4c900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:53.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 -- 172.21.15.38:0/2028105565 shutdown_connections 2024-04-22T00:01:53.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.887+0000 7f4c94db1640 1 -- 172.21.15.38:0/2028105565 wait complete. 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 Processor -- start 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 -- start start 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c9019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010d440 con 0x7f4c901006a0 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010d2c0 con 0x7f4c90102250 2024-04-22T00:01:53.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c94db1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9010d5c0 con 0x7f4c90101890 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52608/0 (socket says 172.21.15.38:52608) 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 -- 172.21.15.38:0/2858970067 learned_addr learned my addr 172.21.15.38:0/2858970067 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8dd9b640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 -- 172.21.15.38:0/2858970067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 msgr2=0x7f4c90198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 -- 172.21.15.38:0/2858970067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 msgr2=0x7f4c9019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8ed9d640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c9019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c9019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8e59c640 1 -- 172.21.15.38:0/2858970067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9019f6e0 con 0x7f4c901006a0 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8dd9b640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.888+0000 7f4c8ed9d640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c9019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:53.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c8e59c640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4c80000f80 tx=0x7f4c80032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80041020 con 0x7f4c901006a0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c80034cd0 con 0x7f4c901006a0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9019f970 con 0x7f4c901006a0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80039040 con 0x7f4c901006a0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.889+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9019fe30 con 0x7f4c901006a0 2024-04-22T00:01:53.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.890+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c901079c0 con 0x7f4c901006a0 2024-04-22T00:01:53.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.891+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4c80044020 con 0x7f4c901006a0 2024-04-22T00:01:53.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.892+0000 7f4c7f7fe640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 0x7f4c780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:53.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.892+0000 7f4c8dd9b640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 0x7f4c780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:53.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.892+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c800f1370 con 0x7f4c901006a0 2024-04-22T00:01:53.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.892+0000 7f4c8dd9b640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 0x7f4c780a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c90198fd0 tx=0x7f4c84005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:53.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:53.896+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c8004b020 con 0x7f4c901006a0 2024-04-22T00:01:54.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.082+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c9002d3f0 con 0x7f4c901006a0 2024-04-22T00:01:54.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.083+0000 7f4c7f7fe640 1 -- 172.21.15.38:0/2858970067 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f4c800bdf10 con 0x7f4c901006a0 2024-04-22T00:01:54.076 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:54.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 msgr2=0x7f4c780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 0x7f4c780a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c90198fd0 tx=0x7f4c84005f90 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 msgr2=0x7f4c90197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4c80000f80 tx=0x7f4c80032ec0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 shutdown_connections 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f4c780a2b90 0x7f4c780a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c90102250 0x7f4c9019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c90101890 0x7f4c90198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 --2- 172.21.15.38:0/2858970067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c901006a0 0x7f4c90197ac0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 >> 172.21.15.38:0/2858970067 conn(0x7f4c900fbc30 msgr2=0x7f4c90104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 shutdown_connections 2024-04-22T00:01:54.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.085+0000 7f4c94db1640 1 -- 172.21.15.38:0/2858970067 wait complete. 2024-04-22T00:01:54.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- 172.21.15.38:0/3232286713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 msgr2=0x7fae58108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 --2- 172.21.15.38:0/3232286713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae58108ac0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae48009ca0 tx=0x7fae4802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- 172.21.15.38:0/3232286713 shutdown_connections 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 --2- 172.21.15.38:0/3232286713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae58108ac0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 --2- 172.21.15.38:0/3232286713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 0x7fae58101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 --2- 172.21.15.38:0/3232286713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 0x7fae58100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- 172.21.15.38:0/3232286713 >> 172.21.15.38:0/3232286713 conn(0x7fae580fbbf0 msgr2=0x7fae580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- 172.21.15.38:0/3232286713 shutdown_connections 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- 172.21.15.38:0/3232286713 wait complete. 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 Processor -- start 2024-04-22T00:01:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.201+0000 7fae5cea7640 1 -- start start 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 0x7fae58069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 0x7fae5819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810c300 con 0x7fae58102230 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810c180 con 0x7fae58100680 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810c480 con 0x7fae58101870 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52632/0 (socket says 172.21.15.38:52632) 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 -- 172.21.15.38:0/1866184911 learned_addr learned my addr 172.21.15.38:0/1866184911 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5659c640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 0x7fae58069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae55d9b640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 0x7fae5819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 -- 172.21.15.38:0/1866184911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 msgr2=0x7fae5819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 0x7fae5819c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 -- 172.21.15.38:0/1866184911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 msgr2=0x7fae58069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 0x7fae58069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 -- 172.21.15.38:0/1866184911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5819f700 con 0x7fae58102230 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae56d9d640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fae48037040 tx=0x7fae48032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48041020 con 0x7fae58102230 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5819f930 con 0x7fae58102230 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae48034d90 con 0x7fae58102230 2024-04-22T00:01:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.202+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48039040 con 0x7fae58102230 2024-04-22T00:01:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.203+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5819fe70 con 0x7fae58102230 2024-04-22T00:01:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.204+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae58107a70 con 0x7fae58102230 2024-04-22T00:01:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.205+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fae48044020 con 0x7fae58102230 2024-04-22T00:01:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.205+0000 7fae477fe640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 0x7fae400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.205+0000 7fae5659c640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 0x7fae400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.206+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fae480f1400 con 0x7fae58102230 2024-04-22T00:01:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.206+0000 7fae5659c640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 0x7fae400a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fae50005d00 tx=0x7fae50005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.210+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fae4804b020 con 0x7fae58102230 2024-04-22T00:01:54.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.395+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae58062340 con 0x7fae58102230 2024-04-22T00:01:54.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.395+0000 7fae477fe640 1 -- 172.21.15.38:0/1866184911 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fae480be000 con 0x7fae58102230 2024-04-22T00:01:54.388 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 msgr2=0x7fae400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 0x7fae400a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fae50005d00 tx=0x7fae50005c50 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 msgr2=0x7fae5819edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fae48037040 tx=0x7fae48032f20 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 shutdown_connections 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fae400a2b90 0x7fae400a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae58102230 0x7fae5819edf0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae58101870 0x7fae5819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 --2- 172.21.15.38:0/1866184911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae58100680 0x7fae58069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 >> 172.21.15.38:0/1866184911 conn(0x7fae580fbbf0 msgr2=0x7fae58104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 shutdown_connections 2024-04-22T00:01:54.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.397+0000 7fae5cea7640 1 -- 172.21.15.38:0/1866184911 wait complete. 2024-04-22T00:01:54.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:01:54.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 -- 172.21.15.38:0/448344932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 msgr2=0x7fcb10108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 --2- 172.21.15.38:0/448344932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb10108ac0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcb08009ca0 tx=0x7fcb0802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 -- 172.21.15.38:0/448344932 shutdown_connections 2024-04-22T00:01:54.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 --2- 172.21.15.38:0/448344932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb10108ac0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 --2- 172.21.15.38:0/448344932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10101870 0x7fcb10101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 --2- 172.21.15.38:0/448344932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10100680 0x7fcb10100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.515+0000 7fcb1654b640 1 -- 172.21.15.38:0/448344932 >> 172.21.15.38:0/448344932 conn(0x7fcb100fbbf0 msgr2=0x7fcb100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- 172.21.15.38:0/448344932 shutdown_connections 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- 172.21.15.38:0/448344932 wait complete. 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 Processor -- start 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- start start 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10100680 0x7fcb101969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10101870 0x7fcb10196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1010c3a0 con 0x7fcb10102230 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1010c220 con 0x7fcb10101870 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.516+0000 7fcb1654b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1010c520 con 0x7fcb10100680 2024-04-22T00:01:54.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52648/0 (socket says 172.21.15.38:52648) 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 -- 172.21.15.38:0/2191171321 learned_addr learned my addr 172.21.15.38:0/2191171321 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb0ffff640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10100680 0x7fcb101969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb0f7fe640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10101870 0x7fcb10196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 -- 172.21.15.38:0/2191171321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10100680 msgr2=0x7fcb101969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10100680 0x7fcb101969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 -- 172.21.15.38:0/2191171321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10101870 msgr2=0x7fcb10196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10101870 0x7fcb10196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 -- 172.21.15.38:0/2191171321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1019e610 con 0x7fcb10102230 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb14ae8640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcb08005e30 tx=0x7fcb08032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08034740 con 0x7fcb10102230 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb100688d0 con 0x7fcb10102230 2024-04-22T00:01:54.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb08032690 con 0x7fcb10102230 2024-04-22T00:01:54.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb080042a0 con 0x7fcb10102230 2024-04-22T00:01:54.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.517+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb10068dc0 con 0x7fcb10102230 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.518+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb10068560 con 0x7fcb10102230 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.519+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcb08032840 con 0x7fcb10102230 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.520+0000 7fcb0d7fa640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 0x7fcaf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.520+0000 7fcb0ffff640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 0x7fcaf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.520+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcb080f30d0 con 0x7fcb10102230 2024-04-22T00:01:54.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.521+0000 7fcb0ffff640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 0x7fcaf80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcb101016d0 tx=0x7fcb00009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.525+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcb08003e20 con 0x7fcb10102230 2024-04-22T00:01:54.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.707+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcb10107990 con 0x7fcb10102230 2024-04-22T00:01:54.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.708+0000 7fcb0d7fa640 1 -- 172.21.15.38:0/2191171321 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fcb080df0a0 con 0x7fcb10102230 2024-04-22T00:01:54.701 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:01:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 msgr2=0x7fcaf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 0x7fcaf80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcb101016d0 tx=0x7fcb00009110 comp rx=0 tx=0).stop 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 msgr2=0x7fcb1019dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcb08005e30 tx=0x7fcb08032f30 comp rx=0 tx=0).stop 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 shutdown_connections 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fcaf80a2b90 0x7fcaf80a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb10102230 0x7fcb1019dfa0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb10101870 0x7fcb10196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 --2- 172.21.15.38:0/2191171321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb10100680 0x7fcb101969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 >> 172.21.15.38:0/2191171321 conn(0x7fcb100fbbf0 msgr2=0x7fcb10104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 shutdown_connections 2024-04-22T00:01:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.710+0000 7fcb1654b640 1 -- 172.21.15.38:0/2191171321 wait complete. 2024-04-22T00:01:54.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:54 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2637173546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:54.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:54 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2222972185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:54 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2858970067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:54 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1866184911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:01:54.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:54 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2637173546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:54.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:54 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2222972185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:54 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2858970067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:54 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1866184911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- 172.21.15.38:0/3497916629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49901017b0 msgr2=0x7f4990101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- 172.21.15.38:0/3497916629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49901017b0 0x7f4990101bd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4988009ca0 tx=0x7f498802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- 172.21.15.38:0/3497916629 shutdown_connections 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- 172.21.15.38:0/3497916629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4990103360 0x7f4990109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- 172.21.15.38:0/3497916629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901029a0 0x7f4990102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- 172.21.15.38:0/3497916629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49901017b0 0x7f4990101bd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- 172.21.15.38:0/3497916629 >> 172.21.15.38:0/3497916629 conn(0x7f49900fcd40 msgr2=0x7f49900ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- 172.21.15.38:0/3497916629 shutdown_connections 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- 172.21.15.38:0/3497916629 wait complete. 2024-04-22T00:01:54.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 Processor -- start 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- start start 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901017b0 0x7f4990199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 0x7f499019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4990073e40 con 0x7f4990103360 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.824+0000 7f4997240640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4990073cc0 con 0x7f49901017b0 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f4997240640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4990073fc0 con 0x7f49901029a0 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52662/0 (socket says 172.21.15.38:52662) 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 -- 172.21.15.38:0/2957566776 learned_addr learned my addr 172.21.15.38:0/2957566776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f4994fdc640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901017b0 0x7f4990199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f498ffff640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 0x7f499019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 -- 172.21.15.38:0/2957566776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 msgr2=0x7f499019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 0x7f499019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 -- 172.21.15.38:0/2957566776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901017b0 msgr2=0x7f4990199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901017b0 0x7f4990199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 -- 172.21.15.38:0/2957566776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49901a18c0 con 0x7f4990103360 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f498ffff640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 0x7f499019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f49957dd640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f498000e970 tx=0x7f498000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498000ccc0 con 0x7f4990103360 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f498000ce60 con 0x7f4990103360 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49901a1bb0 con 0x7f4990103360 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.825+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4980010490 con 0x7f4990103360 2024-04-22T00:01:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.826+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49901a2100 con 0x7f4990103360 2024-04-22T00:01:54.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.827+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49901072e0 con 0x7f4990103360 2024-04-22T00:01:54.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.828+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4980005020 con 0x7f4990103360 2024-04-22T00:01:54.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.828+0000 7f498dffb640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 0x7f497c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:54.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.829+0000 7f4994fdc640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 0x7f497c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:54.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.829+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f49800cf870 con 0x7f4990103360 2024-04-22T00:01:54.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.829+0000 7f4994fdc640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 0x7f497c0a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f499019ab50 tx=0x7f4988032590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:54.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:54.833+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f498009c470 con 0x7f4990103360 2024-04-22T00:01:54.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:54 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2637173546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:54.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:54 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2222972185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:54 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2858970067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:01:54.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:54 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1866184911' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:01:55.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.055+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f499002d170 con 0x7f4990103360 2024-04-22T00:01:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.056+0000 7f498dffb640 1 -- 172.21.15.38:0/2957566776 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11935 (secure 0 0 0) 0x7f498009bd70 con 0x7f4990103360 2024-04-22T00:01:55.048 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:55.048 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:00:05.266438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:01:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 27 2024-04-22T00:01:55.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 msgr2=0x7f497c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 0x7f497c0a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f499019ab50 tx=0x7f4988032590 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 msgr2=0x7f49901a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f498000e970 tx=0x7f498000ed70 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 shutdown_connections 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f497c0a2b90 0x7f497c0a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4990103360 0x7f49901a1220 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49901029a0 0x7f499019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 --2- 172.21.15.38:0/2957566776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49901017b0 0x7f4990199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 >> 172.21.15.38:0/2957566776 conn(0x7f49900fcd40 msgr2=0x7f49900fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 shutdown_connections 2024-04-22T00:01:55.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.058+0000 7f4997240640 1 -- 172.21.15.38:0/2957566776 wait complete. 2024-04-22T00:01:55.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:01:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 -- 172.21.15.38:0/3149998697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c109660 msgr2=0x7fd43c10baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 --2- 172.21.15.38:0/3149998697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c109660 0x7fd43c10baf0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd434009ca0 tx=0x7fd43402c180 comp rx=0 tx=0).stop 2024-04-22T00:01:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 -- 172.21.15.38:0/3149998697 shutdown_connections 2024-04-22T00:01:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 --2- 172.21.15.38:0/3149998697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c109660 0x7fd43c10baf0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 --2- 172.21.15.38:0/3149998697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c074120 0x7fd43c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 --2- 172.21.15.38:0/3149998697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 0x7fd43c073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 -- 172.21.15.38:0/3149998697 >> 172.21.15.38:0/3149998697 conn(0x7fd43c0fcd40 msgr2=0x7fd43c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 -- 172.21.15.38:0/3149998697 shutdown_connections 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.171+0000 7fd44372c640 1 -- 172.21.15.38:0/3149998697 wait complete. 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 Processor -- start 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 -- start start 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 0x7fd43c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c109660 0x7fd43c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c10f620 con 0x7fd43c074120 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c10f4a0 con 0x7fd43c075700 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd44372c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c10f7a0 con 0x7fd43c109660 2024-04-22T00:01:55.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52690/0 (socket says 172.21.15.38:52690) 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 -- 172.21.15.38:0/2348925954 learned_addr learned my addr 172.21.15.38:0/2348925954 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd440cc7640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 0x7fd43c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd441cc9640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c109660 0x7fd43c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 -- 172.21.15.38:0/2348925954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c109660 msgr2=0x7fd43c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c109660 0x7fd43c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 -- 172.21.15.38:0/2348925954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 msgr2=0x7fd43c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 0x7fd43c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.172+0000 7fd4414c8640 1 -- 172.21.15.38:0/2348925954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43c1a1850 con 0x7fd43c074120 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd4414c8640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd43800d950 tx=0x7fd43800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438017070 con 0x7fd43c074120 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd438009c30 con 0x7fd43c074120 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43800f6b0 con 0x7fd43c074120 2024-04-22T00:01:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43c1a1b40 con 0x7fd43c074120 2024-04-22T00:01:55.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.173+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd43c1a2040 con 0x7fd43c074120 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.174+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43c108420 con 0x7fd43c074120 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.175+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd4380109d0 con 0x7fd43c074120 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.176+0000 7fd4327fc640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 0x7fd4280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.176+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd4380cef10 con 0x7fd43c074120 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.176+0000 7fd440cc7640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 0x7fd4280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.176+0000 7fd440cc7640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 0x7fd4280a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd42c005d00 tx=0x7fd42c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.180+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd438009de0 con 0x7fd43c074120 2024-04-22T00:01:55.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.367+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd43c02d3f0 con 0x7fd43c074120 2024-04-22T00:01:55.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.368+0000 7fd4327fc640 1 -- 172.21.15.38:0/2348925954 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24638 (secure 0 0 0) 0x7fd43809b4c0 con 0x7fd43c074120 2024-04-22T00:01:55.361 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:01:55.361 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":91,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:00:27.347647+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:01:55.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.370+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 msgr2=0x7fd4280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.370+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 0x7fd4280a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd42c005d00 tx=0x7fd42c00b410 comp rx=0 tx=0).stop 2024-04-22T00:01:55.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.370+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 msgr2=0x7fd43c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.370+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd43800d950 tx=0x7fd43800de10 comp rx=0 tx=0).stop 2024-04-22T00:01:55.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 shutdown_connections 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fd4280a2b90 0x7fd4280a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd43c109660 0x7fd43c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43c075700 0x7fd43c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 --2- 172.21.15.38:0/2348925954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd43c074120 0x7fd43c199c40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 >> 172.21.15.38:0/2348925954 conn(0x7fd43c0fcd40 msgr2=0x7fd43c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 shutdown_connections 2024-04-22T00:01:55.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.371+0000 7fd44372c640 1 -- 172.21.15.38:0/2348925954 wait complete. 2024-04-22T00:01:55.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:59:36.029248+0000', 'modified': '2024-04-22T00:00:05.266438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24487, 'mds_2': 24460, 'mds_3': 24455, 'mds_4': 24448}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6839/3894275312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 3894275312}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 3894275312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14634': {'gid': 14634, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6841/3401657425', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6840', 'nonce': 3401657425}, {'type': 'v1', 'addr': '172.21.15.38:6841', 'nonce': 3401657425}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24395': {'gid': 24395, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6833/618180230', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6832', 'nonce': 618180230}, {'type': 'v1', 'addr': '172.21.15.111:6833', 'nonce': 618180230}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24409': {'gid': 24409, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/1279747281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 1279747281}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 1279747281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.111:6835/3450283861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 3450283861}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 3450283861}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.203:6835/521403621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 521403621}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 521403621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.111:6837/782972998', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6836', 'nonce': 782972998}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 782972998}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/1809489419', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 1809489419}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1809489419}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.203:6839/3602809941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3602809941}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3602809941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6839/3890542559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6838', 'nonce': 3890542559}, {'type': 'v1', 'addr': '172.21.15.111:6839', 'nonce': 3890542559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24487, 24500, 24460, 24455, 24448]} 2024-04-22T00:01:55.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T00:01:55.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-22T00:01:55.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 -- 172.21.15.38:0/1439377375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954107460 msgr2=0x7f19541098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 --2- 172.21.15.38:0/1439377375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954107460 0x7f19541098b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f194c009ca0 tx=0x7f194c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 -- 172.21.15.38:0/1439377375 shutdown_connections 2024-04-22T00:01:55.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 --2- 172.21.15.38:0/1439377375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954107460 0x7f19541098b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 --2- 172.21.15.38:0/1439377375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 --2- 172.21.15.38:0/1439377375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954102200 0x7f19541045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 -- 172.21.15.38:0/1439377375 >> 172.21.15.38:0/1439377375 conn(0x7f19540fbbf0 msgr2=0x7f19540fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 -- 172.21.15.38:0/1439377375 shutdown_connections 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.487+0000 7f195b3ba640 1 -- 172.21.15.38:0/1439377375 wait complete. 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 Processor -- start 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 -- start start 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 0x7f195419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d420 con 0x7f1954102200 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d2a0 con 0x7f1954104b30 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f195b3ba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f195410d5a0 con 0x7f1954107460 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f1959156640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f1959156640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52716/0 (socket says 172.21.15.38:52716) 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f1959156640 1 -- 172.21.15.38:0/4048989411 learned_addr learned my addr 172.21.15.38:0/4048989411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f1959957640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 0x7f195419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.488+0000 7f1958955640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 -- 172.21.15.38:0/4048989411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 msgr2=0x7f195419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 0x7f195419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 -- 172.21.15.38:0/4048989411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 msgr2=0x7f1954197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 -- 172.21.15.38:0/4048989411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195419f6b0 con 0x7f1954102200 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959957640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 0x7f195419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1958955640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f1959156640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f195000c970 tx=0x7f195000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f194a7fc640 1 -- 172.21.15.38:0/4048989411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950007bb0 con 0x7f1954102200 2024-04-22T00:01:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f194a7fc640 1 -- 172.21.15.38:0/4048989411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950007d50 con 0x7f1954102200 2024-04-22T00:01:55.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f194a7fc640 1 -- 172.21.15.38:0/4048989411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000f450 con 0x7f1954102200 2024-04-22T00:01:55.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195419f9a0 con 0x7f1954102200 2024-04-22T00:01:55.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.489+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195419fea0 con 0x7f1954102200 2024-04-22T00:01:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.491+0000 7f194a7fc640 1 -- 172.21.15.38:0/4048989411 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1950005020 con 0x7f1954102200 2024-04-22T00:01:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 Processor -- start 2024-04-22T00:01:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 -- start start 2024-04-22T00:01:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3070 0x55ac7dbc3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbcb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac7db27df0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dad0640 con 0x55ac7dbc3070 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f195df9c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ac7db23db0 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f194a7fc640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 0x7f19400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52718/0 (socket says 172.21.15.38:52718) 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 -- 172.21.15.38:0/41481567 learned_addr learned my addr 172.21.15.38:0/41481567 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959957640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 0x7f19400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959957640 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3070 0x55ac7dbc3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1958955640 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbcb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f194a7fc640 1 -- 172.21.15.38:0/4048989411 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19500cf290 con 0x7f1954102200 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 -- 172.21.15.38:0/41481567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc8bd0 msgr2=0x55ac7dbcb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbcb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 -- 172.21.15.38:0/41481567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3070 msgr2=0x55ac7dbc3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3070 0x55ac7dbc3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.492+0000 7f1959156640 1 -- 172.21.15.38:0/41481567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7d83c2d0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f1959156640 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f19500cc940 tx=0x7f1950004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcf2ed922de3e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f1959957640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 0x7f19400a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f194c006010 tx=0x7f194c009990 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f193f7fe640 1 -- 172.21.15.38:0/41481567 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195001d070 con 0x55ac7dbc3a60 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f193f7fe640 1 -- 172.21.15.38:0/41481567 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19500a43a0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f193f7fe640 1 -- 172.21.15.38:0/41481567 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950099b10 con 0x55ac7dbc3a60 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 -- 172.21.15.38:0/41481567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 msgr2=0x55ac7dbc8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f19500cc940 tx=0x7f1950004a20 comp rx=0 tx=0).stop 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 -- 172.21.15.38:0/41481567 shutdown_connections 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbcb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbc8560 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 --2- 172.21.15.38:0/41481567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3070 0x55ac7dbc3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 -- 172.21.15.38:0/41481567 >> 172.21.15.38:0/41481567 conn(0x55ac7dbbe5e0 msgr2=0x55ac7dbbe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 -- 172.21.15.38:0/41481567 shutdown_connections 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.493+0000 7f195df9c740 1 -- 172.21.15.38:0/41481567 wait complete. 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 -- start start 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 Processor -- start 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbcd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbcbac0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dae3be0 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbcbc40 con 0x55ac7dbc3070 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f1958955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f195df9c740 10 client.0 _ll_register_callbacks cb 0x55ac7dae35d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f1958955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52732/0 (socket says 172.21.15.38:52732) 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f1958955640 1 -- 172.21.15.38:0/2763766817 learned_addr learned my addr 172.21.15.38:0/2763766817 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f1959156640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbcd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.494+0000 7f1959957640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 -- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 msgr2=0x55ac7dbcd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbcd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 -- 172.21.15.38:0/2763766817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 msgr2=0x55ac7dbd49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 -- 172.21.15.38:0/2763766817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7d9066b0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1959957640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f1958955640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f194400bf00 tx=0x7f19440117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddd5b07a7953eda server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f193cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944004440 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19440035f0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19500a43a0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19440037f0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f195df9c740 10 client.15426 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac7dbd7a50 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.495+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19500a43a0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f194401bc90 con 0x55ac7dbc3a60 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 10 client.15426 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 10 client.15426 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 20 client.15426 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dbda120 0x55ac7dbdb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 4 client.15426 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac7db35330 con 0x55ac7dbda120 2024-04-22T00:01:55.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f1959156640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dbda120 0x55ac7dbdb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f193cff9640 10 client.15426 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.496+0000 7f1959156640 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dbda120 0x55ac7dbdb590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c33ec4bb0ed3e7d1 server_cookie=7656eab7af3dc2d2 in_seq=0 out_seq=0 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f193cff9640 1 -- 172.21.15.38:0/2763766817 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac7db35330 con 0x55ac7dbda120 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f193cff9640 10 client.15426 handle_command_reply: tid=0 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 client.15426 shutdown 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 msgr2=0x55ac7dbcd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f194400bf00 tx=0x7f19440117b0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 shutdown_connections 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dbda120 0x55ac7dbdb590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f193cff9640 0 client.15426 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbcd960 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 --2- 172.21.15.38:0/2763766817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbcd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 >> 172.21.15.38:0/2763766817 conn(0x55ac7dbbe5e0 msgr2=0x55ac7dbbd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 shutdown_connections 2024-04-22T00:01:55.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.497+0000 7f195df9c740 1 -- 172.21.15.38:0/2763766817 wait complete. 2024-04-22T00:01:55.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.498+0000 7f195df9c740 20 client.15426 trim_cache size 0 max 16384 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 Processor -- start 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 -- start start 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbbce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbbd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbd98d0 con 0x55ac7dbc3a60 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbd9fa0 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f195df9c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbda670 con 0x55ac7dbc3070 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1958955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1958955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52748/0 (socket says 172.21.15.38:52748) 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1958955640 1 -- 172.21.15.38:0/1546222138 learned_addr learned my addr 172.21.15.38:0/1546222138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1959156640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbbce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1959957640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbbd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.516+0000 7f1958955640 1 -- 172.21.15.38:0/1546222138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 msgr2=0x55ac7dbbce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1958955640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbbce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1958955640 1 -- 172.21.15.38:0/1546222138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 msgr2=0x55ac7dbbd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1958955640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbbd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1958955640 1 -- 172.21.15.38:0/1546222138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7dbbf350 con 0x55ac7dbc3a60 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1959957640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbbd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f1958955640 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1944000c00 tx=0x7f1944003670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923e4be526da80bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f193effd640 1 -- 172.21.15.38:0/1546222138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194402d020 con 0x55ac7dbc3a60 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f193effd640 1 -- 172.21.15.38:0/1546222138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1944016070 con 0x55ac7dbc3a60 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f193effd640 1 -- 172.21.15.38:0/1546222138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944012740 con 0x55ac7dbc3a60 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f195df9c740 1 -- 172.21.15.38:0/1546222138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 msgr2=0x55ac7dbd4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.517+0000 7f195df9c740 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1944000c00 tx=0x7f1944003670 comp rx=0 tx=0).stop 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- 172.21.15.38:0/1546222138 shutdown_connections 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbbd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd4960 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- 172.21.15.38:0/1546222138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbbce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- 172.21.15.38:0/1546222138 >> 172.21.15.38:0/1546222138 conn(0x55ac7dbbe5e0 msgr2=0x55ac7dbbe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- 172.21.15.38:0/1546222138 shutdown_connections 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- 172.21.15.38:0/1546222138 wait complete. 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- start start 2024-04-22T00:01:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 Processor -- start 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbd0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbdb410 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbdad40 con 0x55ac7dbc3a60 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ac7dbdbae0 con 0x55ac7dbc3070 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f195df9c740 10 client.0 _ll_register_callbacks cb 0x7f194400bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.518+0000 7f1959957640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52764/0 (socket says 172.21.15.38:52764) 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 -- 172.21.15.38:0/762422499 learned_addr learned my addr 172.21.15.38:0/762422499 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959156640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbd0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1958955640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 -- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 msgr2=0x55ac7dbd0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbd0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 -- 172.21.15.38:0/762422499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3a60 msgr2=0x55ac7dbd1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 -- 172.21.15.38:0/762422499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac7dbd4310 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f1959957640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f194c009ca0 tx=0x7f194c046070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a121302a8f41234 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f193cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c04a050 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f194c004040 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1944016070 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f194c05d050 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f195df9c740 10 client.15432 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.519+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ac7dc407f0 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1944016070 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f194c04f7c0 con 0x55ac7dbc8bd0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 10 client.15432 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 10 client.15432 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 20 client.15432 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dc454a0 0x55ac7dc441b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 4 client.15432 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "status"}) -- 0x55ac7dbcbdd0 con 0x55ac7dc454a0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f1959156640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dc454a0 0x55ac7dc441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f1959156640 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dc454a0 0x55ac7dc441b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=43d565b14eba4457 server_cookie=a1ec633ac51f16b in_seq=0 out_seq=0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.520+0000 7f193cff9640 10 client.15432 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f193cff9640 1 -- 172.21.15.38:0/762422499 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ac7dbcbdd0 con 0x55ac7dc454a0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f193cff9640 10 client.15432 handle_command_reply: tid=0 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 client.15432 shutdown 2024-04-22T00:01:55.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 msgr2=0x55ac7dbd3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f194c009ca0 tx=0x7f194c046070 comp rx=0 tx=0).stop 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 shutdown_connections 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ac7dc454a0 0x55ac7dc441b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac7dbc8bd0 0x55ac7dbd3cd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f193cff9640 0 client.15432 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ac7dbc3a60 0x55ac7dbd1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 --2- 172.21.15.38:0/762422499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ac7dbc3070 0x55ac7dbd0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 >> 172.21.15.38:0/762422499 conn(0x55ac7dbbe5e0 msgr2=0x55ac7dbbce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 shutdown_connections 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 1 -- 172.21.15.38:0/762422499 wait complete. 2024-04-22T00:01:55.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195df9c740 20 client.15432 trim_cache size 0 max 16384 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 4, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24448, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 111.257462614, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:55.521 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 161.849205944 2024-04-22T00:01:55.522 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:55.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 msgr2=0x7f19400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 0x7f19400a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f194c006010 tx=0x7f194c009990 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 msgr2=0x7f1954197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.521+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f195000c970 tx=0x7f195000ce30 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 shutdown_connections 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f19400a2b90 0x7f19400a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954107460 0x7f195419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1954104b30 0x7f1954197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 --2- 172.21.15.38:0/4048989411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1954102200 0x7f1954197a90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 >> 172.21.15.38:0/4048989411 conn(0x7f19540fbbf0 msgr2=0x7f1954102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 shutdown_connections 2024-04-22T00:01:55.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.522+0000 7f195b3ba640 1 -- 172.21.15.38:0/4048989411 wait complete. 2024-04-22T00:01:55.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 -- 172.21.15.38:0/4258759764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c101790 msgr2=0x7fe37c101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 --2- 172.21.15.38:0/4258759764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c101790 0x7fe37c101bb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009ca0 tx=0x7fe36c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 -- 172.21.15.38:0/4258759764 shutdown_connections 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 --2- 172.21.15.38:0/4258759764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c103340 0x7fe37c109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 --2- 172.21.15.38:0/4258759764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c102980 0x7fe37c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 --2- 172.21.15.38:0/4258759764 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c101790 0x7fe37c101bb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 -- 172.21.15.38:0/4258759764 >> 172.21.15.38:0/4258759764 conn(0x7fe37c0fcd20 msgr2=0x7fe37c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 -- 172.21.15.38:0/4258759764 shutdown_connections 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.635+0000 7fe380adf640 1 -- 172.21.15.38:0/4258759764 wait complete. 2024-04-22T00:01:55.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 Processor -- start 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 -- start start 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c101790 0x7fe37c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c102980 0x7fe37c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c10d4e0 con 0x7fe37c103340 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c10d360 con 0x7fe37c102980 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe380adf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c10d660 con 0x7fe37c101790 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52788/0 (socket says 172.21.15.38:52788) 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 -- 172.21.15.38:0/3400800403 learned_addr learned my addr 172.21.15.38:0/3400800403 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37a59c640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c101790 0x7fe37c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe379d9b640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c102980 0x7fe37c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 -- 172.21.15.38:0/3400800403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c101790 msgr2=0x7fe37c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c101790 0x7fe37c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 -- 172.21.15.38:0/3400800403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c102980 msgr2=0x7fe37c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c102980 0x7fe37c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.636+0000 7fe37ad9d640 1 -- 172.21.15.38:0/3400800403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c074900 con 0x7fe37c103340 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe37ad9d640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe374009fd0 tx=0x7fe37400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe36b7fe640 1 -- 172.21.15.38:0/3400800403 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374011830 con 0x7fe37c103340 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c1a1d00 con 0x7fe37c103340 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe36b7fe640 1 -- 172.21.15.38:0/3400800403 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe374016070 con 0x7fe37c103340 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe36b7fe640 1 -- 172.21.15.38:0/3400800403 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37400f2e0 con 0x7fe37c103340 2024-04-22T00:01:55.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.637+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c1a2220 con 0x7fe37c103340 2024-04-22T00:01:55.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.639+0000 7fe36b7fe640 1 -- 172.21.15.38:0/3400800403 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe3740119d0 con 0x7fe37c103340 2024-04-22T00:01:55.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.639+0000 7fe3837a3740 1 Processor -- start 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 -- start start 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03070 0x55d7ecc03490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc08560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecb67df0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecb10640 con 0x55d7ecc03a60 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe3837a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecb63db0 con 0x55d7ecc03070 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52792/0 (socket says 172.21.15.38:52792) 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 -- 172.21.15.38:0/3851440977 learned_addr learned my addr 172.21.15.38:0/3851440977 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03070 0x55d7ecc03490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe37a59c640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc08560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe36b7fe640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 0x7fe3640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 -- 172.21.15.38:0/3851440977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03070 msgr2=0x55d7ecc03490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03070 0x55d7ecc03490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 -- 172.21.15.38:0/3851440977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 msgr2=0x55d7ecc08560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc08560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 -- 172.21.15.38:0/3851440977 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ec87c2d0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe37a59c640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc08560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe37ad9d640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 0x7fe3640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe379d9b640 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe370016500 tx=0x7fe37000a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2382b0ca247914b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.640+0000 7fe36b7fe640 1 -- 172.21.15.38:0/3400800403 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe3740ceb70 con 0x7fe37c103340 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe368ff9640 1 -- 172.21.15.38:0/3851440977 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370018430 con 0x55d7ecc08bd0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe368ff9640 1 -- 172.21.15.38:0/3851440977 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe370018ab0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe368ff9640 1 -- 172.21.15.38:0/3851440977 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370017a10 con 0x55d7ecc08bd0 2024-04-22T00:01:55.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 -- 172.21.15.38:0/3851440977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 msgr2=0x55d7ecc0b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe37ad9d640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 0x7fe3640a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe37409ad70 tx=0x7fe37406e130 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe370016500 tx=0x7fe37000a2b0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 -- 172.21.15.38:0/3851440977 shutdown_connections 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0b060 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc08560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 --2- 172.21.15.38:0/3851440977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03070 0x55d7ecc03490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 -- 172.21.15.38:0/3851440977 >> 172.21.15.38:0/3851440977 conn(0x55d7ecbfe5e0 msgr2=0x55d7ecbfe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.641+0000 7fe3837a3740 1 -- 172.21.15.38:0/3851440977 shutdown_connections 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 -- 172.21.15.38:0/3851440977 wait complete. 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 -- start start 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 Processor -- start 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc0d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc0d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc0bac0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecb23be0 con 0x55d7ecc03070 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc0bc40 con 0x55d7ecc03a60 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe3837a3740 10 client.0 _ll_register_callbacks cb 0x55d7ecb235d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52798/0 (socket says 172.21.15.38:52798) 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 -- 172.21.15.38:0/2687690466 learned_addr learned my addr 172.21.15.38:0/2687690466 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc0d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc0d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 -- 172.21.15.38:0/2687690466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 msgr2=0x55d7ecc0d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc0d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.642+0000 7fe379d9b640 1 -- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 msgr2=0x55d7ecc0d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc0d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe379d9b640 1 -- 172.21.15.38:0/2687690466 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ec9466b0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe37001a040 tx=0x7fe3700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8360c5b8800c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370018900 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe370017790 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe370018ab0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe3700059c0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 10 client.15444 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d7ecc17970 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe370018ab0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe370012050 con 0x55d7ecc08bd0 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 10 client.15444 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 10 client.15444 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.643+0000 7fe3837a3740 20 client.15444 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc19c40 0x55d7ecc1b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe3837a3740 4 client.15444 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7ecb75330 con 0x55d7ecc19c40 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc19c40 0x55d7ecc1b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe361ffb640 10 client.15444 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.644+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc19c40 0x55d7ecc1b0b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=53d7bcb021ebf779 server_cookie=7aa3b4c8b8d6e325 in_seq=0 out_seq=0 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe361ffb640 1 -- 172.21.15.38:0/2687690466 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7ecb75330 con 0x55d7ecc19c40 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe361ffb640 10 client.15444 handle_command_reply: tid=0 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 client.15444 shutdown 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 msgr2=0x55d7ecc14990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe37001a040 tx=0x7fe3700046c0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 shutdown_connections 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc19c40 0x55d7ecc1b0b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc14990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc0d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe361ffb640 0 client.15444 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2687690466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc0d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 >> 172.21.15.38:0/2687690466 conn(0x55d7ecbfe5e0 msgr2=0x55d7ecbfd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 shutdown_connections 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 1 -- 172.21.15.38:0/2687690466 wait complete. 2024-04-22T00:01:55.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.645+0000 7fe3837a3740 20 client.15444 trim_cache size 0 max 16384 2024-04-22T00:01:55.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 Processor -- start 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 -- start start 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc193f0 con 0x55d7ecc03070 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc19ac0 con 0x55d7ecc03a60 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe3837a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc1a190 con 0x55d7ecc08bd0 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52804/0 (socket says 172.21.15.38:52804) 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 -- 172.21.15.38:0/2171617833 learned_addr learned my addr 172.21.15.38:0/2171617833 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 -- 172.21.15.38:0/2171617833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc08bd0 msgr2=0x55d7ecc0d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 -- 172.21.15.38:0/2171617833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 msgr2=0x55d7ecc148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc148b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 -- 172.21.15.38:0/2171617833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7ecc03e40 con 0x55d7ecc03070 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.664+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe370006420 tx=0x7fe37000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf70c330aab755b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe363fff640 1 -- 172.21.15.38:0/2171617833 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37003a070 con 0x55d7ecc03070 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe363fff640 1 -- 172.21.15.38:0/2171617833 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37000a6c0 con 0x55d7ecc03070 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe363fff640 1 -- 172.21.15.38:0/2171617833 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370005f50 con 0x55d7ecc03070 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 -- 172.21.15.38:0/2171617833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 msgr2=0x55d7ecc18dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe370006420 tx=0x7fe37000af40 comp rx=0 tx=0).stop 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 -- 172.21.15.38:0/2171617833 shutdown_connections 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc0d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03a60 0x55d7ecc148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2171617833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc03070 0x55d7ecc18dd0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 -- 172.21.15.38:0/2171617833 >> 172.21.15.38:0/2171617833 conn(0x55d7ecbfe5e0 msgr2=0x55d7ecc0d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 -- 172.21.15.38:0/2171617833 shutdown_connections 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.665+0000 7fe3837a3740 1 -- 172.21.15.38:0/2171617833 wait complete. 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 -- start start 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 Processor -- start 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc14210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc14750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc1af30 con 0x55d7ecc08bd0 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc1a860 con 0x55d7ecc03070 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d7ecc1b600 con 0x55d7ecc03a60 2024-04-22T00:01:55.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe3837a3740 10 client.0 _ll_register_callbacks cb 0x7fe370016400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52812/0 (socket says 172.21.15.38:52812) 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 -- 172.21.15.38:0/2360530359 learned_addr learned my addr 172.21.15.38:0/2360530359 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc14210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc14750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 -- 172.21.15.38:0/2360530359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 msgr2=0x55d7ecc14750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc14750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 -- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 msgr2=0x55d7ecc14210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc14210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 -- 172.21.15.38:0/2360530359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7ecc114e0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe379d9b640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe370038040 tx=0x7fe370006ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db1ae28017d3cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.666+0000 7fe361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37003a070 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe370005cd0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe37000a6c0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe37004c050 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 10 client.15450 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d7ecc80900 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37000a6c0 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe370012040 con 0x55d7ecc08bd0 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 10 client.15450 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 10 client.15450 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 20 client.15450 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc85070 0x55d7ecc83d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 4 client.15450 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "status"}) -- 0x55d7ecb6f680 con 0x55d7ecc85070 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.667+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc85070 0x55d7ecc83d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe361ffb640 10 client.15450 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe37a59c640 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc85070 0x55d7ecc83d80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15d473e686b8cc2c server_cookie=f7a96259c44e4c31 in_seq=0 out_seq=0 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe361ffb640 1 -- 172.21.15.38:0/2360530359 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7ecb6f680 con 0x55d7ecc85070 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe361ffb640 10 client.15450 handle_command_reply: tid=0 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 client.15450 shutdown 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 msgr2=0x55d7ecc10cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe370038040 tx=0x7fe370006ad0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 shutdown_connections 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d7ecc85070 0x55d7ecc83d80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d7ecc08bd0 0x55d7ecc10cb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d7ecc03a60 0x55d7ecc14750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe361ffb640 0 client.15450 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 --2- 172.21.15.38:0/2360530359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d7ecc03070 0x55d7ecc14210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 >> 172.21.15.38:0/2360530359 conn(0x55d7ecbfe5e0 msgr2=0x55d7ecbfce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 shutdown_connections 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 1 -- 172.21.15.38:0/2360530359 wait complete. 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.668+0000 7fe3837a3740 20 client.15450 trim_cache size 0 max 16384 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 3, 2024-04-22T00:01:55.668 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24455, 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 112.41679101299999, 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 156.13941038999999 2024-04-22T00:01:55.669 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 msgr2=0x7fe3640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 0x7fe3640a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe37409ad70 tx=0x7fe37406e130 comp rx=0 tx=0).stop 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 msgr2=0x7fe37c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe374009fd0 tx=0x7fe37400b8e0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 shutdown_connections 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fe3640a2b90 0x7fe3640a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe37c103340 0x7fe37c074070 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe37c102980 0x7fe37c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 --2- 172.21.15.38:0/3400800403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe37c101790 0x7fe37c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 >> 172.21.15.38:0/3400800403 conn(0x7fe37c0fcd20 msgr2=0x7fe37c105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 shutdown_connections 2024-04-22T00:01:55.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.669+0000 7fe380adf640 1 -- 172.21.15.38:0/3400800403 wait complete. 2024-04-22T00:01:55.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:55 smithi038 ceph-mon[27972]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:55.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:55 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2191171321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:55.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:55 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2957566776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:55 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2348925954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 -- 172.21.15.38:0/3564502232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 msgr2=0x7f5e40102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 --2- 172.21.15.38:0/3564502232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40102e00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009900 tx=0x7f5e2c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 -- 172.21.15.38:0/3564502232 shutdown_connections 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 --2- 172.21.15.38:0/3564502232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40103340 0x7f5e40109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 --2- 172.21.15.38:0/3564502232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40102e00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 --2- 172.21.15.38:0/3564502232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40101790 0x7f5e40101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 -- 172.21.15.38:0/3564502232 >> 172.21.15.38:0/3564502232 conn(0x7f5e400fcd20 msgr2=0x7f5e400ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 -- 172.21.15.38:0/3564502232 shutdown_connections 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 -- 172.21.15.38:0/3564502232 wait complete. 2024-04-22T00:01:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.784+0000 7f5e4705f640 1 Processor -- start 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 -- start start 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40101790 0x7f5e40197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 0x7f5e4019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4010d410 con 0x7f5e40102980 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4010d290 con 0x7f5e40101790 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e4705f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4010d590 con 0x7f5e40103340 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52830/0 (socket says 172.21.15.38:52830) 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 -- 172.21.15.38:0/790006184 learned_addr learned my addr 172.21.15.38:0/790006184 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40101790 0x7f5e40197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e455fc640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 0x7f5e4019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 -- 172.21.15.38:0/790006184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 msgr2=0x7f5e4019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 0x7f5e4019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 -- 172.21.15.38:0/790006184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40101790 msgr2=0x7f5e40197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40101790 0x7f5e40197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 -- 172.21.15.38:0/790006184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4019f6d0 con 0x7f5e40102980 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e455fc640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 0x7f5e4019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.785+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009230 tx=0x7f5e2c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.786+0000 7f5e3dffb640 1 -- 172.21.15.38:0/790006184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c041070 con 0x7f5e40102980 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.786+0000 7f5e3dffb640 1 -- 172.21.15.38:0/790006184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c0054e0 con 0x7f5e40102980 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.786+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4019f960 con 0x7f5e40102980 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.786+0000 7f5e3dffb640 1 -- 172.21.15.38:0/790006184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c034440 con 0x7f5e40102980 2024-04-22T00:01:55.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.786+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4019fe50 con 0x7f5e40102980 2024-04-22T00:01:55.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:55 smithi203 ceph-mon[31431]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:55.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:55 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2191171321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:55.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:55 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2957566776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:55 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2348925954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.788+0000 7f5e3dffb640 1 -- 172.21.15.38:0/790006184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5e2c044070 con 0x7f5e40102980 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.788+0000 7f5e49c42740 1 Processor -- start 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.788+0000 7f5e49c42740 1 -- start start 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3dffb640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 0x7f5e300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 0x55b1df5c1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1df519df0 con 0x55b1df5b9e70 2024-04-22T00:01:55.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b1df4c2640 con 0x55b1df5befe0 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e49c42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1df515db0 con 0x55b1df5b9480 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 0x7f5e300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52838/0 (socket says 172.21.15.38:52838) 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3620634810 learned_addr learned my addr 172.21.15.38:0/3620634810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 0x55b1df5c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3dffb640 1 -- 172.21.15.38:0/790006184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e2c0f1180 con 0x7f5e40102980 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3620634810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 msgr2=0x55b1df5b98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5b98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3620634810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 msgr2=0x55b1df5c1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 0x55b1df5c1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3620634810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1df22e2d0 con 0x55b1df5b9e70 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 0x55b1df5c1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 0x7f5e300a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e34005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ee8c0 tx=0x7f5e2c088690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f32cf3e04be4bd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.789+0000 7f5e2affd640 1 -- 172.21.15.38:0/3620634810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c04b020 con 0x55b1df5b9e70 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e2affd640 1 -- 172.21.15.38:0/3620634810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c005690 con 0x55b1df5b9e70 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e2affd640 1 -- 172.21.15.38:0/3620634810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c088700 con 0x55b1df5b9e70 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- 172.21.15.38:0/3620634810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 msgr2=0x55b1df5be970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ee8c0 tx=0x7f5e2c088690 comp rx=0 tx=0).stop 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- 172.21.15.38:0/3620634810 shutdown_connections 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5befe0 0x55b1df5c1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5be970 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3620634810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5b98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- 172.21.15.38:0/3620634810 >> 172.21.15.38:0/3620634810 conn(0x55b1df5b05e0 msgr2=0x55b1df5b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- 172.21.15.38:0/3620634810 shutdown_connections 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- 172.21.15.38:0/3620634810 wait complete. 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.790+0000 7f5e49c42740 1 -- start start 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 Processor -- start 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1df519df0 con 0x55b1df5befe0 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b1df4d5be0 con 0x55b1df5b9e70 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5c1d50 con 0x55b1df5b9480 2024-04-22T00:01:55.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e49c42740 10 client.0 _ll_register_callbacks cb 0x55b1df4d55d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52848/0 (socket says 172.21.15.38:52848) 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 -- 172.21.15.38:0/3542042929 learned_addr learned my addr 172.21.15.38:0/3542042929 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 -- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 msgr2=0x55b1df5c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 -- 172.21.15.38:0/3542042929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 msgr2=0x55b1df5c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.791+0000 7f5e455fc640 1 -- 172.21.15.38:0/3542042929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1df2f86b0 con 0x55b1df5befe0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5e38012c80 tx=0x7f5e380117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca6dc6a2b069554 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38004d20 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e38005020 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1df527330 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e38027050 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 10 client.15462 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1df5cdc00 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b1df527330 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5e380225d0 con 0x55b1df5befe0 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 10 client.15462 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 10 client.15462 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 20 client.15462 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df5d0370 0x55b1df5d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 4 client.15462 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.792+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1df521680 con 0x55b1df5d0370 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.793+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df5d0370 0x55b1df5d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.793+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df5d0370 0x55b1df5d17e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8d7646366a55e2d server_cookie=83dec4771cf00235 in_seq=0 out_seq=0 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.793+0000 7f5e1ffff640 10 client.15462 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3542042929 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1df521680 con 0x55b1df5d0370 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e1ffff640 10 client.15462 handle_command_reply: tid=0 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 client.15462 shutdown 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 msgr2=0x55b1df5cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5e38012c80 tx=0x7f5e380117b0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 shutdown_connections 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df5d0370 0x55b1df5d17e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5cabb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e1ffff640 0 client.15462 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3542042929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 >> 172.21.15.38:0/3542042929 conn(0x55b1df5b05e0 msgr2=0x55b1df5af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 shutdown_connections 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 1 -- 172.21.15.38:0/3542042929 wait complete. 2024-04-22T00:01:55.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.794+0000 7f5e49c42740 20 client.15462 trim_cache size 0 max 16384 2024-04-22T00:01:55.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 Processor -- start 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 -- start start 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9480 0x55b1df5aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 0x55b1df5af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5cfb20 con 0x55b1df5b9e70 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5d01f0 con 0x55b1df5b9480 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e49c42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5d08c0 con 0x55b1df5befe0 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52858/0 (socket says 172.21.15.38:52858) 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3330407034 learned_addr learned my addr 172.21.15.38:0/3330407034 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9480 0x55b1df5aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 0x55b1df5af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3330407034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 msgr2=0x55b1df5af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 0x55b1df5af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3330407034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9480 msgr2=0x55b1df5aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9480 0x55b1df5aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 -- 172.21.15.38:0/3330407034 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1df5c47b0 con 0x55b1df5b9e70 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 0x55b1df5af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0bd320 tx=0x7f5e2c0b92c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e05e2119063a430 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e2a7fc640 1 -- 172.21.15.38:0/3330407034 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c04b040 con 0x55b1df5b9e70 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e2a7fc640 1 -- 172.21.15.38:0/3330407034 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c004c60 con 0x55b1df5b9e70 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.813+0000 7f5e2a7fc640 1 -- 172.21.15.38:0/3330407034 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0bd620 con 0x55b1df5b9e70 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- 172.21.15.38:0/3330407034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 msgr2=0x55b1df5caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0bd320 tx=0x7f5e2c0b92c0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- 172.21.15.38:0/3330407034 shutdown_connections 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5befe0 0x55b1df5af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5b9e70 0x55b1df5caad0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3330407034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9480 0x55b1df5aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- 172.21.15.38:0/3330407034 >> 172.21.15.38:0/3330407034 conn(0x55b1df5b05e0 msgr2=0x55b1df5ab340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- 172.21.15.38:0/3330407034 shutdown_connections 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- 172.21.15.38:0/3330407034 wait complete. 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 -- start start 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.814+0000 7f5e49c42740 1 Processor -- start 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5d1660 con 0x55b1df5befe0 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5d0f90 con 0x55b1df5b9e70 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1df5d1d30 con 0x55b1df5b9480 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e49c42740 10 client.0 _ll_register_callbacks cb 0x7f5e38012950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52860/0 (socket says 172.21.15.38:52860) 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 -- 172.21.15.38:0/3650933589 learned_addr learned my addr 172.21.15.38:0/3650933589 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e3ffff640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 -- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 msgr2=0x55b1df5c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 -- 172.21.15.38:0/3650933589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 msgr2=0x55b1df5c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 -- 172.21.15.38:0/3650933589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1df5c9e80 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e455fc640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e38000f80 tx=0x7f5e38011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c78d603507268d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.815+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38009b30 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e2c004c60 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5e38004ab0 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e3803a050 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 10 client.15468 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1df636320 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e2c004c60 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5e3802b760 con 0x55b1df5befe0 2024-04-22T00:01:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 10 client.15468 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 20 client.15468 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df63aeb0 0x55b1df639bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 4 client.15468 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "status"}) -- 0x55b1df49fa70 con 0x55b1df63aeb0 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.816+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df63aeb0 0x55b1df639bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e1ffff640 10 client.15468 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e44dfb640 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df63aeb0 0x55b1df639bc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=209686a5f26a4110 server_cookie=7b0ca7bf8b814251 in_seq=0 out_seq=0 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e1ffff640 1 -- 172.21.15.38:0/3650933589 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b1df49fa70 con 0x55b1df63aeb0 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e1ffff640 10 client.15468 handle_command_reply: tid=0 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 client.15468 shutdown 2024-04-22T00:01:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 msgr2=0x55b1df5c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e38000f80 tx=0x7f5e38011cb0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 shutdown_connections 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b1df63aeb0 0x55b1df639bc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b1df5befe0 0x55b1df5c9890 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b1df5b9e70 0x55b1df5c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e1ffff640 0 client.15468 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 --2- 172.21.15.38:0/3650933589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1df5b9480 0x55b1df5c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 >> 172.21.15.38:0/3650933589 conn(0x55b1df5b05e0 msgr2=0x55b1df5aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 shutdown_connections 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 1 -- 172.21.15.38:0/3650933589 wait complete. 2024-04-22T00:01:55.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.817+0000 7f5e49c42740 20 client.15468 trim_cache size 0 max 16384 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24460, 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 113.57297699999999, 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:55.819 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:55.820 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:55.820 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 159.27112809600001 2024-04-22T00:01:55.820 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 msgr2=0x7f5e300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 0x7f5e300a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e34005850 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 msgr2=0x7f5e40197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009230 tx=0x7f5e2c005dd0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 shutdown_connections 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f5e300a2b90 0x7f5e300a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e40103340 0x7f5e4019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e40102980 0x7f5e40197ff0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 --2- 172.21.15.38:0/790006184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e40101790 0x7f5e40197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 >> 172.21.15.38:0/790006184 conn(0x7f5e400fcd20 msgr2=0x7f5e40105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 shutdown_connections 2024-04-22T00:01:55.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.818+0000 7f5e4705f640 1 -- 172.21.15.38:0/790006184 wait complete. 2024-04-22T00:01:55.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T00:01:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:55 smithi111 ceph-mon[31543]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:01:55.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:55 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2191171321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:01:55.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:55 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2957566776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:55 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2348925954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:01:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 -- 172.21.15.38:0/1330937397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 msgr2=0x7f9784101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 --2- 172.21.15.38:0/1330937397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f9784101d10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f976c009900 tx=0x7f976c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 -- 172.21.15.38:0/1330937397 shutdown_connections 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 --2- 172.21.15.38:0/1330937397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 0x7f9784108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 --2- 172.21.15.38:0/1330937397 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f9784101d10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 --2- 172.21.15.38:0/1330937397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 0x7f9784100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 -- 172.21.15.38:0/1330937397 >> 172.21.15.38:0/1330937397 conn(0x7f97840fbc10 msgr2=0x7f97840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 -- 172.21.15.38:0/1330937397 shutdown_connections 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.933+0000 7f9788977640 1 -- 172.21.15.38:0/1330937397 wait complete. 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 Processor -- start 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 -- start start 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 0x7f978406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 0x7f978419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f978410c300 con 0x7f9784101890 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f978410c180 con 0x7f9784102250 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.934+0000 7f9788977640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f978410c480 con 0x7f97841006a0 2024-04-22T00:01:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52870/0 (socket says 172.21.15.38:52870) 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 -- 172.21.15.38:0/1844118645 learned_addr learned my addr 172.21.15.38:0/1844118645 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f978259c640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 0x7f978406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 0x7f978419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 -- 172.21.15.38:0/1844118645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 msgr2=0x7f978406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 0x7f978406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 -- 172.21.15.38:0/1844118645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 msgr2=0x7f978419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 0x7f978419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 -- 172.21.15.38:0/1844118645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978419f750 con 0x7f9784101890 2024-04-22T00:01:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977bfff640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f976c009230 tx=0x7f976c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977b7fe640 1 -- 172.21.15.38:0/1844118645 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c041070 con 0x7f9784101890 2024-04-22T00:01:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977b7fe640 1 -- 172.21.15.38:0/1844118645 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976c0054e0 con 0x7f9784101890 2024-04-22T00:01:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978419f980 con 0x7f9784101890 2024-04-22T00:01:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f977b7fe640 1 -- 172.21.15.38:0/1844118645 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c034440 con 0x7f9784101890 2024-04-22T00:01:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.935+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978419fe70 con 0x7f9784101890 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.937+0000 7f977b7fe640 1 -- 172.21.15.38:0/1844118645 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f976c044070 con 0x7f9784101890 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 Processor -- start 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 -- start start 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29ca60 0x55c06d2a1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2a4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c06d200df0 con 0x55c06d29c070 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c06d1a9640 con 0x55c06d29ca60 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978b636740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c06d1fcdb0 con 0x55c06d2a1bd0 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52884/0 (socket says 172.21.15.38:52884) 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 -- 172.21.15.38:0/1130195855 learned_addr learned my addr 172.21.15.38:0/1130195855 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f977b7fe640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 0x7f97700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f977bfff640 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f978259c640 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29ca60 0x55c06d2a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 0x7f97700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 -- 172.21.15.38:0/1130195855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 msgr2=0x55c06d2a4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.938+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2a4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9782d9d640 1 -- 172.21.15.38:0/1130195855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29ca60 msgr2=0x55c06d2a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f977b7fe640 1 -- 172.21.15.38:0/1844118645 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f976c0f1040 con 0x7f9784101890 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29ca60 0x55c06d2a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9782d9d640 1 -- 172.21.15.38:0/1130195855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06cf152d0 con 0x55c06d29c070 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f977c014a60 tx=0x7f977c014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da8136794853abb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9778ff9640 1 -- 172.21.15.38:0/1130195855 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c01ec50 con 0x55c06d29c070 2024-04-22T00:01:55.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9778ff9640 1 -- 172.21.15.38:0/1130195855 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f977c023070 con 0x55c06d29c070 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.939+0000 7f9778ff9640 1 -- 172.21.15.38:0/1130195855 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c0055e0 con 0x55c06d29c070 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- 172.21.15.38:0/1130195855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 msgr2=0x55c06d29c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 0x7f97700a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f977c022000 tx=0x7f977c01e000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f977c014a60 tx=0x7f977c014f20 comp rx=0 tx=0).stop 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- 172.21.15.38:0/1130195855 shutdown_connections 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29ca60 0x55c06d2a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- 172.21.15.38:0/1130195855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d29c490 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- 172.21.15.38:0/1130195855 >> 172.21.15.38:0/1130195855 conn(0x55c06d2975e0 msgr2=0x55c06d2979e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- 172.21.15.38:0/1130195855 shutdown_connections 2024-04-22T00:01:55.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- 172.21.15.38:0/1130195855 wait complete. 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- start start 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 Processor -- start 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d2a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 0x55c06d2adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2a4b80 con 0x55c06d29c070 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c06d1bcbe0 con 0x55c06d2a1bd0 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978b636740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2a4d00 con 0x55c06d29ca60 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.940+0000 7f978259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978b636740 10 client.0 _ll_register_callbacks cb 0x55c06d1bc5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52892/0 (socket says 172.21.15.38:52892) 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 -- 172.21.15.38:0/1775295157 learned_addr learned my addr 172.21.15.38:0/1775295157 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f977bfff640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d2a6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 0x55c06d2adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 -- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 msgr2=0x55c06d2a6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d2a6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 -- 172.21.15.38:0/1775295157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 msgr2=0x55c06d2adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 0x55c06d2adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 -- 172.21.15.38:0/1775295157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c06cfdf6b0 con 0x55c06d29c070 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9782d9d640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 0x55c06d2adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978259c640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f977400b9d0 tx=0x7f977400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49da55c01333ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9769ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774003dc0 con 0x55c06d29c070 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9774011440 con 0x55c06d29c070 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f977c023070 con 0x55c06d29c070 2024-04-22T00:01:55.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9774011640 con 0x55c06d29c070 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978b636740 10 client.15480 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c06d2b0b50 con 0x55c06d29c070 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.941+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f977c023070 con 0x55c06d29c070 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f977401b830 con 0x55c06d29c070 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 10 client.15480 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 10 client.15480 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 20 client.15480 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d2b32c0 0x55c06d2b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 4 client.15480 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c06d20e330 con 0x55c06d2b32c0 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f977bfff640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d2b32c0 0x55c06d2b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f977bfff640 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d2b32c0 0x55c06d2b4730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b082156596b40a9 server_cookie=64f667dd4661ccb0 in_seq=0 out_seq=0 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.942+0000 7f9769ffb640 10 client.15480 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f9769ffb640 1 -- 172.21.15.38:0/1775295157 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c06d20e330 con 0x55c06d2b32c0 2024-04-22T00:01:55.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f9769ffb640 10 client.15480 handle_command_reply: tid=0 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 client.15480 shutdown 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 msgr2=0x55c06d2a6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f977400b9d0 tx=0x7f977400bdd0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 shutdown_connections 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d2b32c0 0x55c06d2b4730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d2a1bd0 0x55c06d2adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d2a6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f9769ffb640 0 client.15480 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 --2- 172.21.15.38:0/1775295157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29c070 0x55c06d2a6620 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 >> 172.21.15.38:0/1775295157 conn(0x55c06d2975e0 msgr2=0x55c06d296720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 shutdown_connections 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 1 -- 172.21.15.38:0/1775295157 wait complete. 2024-04-22T00:01:55.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.943+0000 7f978b636740 20 client.15480 trim_cache size 0 max 16384 2024-04-22T00:01:55.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 Processor -- start 2024-04-22T00:01:55.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 -- start start 2024-04-22T00:01:55.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d29d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b2a70 con 0x55c06d29ca60 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b3140 con 0x55c06d29c070 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978b636740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b3810 con 0x55c06d2a1bd0 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52894/0 (socket says 172.21.15.38:52894) 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 -- 172.21.15.38:0/3259434776 learned_addr learned my addr 172.21.15.38:0/3259434776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978259c640 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d29d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 -- 172.21.15.38:0/3259434776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 msgr2=0x55c06d2979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 -- 172.21.15.38:0/3259434776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 msgr2=0x55c06d29d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d29d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 -- 172.21.15.38:0/3259434776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06d295c20 con 0x55c06d29ca60 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f978259c640 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d29d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.962+0000 7f977bfff640 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f976c02cfd0 tx=0x7f976c0a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab217f161bb9bb16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f976bfff640 1 -- 172.21.15.38:0/3259434776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c04b040 con 0x55c06d29ca60 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f976bfff640 1 -- 172.21.15.38:0/3259434776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f976c005690 con 0x55c06d29ca60 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f976bfff640 1 -- 172.21.15.38:0/3259434776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976c0ee890 con 0x55c06d29ca60 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- 172.21.15.38:0/3259434776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 msgr2=0x55c06d2ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f976c02cfd0 tx=0x7f976c0a8040 comp rx=0 tx=0).stop 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- 172.21.15.38:0/3259434776 shutdown_connections 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d2a1bd0 0x55c06d2979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d29ca60 0x55c06d2ada50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 --2- 172.21.15.38:0/3259434776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d29d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- 172.21.15.38:0/3259434776 >> 172.21.15.38:0/3259434776 conn(0x55c06d2975e0 msgr2=0x55c06d2a5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- 172.21.15.38:0/3259434776 shutdown_connections 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- 172.21.15.38:0/3259434776 wait complete. 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.963+0000 7f978b636740 1 -- start start 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 Processor -- start 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d2aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d315250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b45b0 con 0x55c06d2a1bd0 2024-04-22T00:01:55.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b3ee0 con 0x55c06d29c070 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c06d2b4c80 con 0x55c06d29ca60 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978b636740 10 client.0 _ll_register_callbacks cb 0x7f9774004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52902/0 (socket says 172.21.15.38:52902) 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 -- 172.21.15.38:0/3458300814 learned_addr learned my addr 172.21.15.38:0/3458300814 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f978259c640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d2aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f977bfff640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d315250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 -- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 msgr2=0x55c06d315250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d315250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 -- 172.21.15.38:0/3458300814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 msgr2=0x55c06d2aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d2aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 -- 172.21.15.38:0/3458300814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c06d317ee0 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9782d9d640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f977c0128d0 tx=0x7f977c012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd5370390c962cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.964+0000 7f9769ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c0144f0 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f977c01ec50 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c06d208680 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f977c03f050 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 10 client.15486 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c06d319130 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c06d208680 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f977c03a030 con 0x55c06d2a1bd0 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 10 client.15486 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:55.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 10 client.15486 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 20 client.15486 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d31da30 0x55c06d31a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 4 client.15486 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.965+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "status"}) -- 0x55c06d186a70 con 0x55c06d31da30 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978259c640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d31da30 0x55c06d31a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:55.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978259c640 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d31da30 0x55c06d31a540 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa793bba509e2dd1 server_cookie=9ef3223bdccf07ee in_seq=0 out_seq=0 2024-04-22T00:01:55.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f9769ffb640 10 client.15486 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:55.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f9769ffb640 1 -- 172.21.15.38:0/3458300814 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c06d186a70 con 0x55c06d31da30 2024-04-22T00:01:55.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f9769ffb640 10 client.15486 handle_command_reply: tid=0 2024-04-22T00:01:55.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 client.15486 shutdown 2024-04-22T00:01:55.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 msgr2=0x55c06d3177a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f977c0128d0 tx=0x7f977c012900 comp rx=0 tx=0).stop 2024-04-22T00:01:55.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 shutdown_connections 2024-04-22T00:01:55.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c06d31da30 0x55c06d31a540 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c06d2a1bd0 0x55c06d3177a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c06d29ca60 0x55c06d315250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f9769ffb640 0 client.15486 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 --2- 172.21.15.38:0/3458300814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c06d29c070 0x55c06d2aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 >> 172.21.15.38:0/3458300814 conn(0x55c06d2975e0 msgr2=0x55c06d2979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 shutdown_connections 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 1 -- 172.21.15.38:0/3458300814 wait complete. 2024-04-22T00:01:55.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.966+0000 7f978b636740 20 client.15486 trim_cache size 0 max 16384 2024-04-22T00:01:55.970 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24487, 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 114.729377529, 2024-04-22T00:01:55.971 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:55.972 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:55.972 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:55.972 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 153.60638353600001 2024-04-22T00:01:55.972 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:55.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 msgr2=0x7f97700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 0x7f97700a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f977c022000 tx=0x7f977c01e000 comp rx=0 tx=0).stop 2024-04-22T00:01:55.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 msgr2=0x7f978419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:55.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f976c009230 tx=0x7f976c005dd0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 shutdown_connections 2024-04-22T00:01:55.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f97700a2b90 0x7f97700a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9784102250 0x7f978419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9784101890 0x7f978419c820 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 --2- 172.21.15.38:0/1844118645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97841006a0 0x7f978406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 >> 172.21.15.38:0/1844118645 conn(0x7f97840fbc10 msgr2=0x7f9784104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 shutdown_connections 2024-04-22T00:01:55.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:55.967+0000 7f9788977640 1 -- 172.21.15.38:0/1844118645 wait complete. 2024-04-22T00:01:55.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1947334723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc102990 msgr2=0x7fc3cc102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1947334723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc102990 0x7fc3cc102e10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009900 tx=0x7fc3c002bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1947334723 shutdown_connections 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1947334723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 0x7fc3cc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1947334723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc102990 0x7fc3cc102e10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1947334723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1947334723 >> 172.21.15.38:0/1947334723 conn(0x7fc3cc0fcd10 msgr2=0x7fc3cc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1947334723 shutdown_connections 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.088+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1947334723 wait complete. 2024-04-22T00:01:56.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 Processor -- start 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 -- start start 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc102990 0x7fc3cc198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 0x7fc3cc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc10d490 con 0x7fc3cc1017a0 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc10d310 con 0x7fc3cc103350 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3d0ff5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3cc10d610 con 0x7fc3cc102990 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52924/0 (socket says 172.21.15.38:52924) 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 -- 172.21.15.38:0/1132099303 learned_addr learned my addr 172.21.15.38:0/1132099303 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 0x7fc3cc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc102990 0x7fc3cc198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 -- 172.21.15.38:0/1132099303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc102990 msgr2=0x7fc3cc198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc102990 0x7fc3cc198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 -- 172.21.15.38:0/1132099303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 msgr2=0x7fc3cc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 0x7fc3cc19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.089+0000 7fc3ca59c640 1 -- 172.21.15.38:0/1132099303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3cc19f770 con 0x7fc3cc1017a0 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00b450 tx=0x7fc3bc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3bb7fe640 1 -- 172.21.15.38:0/1132099303 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc015020 con 0x7fc3cc1017a0 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3bb7fe640 1 -- 172.21.15.38:0/1132099303 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc011d70 con 0x7fc3cc1017a0 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3cc19fa60 con 0x7fc3cc1017a0 2024-04-22T00:01:56.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3bb7fe640 1 -- 172.21.15.38:0/1132099303 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00f340 con 0x7fc3cc1017a0 2024-04-22T00:01:56.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.090+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3cc19ffd0 con 0x7fc3cc1017a0 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.092+0000 7fc3bb7fe640 1 -- 172.21.15.38:0/1132099303 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc3bc018020 con 0x7fc3cc1017a0 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.092+0000 7fc3d3bdc740 1 Processor -- start 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.092+0000 7fc3d3bdc740 1 -- start start 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.092+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7e70 0x55b446eac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eaf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b446e07df0 con 0x55b446ea7480 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b446db0640 con 0x55b446ea7e70 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b446e03db0 con 0x55b446eacfe0 2024-04-22T00:01:56.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3bb7fe640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 0x7fc3b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52926/0 (socket says 172.21.15.38:52926) 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 -- 172.21.15.38:0/351802199 learned_addr learned my addr 172.21.15.38:0/351802199 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 0x7fc3b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3bb7fe640 1 -- 172.21.15.38:0/1132099303 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3bc0cedc0 con 0x7fc3cc1017a0 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7e70 0x55b446eac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eaf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 -- 172.21.15.38:0/351802199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 msgr2=0x55b446eaf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eaf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 -- 172.21.15.38:0/351802199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7e70 msgr2=0x55b446eac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7e70 0x55b446eac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 -- 172.21.15.38:0/351802199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b446b1c2d0 con 0x55b446ea7480 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4012e00 tx=0x7fc3c4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7037629ec7b132e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 0x7fc3b40a50d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4010c40 tx=0x7fc3c4017000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.093+0000 7fc3b8ff9640 1 -- 172.21.15.38:0/351802199 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4024030 con 0x55b446ea7480 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3b8ff9640 1 -- 172.21.15.38:0/351802199 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c4012620 con 0x55b446ea7480 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3b8ff9640 1 -- 172.21.15.38:0/351802199 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4003970 con 0x55b446ea7480 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/351802199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 msgr2=0x55b446ea78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4012e00 tx=0x7fc3c4010740 comp rx=0 tx=0).stop 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/351802199 shutdown_connections 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eaf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7e70 0x55b446eac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/351802199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7480 0x55b446ea78a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/351802199 >> 172.21.15.38:0/351802199 conn(0x55b446e9e5e0 msgr2=0x55b446e9e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/351802199 shutdown_connections 2024-04-22T00:01:56.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.094+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/351802199 wait complete. 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 -- start start 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 Processor -- start 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eb8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b446eaff30 con 0x55b446ea7e70 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b446dc3be0 con 0x55b446ea7480 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b446eb00b0 con 0x55b446eacfe0 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3d3bdc740 10 client.0 _ll_register_callbacks cb 0x55b446dc35d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52932/0 (socket says 172.21.15.38:52932) 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/1612292321 learned_addr learned my addr 172.21.15.38:0/1612292321 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eb8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/1612292321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 msgr2=0x55b446eb8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eb8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 msgr2=0x55b446eb1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.095+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/1612292321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b446be66b0 con 0x55b446ea7e70 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009900 tx=0x7fc3c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a2a45c9122326d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0005d50 con 0x55b446ea7e70 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3c0005450 con 0x55b446ea7e70 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3c4012620 con 0x55b446ea7e70 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3c004f050 con 0x55b446ea7e70 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 10 client.15498 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:56.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b446ebbed0 con 0x55b446ea7e70 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c4012620 con 0x55b446ea7e70 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc3c0003950 con 0x55b446ea7e70 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 10 client.15498 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 10 client.15498 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 20 client.15498 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446ebe570 0x55b446ebf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.096+0000 7fc3d3bdc740 4 client.15498 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.097+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b446e15330 con 0x55b446ebe570 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.097+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446ebe570 0x55b446ebf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.097+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446ebe570 0x55b446ebf9e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=891c976b43c1de1d server_cookie=870a536dd1af7c36 in_seq=0 out_seq=0 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.097+0000 7fc3b1ffb640 10 client.15498 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/1612292321 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b446e15330 con 0x55b446ebe570 2024-04-22T00:01:56.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3b1ffb640 10 client.15498 handle_command_reply: tid=0 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 client.15498 shutdown 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 msgr2=0x55b446eb1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009900 tx=0x7fc3c00042f0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 shutdown_connections 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446eb8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446ebe570 0x55b446ebf9e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb1e40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3b1ffb640 0 client.15498 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/1612292321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 >> 172.21.15.38:0/1612292321 conn(0x55b446e9e5e0 msgr2=0x55b446e9d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 shutdown_connections 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/1612292321 wait complete. 2024-04-22T00:01:56.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.098+0000 7fc3d3bdc740 20 client.15498 trim_cache size 0 max 16384 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 Processor -- start 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 -- start start 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446ea9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebdd20 con 0x55b446ea7e70 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebe3f0 con 0x55b446ea7480 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.116+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebeac0 con 0x55b446eacfe0 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52940/0 (socket says 172.21.15.38:52940) 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/842985445 learned_addr learned my addr 172.21.15.38:0/842985445 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446ea9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/842985445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 msgr2=0x55b446ea9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446ea9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/842985445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 msgr2=0x55b446eb9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/842985445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b446e9cc20 con 0x55b446ea7e70 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446ea9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009850 tx=0x7fc3c0009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e628ed51796a62a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3b3fff640 1 -- 172.21.15.38:0/842985445 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c00044e0 con 0x55b446ea7e70 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3b3fff640 1 -- 172.21.15.38:0/842985445 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c00035f0 con 0x55b446ea7e70 2024-04-22T00:01:56.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.117+0000 7fc3b3fff640 1 -- 172.21.15.38:0/842985445 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0005d50 con 0x55b446ea7e70 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/842985445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 msgr2=0x55b446eb8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009850 tx=0x7fc3c0009230 comp rx=0 tx=0).stop 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/842985445 shutdown_connections 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446eacfe0 0x55b446ea9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb8e40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/842985445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446ea7480 0x55b446eb9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/842985445 >> 172.21.15.38:0/842985445 conn(0x55b446e9e5e0 msgr2=0x55b446e9e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/842985445 shutdown_connections 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/842985445 wait complete. 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- start start 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 Processor -- start 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446ea7480 0x55b446eb4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 0x55b446eb8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebf860 con 0x55b446ea7e70 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebf190 con 0x55b446eacfe0 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b446ebff30 con 0x55b446ea7480 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3d3bdc740 10 client.0 _ll_register_callbacks cb 0x7fc3c0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.118+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52956/0 (socket says 172.21.15.38:52956) 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/421924606 learned_addr learned my addr 172.21.15.38:0/421924606 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 0x55b446eb8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/421924606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446ea7480 msgr2=0x55b446eb4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446ea7480 0x55b446eb4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 msgr2=0x55b446eb8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 0x55b446eb8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 -- 172.21.15.38:0/421924606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b446eb8670 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3cad9d640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 0x55b446eb8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3c9d9b640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009900 tx=0x7fc3c0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e4255a211340c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0057070 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3c00037a0 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3c00035f0 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3c005f050 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3d3bdc740 10 client.15504 fetch_fsmap learned FSMap version 27 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b446f24a30 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.119+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c00035f0 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fc3c005a030 con 0x55b446ea7e70 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 10 client.15504 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 10 client.15504 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T00:01:56.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 20 client.15504 populate_metadata read hostname 'smithi038' 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446f29320 0x55b446f28030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 4 client.15504 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "status"}) -- 0x55b446e0f680 con 0x55b446f29320 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446f29320 0x55b446f28030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3b1ffb640 10 client.15504 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3ca59c640 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446f29320 0x55b446f28030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86924c414351a35e server_cookie=93a12c78265aa5a9 in_seq=0 out_seq=0 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3b1ffb640 1 -- 172.21.15.38:0/421924606 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b446e0f680 con 0x55b446f29320 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3b1ffb640 10 client.15504 handle_command_reply: tid=0 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.120+0000 7fc3d3bdc740 1 client.15504 shutdown 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 msgr2=0x55b446eb50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009900 tx=0x7fc3c0004600 comp rx=0 tx=0).stop 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 shutdown_connections 2024-04-22T00:01:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b446eacfe0 0x55b446eb8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b446f29320 0x55b446f28030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b446ea7e70 0x55b446eb50b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 --2- 172.21.15.38:0/421924606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b446ea7480 0x55b446eb4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3b1ffb640 0 client.15504 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 >> 172.21.15.38:0/421924606 conn(0x55b446e9e5e0 msgr2=0x55b446ea9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 shutdown_connections 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 1 -- 172.21.15.38:0/421924606 wait complete. 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d3bdc740 20 client.15504 trim_cache size 0 max 16384 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:01:56.123 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "eab6f840-0039-11ef-bc93-c7b262605968", 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24500, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 139.34634054099999, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 27, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 91, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 145.32401667799999 2024-04-22T00:01:56.124 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 msgr2=0x7fc3b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 0x7fc3b40a50d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4010c40 tx=0x7fc3c4017000 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 msgr2=0x7fc3cc197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00b450 tx=0x7fc3bc00b910 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 shutdown_connections 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc3b40a2c10 0x7fc3b40a50d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3cc103350 0x7fc3cc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3cc102990 0x7fc3cc198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 --2- 172.21.15.38:0/1132099303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3cc1017a0 0x7fc3cc197b50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 >> 172.21.15.38:0/1132099303 conn(0x7fc3cc0fcd10 msgr2=0x7fc3cc105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 shutdown_connections 2024-04-22T00:01:56.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.121+0000 7fc3d0ff5640 1 -- 172.21.15.38:0/1132099303 wait complete. 2024-04-22T00:01:56.127 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T00:01:56.178 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T00:01:56.178 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T00:01:56.178 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T00:01:56.179 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T00:01:56.179 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T00:01:56.179 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:01:56.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-22T00:01:56.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- 172.21.15.38:0/3278035049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 msgr2=0x7fc88c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- 172.21.15.38:0/3278035049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c100ac0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc888009900 tx=0x7fc88802bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- 172.21.15.38:0/3278035049 shutdown_connections 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- 172.21.15.38:0/3278035049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c102250 0x7fc88c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- 172.21.15.38:0/3278035049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c101890 0x7fc88c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- 172.21.15.38:0/3278035049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c100ac0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- 172.21.15.38:0/3278035049 >> 172.21.15.38:0/3278035049 conn(0x7fc88c0fbc10 msgr2=0x7fc88c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- 172.21.15.38:0/3278035049 shutdown_connections 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- 172.21.15.38:0/3278035049 wait complete. 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 Processor -- start 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- start start 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c101890 0x7fc88c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 0x7fc88c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c10c370 con 0x7fc88c1006a0 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c10c1f0 con 0x7fc88c102250 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.290+0000 7fc89474f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c10c4f0 con 0x7fc88c101890 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52976/0 (socket says 172.21.15.38:52976) 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 -- 172.21.15.38:0/213547003 learned_addr learned my addr 172.21.15.38:0/213547003 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc891cea640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c101890 0x7fc88c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc892cec640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 0x7fc88c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 -- 172.21.15.38:0/213547003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c101890 msgr2=0x7fc88c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c101890 0x7fc88c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 -- 172.21.15.38:0/213547003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 msgr2=0x7fc88c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:56.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 0x7fc88c19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 -- 172.21.15.38:0/213547003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c19f710 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc892cec640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 0x7fc88c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8924eb640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc888036040 tx=0x7fc888005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888041070 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8880054e0 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c19f940 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888034440 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.291+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c19fe80 con 0x7fc88c1006a0 2024-04-22T00:01:56.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.292+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c107960 con 0x7fc88c1006a0 2024-04-22T00:01:56.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.293+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc888044070 con 0x7fc88c1006a0 2024-04-22T00:01:56.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.294+0000 7fc8837fe640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 0x7fc8780a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:56.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.294+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc8880f10a0 con 0x7fc88c1006a0 2024-04-22T00:01:56.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.294+0000 7fc891cea640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 0x7fc8780a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:56.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.294+0000 7fc891cea640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 0x7fc8780a5170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0696d0 tx=0x7fc87c009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:56.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.298+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc8880bdca0 con 0x7fc88c1006a0 2024-04-22T00:01:56.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:56.534+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fc88c062340 con 0x7fc88c1006a0 2024-04-22T00:01:57.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.415+0000 7fc8837fe640 1 -- 172.21.15.38:0/213547003 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fc8880bd5a0 con 0x7fc88c1006a0 2024-04-22T00:01:57.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 msgr2=0x7fc8780a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 0x7fc8780a5170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0696d0 tx=0x7fc87c009110 comp rx=0 tx=0).stop 2024-04-22T00:01:57.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 msgr2=0x7fc88c069960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc888036040 tx=0x7fc888005e50 comp rx=0 tx=0).stop 2024-04-22T00:01:57.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 shutdown_connections 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc8780a2cb0 0x7fc8780a5170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc88c102250 0x7fc88c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc88c101890 0x7fc88c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 --2- 172.21.15.38:0/213547003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc88c1006a0 0x7fc88c069960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.418+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 >> 172.21.15.38:0/213547003 conn(0x7fc88c0fbc10 msgr2=0x7fc88c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.419+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 shutdown_connections 2024-04-22T00:01:57.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.419+0000 7fc89474f640 1 -- 172.21.15.38:0/213547003 wait complete. 2024-04-22T00:01:57.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 -- 172.21.15.38:0/4086077806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc102220 msgr2=0x7f32fc104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 --2- 172.21.15.38:0/4086077806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc102220 0x7f32fc104610 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f32ec009900 tx=0x7f32ec02bc30 comp rx=0 tx=0).stop 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 -- 172.21.15.38:0/4086077806 shutdown_connections 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 --2- 172.21.15.38:0/4086077806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc107480 0x7f32fc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 --2- 172.21.15.38:0/4086077806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 0x7f32fc106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 --2- 172.21.15.38:0/4086077806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc102220 0x7f32fc104610 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 -- 172.21.15.38:0/4086077806 >> 172.21.15.38:0/4086077806 conn(0x7f32fc0fbc10 msgr2=0x7f32fc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 -- 172.21.15.38:0/4086077806 shutdown_connections 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.549+0000 7f330395b640 1 -- 172.21.15.38:0/4086077806 wait complete. 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 Processor -- start 2024-04-22T00:01:57.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 -- start start 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc102220 0x7f32fc199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 0x7f32fc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc10d490 con 0x7f32fc107480 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc10d310 con 0x7f32fc104b50 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f330395b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc10d610 con 0x7f32fc102220 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f3301ef8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f3301ef8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53022/0 (socket says 172.21.15.38:53022) 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f3301ef8640 1 -- 172.21.15.38:0/1680458874 learned_addr learned my addr 172.21.15.38:0/1680458874 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f33016f7640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc102220 0x7f32fc199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.550+0000 7f3300ef6640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 0x7f32fc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc102220 msgr2=0x7f32fc199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc102220 0x7f32fc199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 msgr2=0x7f32fc19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 0x7f32fc19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 -- 172.21.15.38:0/1680458874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32fc1a1850 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f3301ef8640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f32f400eb10 tx=0x7f32f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4004b20 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f4004cc0 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32fc1a1b40 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4010760 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.551+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc1a20b0 con 0x7f32fc107480 2024-04-22T00:01:57.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.552+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32fc108fd0 con 0x7f32fc107480 2024-04-22T00:01:57.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.553+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f32f4005000 con 0x7f32fc107480 2024-04-22T00:01:57.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.554+0000 7f32f27fc640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 0x7f32e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.554+0000 7f33016f7640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 msgr2=0x7f32e80a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:01:57.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.554+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f32f40cfe40 con 0x7f32fc107480 2024-04-22T00:01:57.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.554+0000 7f33016f7640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 0x7f32e80a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T00:01:57.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.558+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f32f409c9e0 con 0x7f32fc107480 2024-04-22T00:01:57.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-mon[27972]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:57.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T00:01:57.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setuser ceph since I am not root 2024-04-22T00:01:57.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: ignoring --setgroup ceph since I am not root 2024-04-22T00:01:57.708 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:57.609+0000 7feb0e9df200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T00:01:57.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.745+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f32fc02d170 con 0x7f32fc107480 2024-04-22T00:01:57.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.754+0000 7f32f27fc640 1 -- 172.21.15.38:0/1680458874 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7f32f409c2e0 con 0x7f32fc107480 2024-04-22T00:01:57.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.755+0000 7f33016f7640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 msgr2=0x7f32e80a51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:01:57.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.755+0000 7f33016f7640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 0x7f32e80a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 msgr2=0x7f32e80a51c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 0x7f32e80a51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 msgr2=0x7f32fc1a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f32f400eb10 tx=0x7f32f400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 shutdown_connections 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7f32e80a2d00 0x7f32e80a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32fc107480 0x7f32fc1a11b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32fc104b50 0x7f32fc19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 --2- 172.21.15.38:0/1680458874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32fc102220 0x7f32fc199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 >> 172.21.15.38:0/1680458874 conn(0x7f32fc0fbc10 msgr2=0x7f32fc1018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:57.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 shutdown_connections 2024-04-22T00:01:57.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.757+0000 7f330395b640 1 -- 172.21.15.38:0/1680458874 wait complete. 2024-04-22T00:01:57.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-mon[31543]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: ignoring --setuser ceph since I am not root 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: ignoring --setgroup ceph since I am not root 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-mgr[32970]: -- 172.21.15.111:0/2373081151 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55890c61a5a0 con 0x55890c5eec00 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:57.597+0000 7f38f1fc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T00:01:57.775 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:57.782+0000 7f38f1fc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-mon[31431]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: ignoring --setuser ceph since I am not root 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: ignoring --setgroup ceph since I am not root 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-mgr[33253]: -- 172.21.15.203:0/423345388 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5637943125a0 con 0x5637942e6c00 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:57.598+0000 7ff720402200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T00:01:57.776 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:57.782+0000 7ff720402200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:01:57.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- 172.21.15.38:0/3162667261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca4101370 msgr2=0x7fbca41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/3162667261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca4101370 0x7fbca41017f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009ca0 tx=0x7fbc9c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- 172.21.15.38:0/3162667261 shutdown_connections 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/3162667261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca4101370 0x7fbca41017f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/3162667261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca41009d0 0x7fbca4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/3162667261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 0x7fbca4100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- 172.21.15.38:0/3162667261 >> 172.21.15.38:0/3162667261 conn(0x7fbca40fbc10 msgr2=0x7fbca40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- 172.21.15.38:0/3162667261 shutdown_connections 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- 172.21.15.38:0/3162667261 wait complete. 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 Processor -- start 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.877+0000 7fbcac7d2640 1 -- start start 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 0x7fbca4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 0x7fbca41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410f610 con 0x7fbca41009d0 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410f490 con 0x7fbca4101370 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcac7d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410f790 con 0x7fbca4100070 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53048/0 (socket says 172.21.15.38:53048) 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 -- 172.21.15.38:0/927561970 learned_addr learned my addr 172.21.15.38:0/927561970 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcaa56e640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 0x7fbca4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcaad6f640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 0x7fbca41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 msgr2=0x7fbca4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 0x7fbca4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 msgr2=0x7fbca41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:57.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 0x7fbca41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 -- 172.21.15.38:0/927561970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca41a17c0 con 0x7fbca41009d0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbcaad6f640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 0x7fbca41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.878+0000 7fbca9d6d640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400e9f0 tx=0x7fbc9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.879+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9400cd80 con 0x7fbca41009d0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.879+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca41a1ab0 con 0x7fbca41009d0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.879+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc94010910 con 0x7fbca41009d0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.879+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc940188f0 con 0x7fbca41009d0 2024-04-22T00:01:57.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.879+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca41a2050 con 0x7fbca41009d0 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.880+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca410adc0 con 0x7fbca41009d0 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.882+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fbc94010ac0 con 0x7fbca41009d0 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.882+0000 7fbc9b7fe640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 0x7fbc900a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.882+0000 7fbcaa56e640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 msgr2=0x7fbc900a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.882+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc940cfb10 con 0x7fbca41009d0 2024-04-22T00:01:57.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.883+0000 7fbcaa56e640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 0x7fbc900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T00:01:57.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:57.885+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbc9409c0c0 con 0x7fbca41009d0 2024-04-22T00:01:58.029 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:57 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:57.884+0000 7ff720402200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T00:01:58.041 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:57.807+0000 7feb0e9df200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:01:58.042 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:57 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:57.923+0000 7feb0e9df200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T00:01:58.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.075+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fbca4062340 con 0x7fbca41009d0 2024-04-22T00:01:58.076 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:57 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:57.884+0000 7f38f1fc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T00:01:58.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.084+0000 7fbcaa56e640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 msgr2=0x7fbc900a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:01:58.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.084+0000 7fbcaa56e640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 0x7fbc900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T00:01:58.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.084+0000 7fbc9b7fe640 1 -- 172.21.15.38:0/927561970 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7fbc940a03c0 con 0x7fbca41009d0 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.086+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 msgr2=0x7fbc900a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.086+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 0x7fbc900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.086+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 msgr2=0x7fbca419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.086+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400e9f0 tx=0x7fbc9400eeb0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 shutdown_connections 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fbc900a2c80 0x7fbc900a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca4101370 0x7fbca41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbca41009d0 0x7fbca419a0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 --2- 172.21.15.38:0/927561970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbca4100070 0x7fbca4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 >> 172.21.15.38:0/927561970 conn(0x7fbca40fbc10 msgr2=0x7fbca40fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 shutdown_connections 2024-04-22T00:01:58.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:01:58.087+0000 7fbcac7d2640 1 -- 172.21.15.38:0/927561970 wait complete. 2024-04-22T00:01:58.090 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T00:01:58.114 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T00:01:58.114 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T00:01:58.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:01:58.115 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T00:01:58.130 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-22T00:01:58.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:01:58.131 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-22T00:01:58.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:01:58.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-04-22T00:01:58.293 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:58.265+0000 7feb0e9df200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T00:01:58.306 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-22T00:01:58.306 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T00:01:58.306 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T00:01:58.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T00:01:58.529 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:58.190+0000 7ff720402200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T00:01:58.529 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:58.276+0000 7ff720402200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-mon[31431]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1680458874' entity='client.admin' 2024-04-22T00:01:58.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:01:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/927561970' entity='client.admin' 2024-04-22T00:01:58.576 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:58.193+0000 7f38f1fc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T00:01:58.576 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:58.278+0000 7f38f1fc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T00:01:58.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T00:01:58.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-mon[31543]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T00:01:58.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1680458874' entity='client.admin' 2024-04-22T00:01:58.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:01:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/927561970' entity='client.admin' 2024-04-22T00:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:01:58.707 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:58.361+0000 7feb0e9df200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T00:01:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/213547003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T00:01:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-mon[27972]: mgrmap e19: x(active, since 9m), standbys: y, z 2024-04-22T00:01:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1680458874' entity='client.admin' 2024-04-22T00:01:58.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:01:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/927561970' entity='client.admin' 2024-04-22T00:01:59.779 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:59 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:59.458+0000 7ff720402200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T00:01:59.779 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:59 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:59.630+0000 7ff720402200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T00:01:59.782 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:59 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:59.450+0000 7f38f1fc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T00:01:59.782 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:59 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:59.621+0000 7f38f1fc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T00:01:59.782 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:01:59 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:01:59.789+0000 7f38f1fc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T00:01:59.808 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:59 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:59.670+0000 7feb0e9df200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T00:02:00.080 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:01:59 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:01:59.800+0000 7ff720402200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T00:02:00.134 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:01:59 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:01:59.851+0000 7feb0e9df200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T00:02:00.134 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:00.033+0000 7feb0e9df200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T00:02:00.326 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:00 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:00.076+0000 7f38f1fc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T00:02:00.326 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:00 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:00.167+0000 7f38f1fc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T00:02:00.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:00 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:00.086+0000 7ff720402200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T00:02:00.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:00 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:00.179+0000 7ff720402200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T00:02:00.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:00 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:00.387+0000 7ff720402200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T00:02:00.404 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:00.343+0000 7feb0e9df200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T00:02:00.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 -- 172.21.15.38:0/2237898771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c81083f0 msgr2=0x7ff4c81087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/2237898771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c81083f0 0x7ff4c81087d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400b0a0 tx=0x7ff4c400b3b0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 -- 172.21.15.38:0/2237898771 shutdown_connections 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/2237898771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102f10 0x7ff4c810f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/2237898771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c81083f0 0x7ff4c81087d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/2237898771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c8102550 0x7ff4c81029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 -- 172.21.15.38:0/2237898771 >> 172.21.15.38:0/2237898771 conn(0x7ff4c807a1c0 msgr2=0x7ff4c807a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 -- 172.21.15.38:0/2237898771 shutdown_connections 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.411+0000 7ff4cfc43700 1 -- 172.21.15.38:0/2237898771 wait complete. 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 Processor -- start 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 -- start start 2024-04-22T00:02:00.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102550 0x7ff4c8198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c8102f10 0x7ff4c81990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c81125f0 con 0x7ff4c8102f10 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c8112470 con 0x7ff4c81083f0 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.412+0000 7ff4cfc43700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c8112770 con 0x7ff4c8102550 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51776/0 (socket says 172.21.15.38:51776) 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 -- 172.21.15.38:0/3510079020 learned_addr learned my addr 172.21.15.38:0/3510079020 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4cd9df700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102550 0x7ff4c8198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102550 msgr2=0x7ff4c8198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102550 0x7ff4c8198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c8102f10 msgr2=0x7ff4c81990c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c8102f10 0x7ff4c81990c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 -- 172.21.15.38:0/3510079020 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c819db80 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4ce1e0700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000e970 tx=0x7ff4c000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0004a30 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0005500 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c819de70 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000cda0 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.413+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c81a5730 con 0x7ff4c81083f0 2024-04-22T00:02:00.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.414+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c810a840 con 0x7ff4c81083f0 2024-04-22T00:02:00.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.415+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7ff4c0044b40 con 0x7ff4c81083f0 2024-04-22T00:02:00.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.416+0000 7ff4beffd700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 0x7ff4b40a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:00.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.416+0000 7ff4cd9df700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 msgr2=0x7ff4b40a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:00.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.416+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff4c00cf8b0 con 0x7ff4c81083f0 2024-04-22T00:02:00.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.416+0000 7ff4cd9df700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 0x7ff4b40a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T00:02:00.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.420+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff4c009c450 con 0x7ff4c81083f0 2024-04-22T00:02:00.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.617+0000 7ff4cd9df700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 msgr2=0x7ff4b40a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:00.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.617+0000 7ff4cd9df700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 0x7ff4b40a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T00:02:00.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.678+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff4c8199ee0 con 0x7ff4c81083f0 2024-04-22T00:02:00.671 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:00.445+0000 7feb0e9df200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T00:02:00.671 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:00.659+0000 7feb0e9df200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T00:02:00.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.678+0000 7ff4beffd700 1 -- 172.21.15.38:0/3510079020 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7ff4c009bd50 con 0x7ff4c81083f0 2024-04-22T00:02:00.672 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-22T00:02:00.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.680+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 msgr2=0x7ff4b40a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:02:00.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.680+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 0x7ff4b40a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.680+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 msgr2=0x7ff4c819d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.680+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000e970 tx=0x7ff4c000ec80 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 shutdown_connections 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7ff4b40a2c70 0x7ff4b40a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4c81083f0 0x7ff4c819d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4c8102550 0x7ff4c8198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 --2- 172.21.15.38:0/3510079020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4c8102f10 0x7ff4c81990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 >> 172.21.15.38:0/3510079020 conn(0x7ff4c807a1c0 msgr2=0x7ff4c81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 shutdown_connections 2024-04-22T00:02:00.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:00.681+0000 7ff4cfc43700 1 -- 172.21.15.38:0/3510079020 wait complete. 2024-04-22T00:02:00.722 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:00 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:00.372+0000 7f38f1fc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T00:02:00.723 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:00 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:00.564+0000 7f38f1fc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:02:00.723 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:00 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:00.581+0000 7ff720402200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:02:00.957 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:00.870+0000 7feb0e9df200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T00:02:00.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:00 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3510079020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T00:02:01.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:00 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3510079020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T00:02:01.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:00 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3510079020' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T00:02:01.331 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T00:02:01.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T00:02:01.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-04-22T00:02:01.331 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T00:02:01.331 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:01.332 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T00:02:01.332 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T00:02:01.332 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T00:02:01.332 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T00:02:01.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:01.332 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T00:02:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 42021sec preferred_lft 42021sec 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-04-22T00:02:01.349 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-04-22T00:02:01.350 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T00:02:01.350 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T00:02:01.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:01.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:01.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:01.496 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T00:02:01.521 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-04-22T00:02:01.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:01.523 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-04-22T00:02:01.529 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:01 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:01.182+0000 7ff720402200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T00:02:01.573 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:01.160+0000 7f38f1fc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T00:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-04-22T00:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T00:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-22T00:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-22T00:02:01.580 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T00:02:01.580 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T00:02:01.580 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T00:02:01.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-22T00:02:01.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-22T00:02:01.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-22T00:02:01.581 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T00:02:01.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:01.678 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:01 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:01.525+0000 7feb0e9df200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T00:02:01.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:01.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:01.710 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T00:02:01.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:01.726 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-04-22T00:02:01.783 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T00:02:01.783 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T00:02:01.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T00:02:01.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T00:02:01.784 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T00:02:01.827 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:01.581+0000 7f38f1fc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T00:02:01.827 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:01.676+0000 7f38f1fc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T00:02:01.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:01.892 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:01 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:01.603+0000 7ff720402200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T00:02:01.892 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:01 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:01.700+0000 7ff720402200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T00:02:01.892 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:01 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:01.899+0000 7ff720402200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T00:02:01.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:01.906 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T00:02:01.906 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T00:02:01.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:02.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.021 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T00:02:02.021 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-04-22T00:02:02.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-04-22T00:02:02.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T00:02:02.022 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-04-22T00:02:02.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:02.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.100 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T00:02:02.100 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:02.116 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T00:02:02.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T00:02:02.116 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:02.173 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T00:02:02.173 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T00:02:02.173 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d 2024-04-22T00:02:02.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:02.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-22T00:02:02.235 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:01 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:01.995+0000 7feb0e9df200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T00:02:02.235 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:02.102+0000 7feb0e9df200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T00:02:02.257 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-22T00:02:02.257 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-22T00:02:02.257 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T00:02:02.326 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:01 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:01.875+0000 7f38f1fc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:3300,172.21.15.111:3300,172.21.15.203:3300" 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:3300,172.21.15.111:3300,172.21.15.203:3300:/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d 2024-04-22T00:02:02.391 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=eab6f840-0039-11ef-bc93-c7b262605968" will pass to kernel 2024-04-22T00:02:02.392 INFO:teuthology.orchestra.run.smithi038.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T00:02:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: You just mounted a file system that supports labels which does not 2024-04-22T00:02:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T00:02:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T00:02:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T00:02:02.394 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T00:02:02.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:02.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:02.467 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T00:02:02.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T00:02:02.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.567 INFO:teuthology.orchestra.run.smithi038.stdout:client25090 2024-04-22T00:02:02.568 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:02:02.568 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T00:02:02.585 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T00:02:02.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T00:02:02.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.683 INFO:teuthology.orchestra.run.smithi038.stdout:client25090 2024-04-22T00:02:02.683 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T00:02:02.707 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:02.322+0000 7feb0e9df200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T00:02:02.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:02.391+0000 7f38f1fc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T00:02:02.719 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:02.485+0000 7f38f1fc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T00:02:02.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T00:02:02.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.741 INFO:teuthology.orchestra.run.smithi038.stdout:client25090 2024-04-22T00:02:02.742 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/eab6f840-0039-11ef-bc93-c7b262605968.client25090/status') 2024-04-22T00:02:02.747 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:02 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:02.421+0000 7ff720402200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T00:02:02.747 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:02 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:02.515+0000 7ff720402200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T00:02:02.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T00:02:02.842 INFO:teuthology.orchestra.run.smithi038.stdout:instance: client.25090 (3)172.21.15.38:0/1961957892 2024-04-22T00:02:02.842 INFO:teuthology.orchestra.run.smithi038.stdout:blocklisted: false 2024-04-22T00:02:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2024-04-22T00:02:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2024-04-22T00:02:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:71 bytes copied, 2.4104e-05 s, 2.9 MB/s 2024-04-22T00:02:02.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T00:02:02.850 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T00:02:02.868 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T00:02:02.868 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:02:02.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 -- 172.21.15.38:0/856006320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c1006a0 msgr2=0x7faf8c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 --2- 172.21.15.38:0/856006320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c1006a0 0x7faf8c100ac0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7faf74009900 tx=0x7faf7402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 -- 172.21.15.38:0/856006320 shutdown_connections 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 --2- 172.21.15.38:0/856006320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c102250 0x7faf8c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 --2- 172.21.15.38:0/856006320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 0x7faf8c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 --2- 172.21.15.38:0/856006320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c1006a0 0x7faf8c100ac0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 -- 172.21.15.38:0/856006320 >> 172.21.15.38:0/856006320 conn(0x7faf8c0fbc10 msgr2=0x7faf8c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 -- 172.21.15.38:0/856006320 shutdown_connections 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.980+0000 7faf924b5640 1 -- 172.21.15.38:0/856006320 wait complete. 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 Processor -- start 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 -- start start 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c1006a0 0x7faf8c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 0x7faf8c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10c2f0 con 0x7faf8c102250 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10c170 con 0x7faf8c1006a0 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf924b5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faf8c10c470 con 0x7faf8c101890 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf90a52640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf90a52640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53064/0 (socket says 172.21.15.38:53064) 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.981+0000 7faf90a52640 1 -- 172.21.15.38:0/1912344686 learned_addr learned my addr 172.21.15.38:0/1912344686 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf8bfff640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c1006a0 0x7faf8c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf8b7fe640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 0x7faf8c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 msgr2=0x7faf8c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 0x7faf8c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c1006a0 msgr2=0x7faf8c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c1006a0 0x7faf8c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 -- 172.21.15.38:0/1912344686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8c19f680 con 0x7faf8c102250 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf90a52640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faf84009fd0 tx=0x7faf8400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84017070 con 0x7faf8c102250 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf84003e50 con 0x7faf8c102250 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8c19f970 con 0x7faf8c102250 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840106c0 con 0x7faf8c102250 2024-04-22T00:02:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.982+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf8c19fe70 con 0x7faf8c102250 2024-04-22T00:02:02.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.983+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf8c1a0040 con 0x7faf8c102250 2024-04-22T00:02:02.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.984+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7faf84003970 con 0x7faf8c102250 2024-04-22T00:02:02.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.985+0000 7faf897fa640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 0x7faf780a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.985+0000 7faf8bfff640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 msgr2=0x7faf780a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.985+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faf840cf120 con 0x7faf8c102250 2024-04-22T00:02:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.985+0000 7faf8bfff640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 0x7faf780a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T00:02:02.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:02.990+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faf8409b670 con 0x7faf8c102250 2024-04-22T00:02:03.029 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:02 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:02.754+0000 7ff720402200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T00:02:03.029 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:02 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:02.888+0000 7ff720402200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T00:02:03.029 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:02 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:02.984+0000 7ff720402200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T00:02:03.062 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:02.726+0000 7f38f1fc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T00:02:03.062 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:02.860+0000 7f38f1fc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T00:02:03.062 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:02.954+0000 7f38f1fc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T00:02:03.062 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:03 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:03.069+0000 7f38f1fc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T00:02:03.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.186+0000 7faf8bfff640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 msgr2=0x7faf780a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:03.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.186+0000 7faf8bfff640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 0x7faf780a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T00:02:03.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.215+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7faf8c1a0430 con 0x7faf8c102250 2024-04-22T00:02:03.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:02.887+0000 7feb0e9df200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T00:02:03.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:02 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:02.989+0000 7feb0e9df200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T00:02:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.267+0000 7faf897fa640 1 -- 172.21.15.38:0/1912344686 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7faf8409f970 con 0x7faf8c102250 2024-04-22T00:02:03.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 msgr2=0x7faf780a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 0x7faf780a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 msgr2=0x7faf8c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faf84009fd0 tx=0x7faf8400b260 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 shutdown_connections 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7faf780a2c80 0x7faf780a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faf8c102250 0x7faf8c19f010 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf8c101890 0x7faf8c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 --2- 172.21.15.38:0/1912344686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf8c1006a0 0x7faf8c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 >> 172.21.15.38:0/1912344686 conn(0x7faf8c0fbc10 msgr2=0x7faf8c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 shutdown_connections 2024-04-22T00:02:03.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.270+0000 7faf924b5640 1 -- 172.21.15.38:0/1912344686 wait complete. 2024-04-22T00:02:03.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T00:02:03.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:03 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T00:02:03.326 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:03 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:03.160+0000 7f38f1fc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 -- 172.21.15.38:0/2033151715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 msgr2=0x7fc8b4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2033151715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b4108ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009ca0 tx=0x7fc8ac02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 -- 172.21.15.38:0/2033151715 shutdown_connections 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2033151715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b4108ae0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2033151715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8b4101890 0x7fc8b4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2033151715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b41006a0 0x7fc8b4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 -- 172.21.15.38:0/2033151715 >> 172.21.15.38:0/2033151715 conn(0x7fc8b40fbc10 msgr2=0x7fc8b40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 -- 172.21.15.38:0/2033151715 shutdown_connections 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.388+0000 7fc8ba165640 1 -- 172.21.15.38:0/2033151715 wait complete. 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 Processor -- start 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 -- start start 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8b41006a0 0x7fc8b4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4101890 0x7fc8b419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b410d440 con 0x7fc8b4102250 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b410d2c0 con 0x7fc8b4101890 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8ba165640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b410d5c0 con 0x7fc8b41006a0 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8b37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8b41006a0 0x7fc8b4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:03.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8b3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b41a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53078/0 (socket says 172.21.15.38:53078) 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8b3fff640 1 -- 172.21.15.38:0/2898844238 learned_addr learned my addr 172.21.15.38:0/2898844238 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.389+0000 7fc8b2ffd640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4101890 0x7fc8b419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8b41006a0 msgr2=0x7fc8b4199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8b41006a0 0x7fc8b4199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4101890 msgr2=0x7fc8b419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8b4101890 0x7fc8b419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b41a17e0 con 0x7fc8b4102250 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b3fff640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b41a1170 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac02c700 tx=0x7fc8ac032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac039040 con 0x7fc8b4102250 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b41a1a70 con 0x7fc8b4102250 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac044070 con 0x7fc8b4102250 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0042a0 con 0x7fc8b4102250 2024-04-22T00:02:03.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.390+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b41a1fb0 con 0x7fc8b4102250 2024-04-22T00:02:03.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.391+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8b41a22a0 con 0x7fc8b4102250 2024-04-22T00:02:03.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.393+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fc8ac003970 con 0x7fc8b4102250 2024-04-22T00:02:03.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.393+0000 7fc8b0ff9640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:03.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.393+0000 7fc8b37fe640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 msgr2=0x7fc89c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:03.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.394+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc8ac0f16c0 con 0x7fc8b4102250 2024-04-22T00:02:03.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.394+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T00:02:03.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.398+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc8ac04e020 con 0x7fc8b4102250 2024-04-22T00:02:03.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T00:02:03.495 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:03.249+0000 7feb0e9df200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T00:02:03.496 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:03.399+0000 7feb0e9df200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T00:02:03.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:03 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T00:02:03.529 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:03 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:03.099+0000 7ff720402200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T00:02:03.529 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:03 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:03.188+0000 7ff720402200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T00:02:03.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.595+0000 7fc8b37fe640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 msgr2=0x7fc89c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:03.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.595+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T00:02:03.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.613+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc8b410abc0 con 0x7fc89c0a2c80 2024-04-22T00:02:03.957 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:03.502+0000 7feb0e9df200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T00:02:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:03.626+0000 7feb0e9df200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T00:02:03.958 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:03 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:03.725+0000 7feb0e9df200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T00:02:03.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.996+0000 7fc8b37fe640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 msgr2=0x7fc89c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:03.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:03.996+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T00:02:04.271 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:03 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:03.995+0000 7f38f1fc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T00:02:04.271 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y[32966]: 2024-04-22T00:02:04.213+0000 7f38f1fc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T00:02:04.271 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:04.004+0000 7ff720402200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T00:02:04.271 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z[33201]: 2024-04-22T00:02:04.219+0000 7ff720402200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T00:02:04.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.291+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 20) ==== 137850+0+0 (secure 0 0 0) 0x7fc8ac0ba260 con 0x7fc8b4102250 2024-04-22T00:02:04.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T00:02:04.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:02:04.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: Standby manager daemon y restarted 2024-04-22T00:02:04.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: Standby manager daemon y started 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: Standby manager daemon z restarted 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: Standby manager daemon z started 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T00:02:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:04 smithi203 ceph-mon[31431]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T00:02:04.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:02:04.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: Standby manager daemon y restarted 2024-04-22T00:02:04.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: Standby manager daemon y started 2024-04-22T00:02:04.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: Standby manager daemon z restarted 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: Standby manager daemon z started 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T00:02:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:04 smithi111 ceph-mon[31543]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1912344686' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T00:02:04.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:02:04.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: Standby manager daemon y restarted 2024-04-22T00:02:04.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: Standby manager daemon y started 2024-04-22T00:02:04.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.111:0/3450211971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: Standby manager daemon z restarted 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: Standby manager daemon z started 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T00:02:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-mon[27972]: from='mgr.? 172.21.15.203:0/3315534436' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T00:02:04.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.797+0000 7fc8b37fe640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 msgr2=0x7fc89c0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2340980806 2024-04-22T00:02:04.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.797+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T00:02:04.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.831+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7fc8ac0eee80 con 0x7fc8b4102250 2024-04-22T00:02:04.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.832+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 msgr2=0x7fc89c0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:02:04.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:04.832+0000 7fc8b0ff9640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2340980806,v1:172.21.15.38:6801/2340980806] conn(0x7fc89c0a2c80 0x7fc89c0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:04.957 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:04.586+0000 7feb0e9df200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T00:02:04.957 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:04 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:04.816+0000 7feb0e9df200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea6e3a4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea6e3a4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea6e3a4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea6e3a4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea6e3a4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea703a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea703a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea703a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea703a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.278 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.058+0000 7fea703a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: Active manager daemon x restarted 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: Activating manager daemon x 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: mgrmap e21: x(active, starting, since 0.0139406s), standbys: z, y 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T00:02:05.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T00:02:05.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T00:02:05.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: Manager daemon x is now available 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T00:02:05.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:05 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T00:02:05.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: Active manager daemon x restarted 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: Activating manager daemon x 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: mgrmap e21: x(active, starting, since 0.0139406s), standbys: z, y 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T00:02:05.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:02:05.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T00:02:05.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: Manager daemon x is now available 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T00:02:05.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:05 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: Active manager daemon x restarted 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: Activating manager daemon x 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: mgrmap e21: x(active, starting, since 0.0139406s), standbys: z, y 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:05.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:02:05.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T00:02:05.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T00:02:05.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: Manager daemon x is now available 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T00:02:05.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T00:02:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.839+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mon.0 v2:172.21.15.38:3300/0 9 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fc8ac032690 con 0x7fc8b4102250 2024-04-22T00:02:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.839+0000 7fc8b0ff9640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc89c0352d0 0x7fc89c0356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.840+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc8b410abc0 con 0x7fc89c0352d0 2024-04-22T00:02:05.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.842+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc89c0352d0 0x7fc89c0356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:05.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.845+0000 7fc8b37fe640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc89c0352d0 0x7fc89c0356b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8b41016f0 tx=0x7fc8a400b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:05.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.863+0000 7fc8b0ff9640 1 -- 172.21.15.38:0/2898844238 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc8b410abc0 con 0x7fc89c0352d0 2024-04-22T00:02:05.856 INFO:teuthology.orchestra.run.smithi038.stdout:{} 2024-04-22T00:02:05.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc89c0352d0 msgr2=0x7fc89c0356b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc89c0352d0 0x7fc89c0356b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc8b41016f0 tx=0x7fc8a400b390 comp rx=0 tx=0).stop 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 msgr2=0x7fc8b41a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 --2- 172.21.15.38:0/2898844238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc8b4102250 0x7fc8b41a1170 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac02c700 tx=0x7fc8ac032f20 comp rx=0 tx=0).stop 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8b3fff640 1 -- 172.21.15.38:0/2898844238 reap_dead start 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 shutdown_connections 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 >> 172.21.15.38:0/2898844238 conn(0x7fc8b40fbc10 msgr2=0x7fc8b41094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 shutdown_connections 2024-04-22T00:02:05.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:05.867+0000 7fc8ba165640 1 -- 172.21.15.38:0/2898844238 wait complete. 2024-04-22T00:02:05.874 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T00:02:05.925 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T00:02:05.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:02:06.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 -- 172.21.15.38:0/3642684749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc101790 msgr2=0x7f92fc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 --2- 172.21.15.38:0/3642684749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc101790 0x7f92fc101bb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009900 tx=0x7f92f402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:06.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 -- 172.21.15.38:0/3642684749 shutdown_connections 2024-04-22T00:02:06.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 --2- 172.21.15.38:0/3642684749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 --2- 172.21.15.38:0/3642684749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc102980 0x7f92fc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 --2- 172.21.15.38:0/3642684749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc101790 0x7f92fc101bb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.048+0000 7f9303d40640 1 -- 172.21.15.38:0/3642684749 >> 172.21.15.38:0/3642684749 conn(0x7f92fc0fcd20 msgr2=0x7f92fc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- 172.21.15.38:0/3642684749 shutdown_connections 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- 172.21.15.38:0/3642684749 wait complete. 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 Processor -- start 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- start start 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 0x7f92fc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc10d3e0 con 0x7f92fc102980 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc10d260 con 0x7f92fc103340 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f9303d40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc10d560 con 0x7f92fc101790 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f93012db640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f93012db640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39928/0 (socket says 172.21.15.38:39928) 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.049+0000 7f93012db640 1 -- 172.21.15.38:0/287422410 learned_addr learned my addr 172.21.15.38:0/287422410 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93022dd640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f9301adc640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 0x7f92fc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 -- 172.21.15.38:0/287422410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 msgr2=0x7f92fc197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 0x7f92fc197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 -- 172.21.15.38:0/287422410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 msgr2=0x7f92fc19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 -- 172.21.15.38:0/287422410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc19f620 con 0x7f92fc102980 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f9301adc640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 0x7f92fc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93022dd640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f93012db640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c920 tx=0x7f92ec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec007c70 con 0x7f92fc102980 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ec007e10 con 0x7f92fc102980 2024-04-22T00:02:06.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc19f910 con 0x7f92fc102980 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec018890 con 0x7f92fc102980 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.050+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc19feb0 con 0x7f92fc102980 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.051+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92fc108ab0 con 0x7f92fc102980 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.052+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f92ec018a50 con 0x7f92fc102980 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.053+0000 7f92f2ffd640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 0x7f92e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.053+0000 7f9301adc640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 0x7f92e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.053+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f92ec0cf450 con 0x7f92fc102980 2024-04-22T00:02:06.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.054+0000 7f9301adc640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 0x7f92e80a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f92f402c7b0 tx=0x7f92f4005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.057+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f92ec014030 con 0x7f92fc102980 2024-04-22T00:02:06.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.851+0000 7fea6b39e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:06.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.851+0000 7fea6b39e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:06.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.851+0000 7fea6b39e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:06.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.851+0000 7fea6b39e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:06.120 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:02:05 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:02:05.851+0000 7fea6b39e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:02:06.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.250+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92fc02d3c0 con 0x7f92fc102980 2024-04-22T00:02:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.251+0000 7f92f2ffd640 1 -- 172.21.15.38:0/287422410 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+25015 (secure 0 0 0) 0x7f92ec09c310 con 0x7f92fc102980 2024-04-22T00:02:06.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:06.246 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":92,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:02:04.818207+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:02:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 msgr2=0x7f92e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 0x7f92e80a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f92f402c7b0 tx=0x7f92f4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 msgr2=0x7f92fc197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c920 tx=0x7f92ec00cde0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 shutdown_connections 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f92e80a2a10 0x7f92e80a4ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92fc103340 0x7f92fc19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92fc102980 0x7f92fc197f40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 --2- 172.21.15.38:0/287422410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92fc101790 0x7f92fc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.253+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 >> 172.21.15.38:0/287422410 conn(0x7f92fc0fcd20 msgr2=0x7f92fc105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.254+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 shutdown_connections 2024-04-22T00:02:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.254+0000 7f9303d40640 1 -- 172.21.15.38:0/287422410 wait complete. 2024-04-22T00:02:06.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:02:06.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:02:06.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.381+0000 7f188bd27640 1 -- 172.21.15.38:0/1785825757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884107460 msgr2=0x7f18841098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.381+0000 7f188bd27640 1 --2- 172.21.15.38:0/1785825757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884107460 0x7f18841098b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f187c009900 tx=0x7f187c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- 172.21.15.38:0/1785825757 shutdown_connections 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- 172.21.15.38:0/1785825757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884107460 0x7f18841098b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- 172.21.15.38:0/1785825757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884104b30 0x7f1884106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- 172.21.15.38:0/1785825757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f18841045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- 172.21.15.38:0/1785825757 >> 172.21.15.38:0/1785825757 conn(0x7f18840fbbf0 msgr2=0x7f18840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- 172.21.15.38:0/1785825757 shutdown_connections 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- 172.21.15.38:0/1785825757 wait complete. 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 Processor -- start 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- start start 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f1884197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884107460 0x7f188419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f188410d420 con 0x7f1884104b30 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f188410d2a0 con 0x7f1884107460 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.382+0000 7f188bd27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f188410d5a0 con 0x7f1884102200 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39940/0 (socket says 172.21.15.38:39940) 2024-04-22T00:02:06.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 -- 172.21.15.38:0/2031779148 learned_addr learned my addr 172.21.15.38:0/2031779148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f1889ac3640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f1884197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f188a2c4640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884107460 0x7f188419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 -- 172.21.15.38:0/2031779148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 msgr2=0x7f1884197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f1884197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 -- 172.21.15.38:0/2031779148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884107460 msgr2=0x7f188419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884107460 0x7f188419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 -- 172.21.15.38:0/2031779148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188419f6b0 con 0x7f1884104b30 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f1889ac3640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f1884197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f18892c2640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f187400c970 tx=0x7f187400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874007bb0 con 0x7f1884104b30 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188419f9a0 con 0x7f1884104b30 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1874007d50 con 0x7f1884104b30 2024-04-22T00:02:06.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187400f450 con 0x7f1884104b30 2024-04-22T00:02:06.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.383+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188419fef0 con 0x7f1884104b30 2024-04-22T00:02:06.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.384+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18840fd6e0 con 0x7f1884104b30 2024-04-22T00:02:06.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.385+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f1874004f90 con 0x7f1884104b30 2024-04-22T00:02:06.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.386+0000 7f187affd640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 0x7f18700a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.386+0000 7f1889ac3640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 0x7f18700a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.386+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f18740ced70 con 0x7f1884104b30 2024-04-22T00:02:06.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.386+0000 7f1889ac3640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 0x7f18700a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1880005fd0 tx=0x7f1880009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.391+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1874014030 con 0x7f1884104b30 2024-04-22T00:02:06.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.577+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1884062340 con 0x7f1884104b30 2024-04-22T00:02:06.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.577+0000 7f187affd640 1 -- 172.21.15.38:0/2031779148 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f187409b4d0 con 0x7f1884104b30 2024-04-22T00:02:06.571 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:02:06.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 msgr2=0x7f18700a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 0x7f18700a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1880005fd0 tx=0x7f1880009500 comp rx=0 tx=0).stop 2024-04-22T00:02:06.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 msgr2=0x7f1884197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f187400c970 tx=0x7f187400ce30 comp rx=0 tx=0).stop 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 shutdown_connections 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18700a2990 0x7f18700a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1884107460 0x7f188419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1884104b30 0x7f1884197fd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 --2- 172.21.15.38:0/2031779148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1884102200 0x7f1884197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 >> 172.21.15.38:0/2031779148 conn(0x7f18840fbbf0 msgr2=0x7f1884102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 shutdown_connections 2024-04-22T00:02:06.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.580+0000 7f188bd27640 1 -- 172.21.15.38:0/2031779148 wait complete. 2024-04-22T00:02:06.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 -- 172.21.15.38:0/1702418857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 msgr2=0x7fcfc8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/1702418857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc8100ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009900 tx=0x7fcfbc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 -- 172.21.15.38:0/1702418857 shutdown_connections 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/1702418857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8102250 0x7fcfc8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/1702418857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8101890 0x7fcfc8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/1702418857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc8100ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 -- 172.21.15.38:0/1702418857 >> 172.21.15.38:0/1702418857 conn(0x7fcfc80fbc30 msgr2=0x7fcfc80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 -- 172.21.15.38:0/1702418857 shutdown_connections 2024-04-22T00:02:06.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.711+0000 7fcfcc84e640 1 -- 172.21.15.38:0/1702418857 wait complete. 2024-04-22T00:02:06.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 Processor -- start 2024-04-22T00:02:06.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 -- start start 2024-04-22T00:02:06.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 0x7fcfc819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8102250 0x7fcfc819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810c320 con 0x7fcfc81006a0 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810c1a0 con 0x7fcfc8101890 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfcc84e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc810c4a0 con 0x7fcfc8102250 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39960/0 (socket says 172.21.15.38:39960) 2024-04-22T00:02:06.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 0x7fcfc819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc659c640 1 -- 172.21.15.38:0/3575877984 learned_addr learned my addr 172.21.15.38:0/3575877984 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc6d9d640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8102250 0x7fcfc819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 -- 172.21.15.38:0/3575877984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8102250 msgr2=0x7fcfc819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8102250 0x7fcfc819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 -- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 msgr2=0x7fcfc806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc5d9b640 1 -- 172.21.15.38:0/3575877984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc819f7a0 con 0x7fcfc8101890 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.712+0000 7fcfc659c640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:06.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfc5d9b640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 0x7fcfc819c870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800e9f0 tx=0x7fcfb800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb800ccc0 con 0x7fcfc8101890 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb8005020 con 0x7fcfc8101890 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc819fa30 con 0x7fcfc8101890 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80049e0 con 0x7fcfc8101890 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.713+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc819ff80 con 0x7fcfc8101890 2024-04-22T00:02:06.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.714+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc8107ae0 con 0x7fcfc8101890 2024-04-22T00:02:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.715+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fcfb8004be0 con 0x7fcfc8101890 2024-04-22T00:02:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.716+0000 7fcfb77fe640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 0x7fcfb00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:06.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.716+0000 7fcfc659c640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 0x7fcfb00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.716+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcfb80cefd0 con 0x7fcfc8101890 2024-04-22T00:02:06.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.716+0000 7fcfc659c640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 0x7fcfb00a4ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc02c8e0 tx=0x7fcfbc005b50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:06.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.721+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcfb809bd80 con 0x7fcfc8101890 2024-04-22T00:02:06.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.848+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fcfb8005560 con 0x7fcfc8101890 2024-04-22T00:02:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.949+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcfc8062340 con 0x7fcfc8101890 2024-04-22T00:02:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.949+0000 7fcfb77fe640 1 -- 172.21.15.38:0/3575877984 <== mon.2 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fcfb80051d0 con 0x7fcfc8101890 2024-04-22T00:02:06.942 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 msgr2=0x7fcfb00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 0x7fcfb00a4ed0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc02c8e0 tx=0x7fcfbc005b50 comp rx=0 tx=0).stop 2024-04-22T00:02:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 msgr2=0x7fcfc819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 0x7fcfc819c870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800e9f0 tx=0x7fcfb800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 shutdown_connections 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcfb00a2a10 0x7fcfb00a4ed0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc8102250 0x7fcfc819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfc8101890 0x7fcfc819c870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 --2- 172.21.15.38:0/3575877984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfc81006a0 0x7fcfc806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 >> 172.21.15.38:0/3575877984 conn(0x7fcfc80fbc30 msgr2=0x7fcfc8104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 shutdown_connections 2024-04-22T00:02:06.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:06.951+0000 7fcfcc84e640 1 -- 172.21.15.38:0/3575877984 wait complete. 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: [22/Apr/2024:00:02:05] ENGINE Bus STARTING 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: mgrmap e22: x(active, since 1.02416s), standbys: z, y 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: [22/Apr/2024:00:02:05] ENGINE Serving on http://172.21.15.38:8765 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: [22/Apr/2024:00:02:05] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: [22/Apr/2024:00:02:05] ENGINE Bus STARTED 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/287422410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:06.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:06 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2031779148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:06.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 -- 172.21.15.38:0/844671988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4101870 msgr2=0x7f12b4101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 --2- 172.21.15.38:0/844671988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4101870 0x7f12b4101cf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009900 tx=0x7f12a802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 -- 172.21.15.38:0/844671988 shutdown_connections 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 --2- 172.21.15.38:0/844671988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4102230 0x7f12b4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 --2- 172.21.15.38:0/844671988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4101870 0x7f12b4101cf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 --2- 172.21.15.38:0/844671988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.094+0000 7f12b933e640 1 -- 172.21.15.38:0/844671988 >> 172.21.15.38:0/844671988 conn(0x7f12b40fbbf0 msgr2=0x7f12b40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.095+0000 7f12b933e640 1 -- 172.21.15.38:0/844671988 shutdown_connections 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.095+0000 7f12b933e640 1 -- 172.21.15.38:0/844671988 wait complete. 2024-04-22T00:02:07.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.095+0000 7f12b933e640 1 Processor -- start 2024-04-22T00:02:07.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 -- start start 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4101870 0x7f12b419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f12b410c2e0 con 0x7f12b4102230 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f12b410c160 con 0x7f12b4100680 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b933e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12b410c460 con 0x7f12b4101870 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39982/0 (socket says 172.21.15.38:39982) 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b27fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4101870 0x7f12b419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 -- 172.21.15.38:0/709100596 learned_addr learned my addr 172.21.15.38:0/709100596 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:07.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b2ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:50724/0 (socket says 172.21.15.38:50724) 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 -- 172.21.15.38:0/709100596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4101870 msgr2=0x7f12b419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4101870 0x7f12b419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 -- 172.21.15.38:0/709100596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 msgr2=0x7f12b4069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 -- 172.21.15.38:0/709100596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b419f730 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.096+0000 7f12b37fe640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00eb10 tx=0x7f12ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.097+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac004b20 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.097+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12ac004cc0 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.097+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac010760 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.097+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b419f9c0 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.097+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b419fee0 con 0x7f12b4102230 2024-04-22T00:02:07.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.098+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b4107a20 con 0x7f12b4102230 2024-04-22T00:02:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.099+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f12ac005020 con 0x7f12b4102230 2024-04-22T00:02:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.100+0000 7f129bfff640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 0x7f129c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.100+0000 7f12b2ffd640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 0x7f129c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.100+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f12ac0cfec0 con 0x7f12b4102230 2024-04-22T00:02:07.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.101+0000 7f12b2ffd640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 0x7f129c0a52d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f12a40095f0 tx=0x7f12a4005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.104+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f12ac09c5d0 con 0x7f12b4102230 2024-04-22T00:02:07.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: [22/Apr/2024:00:02:05] ENGINE Bus STARTING 2024-04-22T00:02:07.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: mgrmap e22: x(active, since 1.02416s), standbys: z, y 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: [22/Apr/2024:00:02:05] ENGINE Serving on http://172.21.15.38:8765 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: [22/Apr/2024:00:02:05] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: [22/Apr/2024:00:02:05] ENGINE Bus STARTED 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/287422410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:07.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:06 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2031779148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:07.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.294+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12b402d370 con 0x7f12b4102230 2024-04-22T00:02:07.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.294+0000 7f129bfff640 1 -- 172.21.15.38:0/709100596 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f12ac09bed0 con 0x7f12b4102230 2024-04-22T00:02:07.287 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:02:07.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 msgr2=0x7f129c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 0x7f129c0a52d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f12a40095f0 tx=0x7f12a4005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 msgr2=0x7f12b419ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00eb10 tx=0x7f12ac00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 shutdown_connections 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f129c0a2e10 0x7f129c0a52d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f12b4102230 0x7f12b419ee20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12b4101870 0x7f12b419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 --2- 172.21.15.38:0/709100596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b4100680 0x7f12b4069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 >> 172.21.15.38:0/709100596 conn(0x7f12b40fbbf0 msgr2=0x7f12b4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 shutdown_connections 2024-04-22T00:02:07.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.297+0000 7f12b933e640 1 -- 172.21.15.38:0/709100596 wait complete. 2024-04-22T00:02:07.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:02:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: [22/Apr/2024:00:02:05] ENGINE Bus STARTING 2024-04-22T00:02:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: mgrmap e22: x(active, since 1.02416s), standbys: z, y 2024-04-22T00:02:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: [22/Apr/2024:00:02:05] ENGINE Serving on http://172.21.15.38:8765 2024-04-22T00:02:07.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: [22/Apr/2024:00:02:05] ENGINE Serving on https://172.21.15.38:7150 2024-04-22T00:02:07.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: [22/Apr/2024:00:02:05] ENGINE Bus STARTED 2024-04-22T00:02:07.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/287422410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:07.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:06 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2031779148' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:07.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 -- 172.21.15.38:0/1448673004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060101870 msgr2=0x7f7060101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 --2- 172.21.15.38:0/1448673004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060101870 0x7f7060101cf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7054009900 tx=0x7f705402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 -- 172.21.15.38:0/1448673004 shutdown_connections 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 --2- 172.21.15.38:0/1448673004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 0x7f7060108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 --2- 172.21.15.38:0/1448673004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060101870 0x7f7060101cf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 --2- 172.21.15.38:0/1448673004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060100680 0x7f7060100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 -- 172.21.15.38:0/1448673004 >> 172.21.15.38:0/1448673004 conn(0x7f70600fbbf0 msgr2=0x7f70600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 -- 172.21.15.38:0/1448673004 shutdown_connections 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.417+0000 7f7065cc2640 1 -- 172.21.15.38:0/1448673004 wait complete. 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 Processor -- start 2024-04-22T00:02:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 -- start start 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060101870 0x7f70601980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 0x7f706019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f706010c300 con 0x7f7060100680 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f706010c180 con 0x7f7060102230 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.418+0000 7f7065cc2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f706010c480 con 0x7f7060101870 2024-04-22T00:02:07.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40014/0 (socket says 172.21.15.38:40014) 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 -- 172.21.15.38:0/406161962 learned_addr learned my addr 172.21.15.38:0/406161962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705effd640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060101870 0x7f70601980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705ffff640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 0x7f706019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 -- 172.21.15.38:0/406161962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060101870 msgr2=0x7f70601980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060101870 0x7f70601980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 -- 172.21.15.38:0/406161962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 msgr2=0x7f706019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 0x7f706019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 -- 172.21.15.38:0/406161962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706019f7b0 con 0x7f7060100680 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705f7fe640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f705000e9c0 tx=0x7f705000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705000ccc0 con 0x7f7060100680 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706019faa0 con 0x7f7060100680 2024-04-22T00:02:07.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70500035f0 con 0x7f7060100680 2024-04-22T00:02:07.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.419+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706019ffc0 con 0x7f7060100680 2024-04-22T00:02:07.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.420+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705000ce60 con 0x7f7060100680 2024-04-22T00:02:07.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.421+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7060068560 con 0x7f7060100680 2024-04-22T00:02:07.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.422+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f70500037a0 con 0x7f7060100680 2024-04-22T00:02:07.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.422+0000 7f705cff9640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 0x7f70480a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.423+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f70500cfc20 con 0x7f7060100680 2024-04-22T00:02:07.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.423+0000 7f705effd640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 0x7f70480a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.423+0000 7f705effd640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 0x7f70480a52d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7054009230 tx=0x7f7054005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.427+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f705009c390 con 0x7f7060100680 2024-04-22T00:02:07.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.619+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f70601a0460 con 0x7f7060100680 2024-04-22T00:02:07.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.619+0000 7f705cff9640 1 -- 172.21.15.38:0/406161962 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f705009bc90 con 0x7f7060100680 2024-04-22T00:02:07.612 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:07.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 msgr2=0x7f70480a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 0x7f70480a52d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7054009230 tx=0x7f7054005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 msgr2=0x7f7060197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f705000e9c0 tx=0x7f705000edc0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 shutdown_connections 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f70480a2e10 0x7f70480a52d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7060102230 0x7f706019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7060101870 0x7f70601980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 --2- 172.21.15.38:0/406161962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7060100680 0x7f7060197b90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 >> 172.21.15.38:0/406161962 conn(0x7f70600fbbf0 msgr2=0x7f70600fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 shutdown_connections 2024-04-22T00:02:07.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.622+0000 7f7065cc2640 1 -- 172.21.15.38:0/406161962 wait complete. 2024-04-22T00:02:07.625 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83459]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:07.625 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:07.625 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83459]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:07.625 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83459]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:07.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:02:07.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- 172.21.15.38:0/933253537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 msgr2=0x7fdba4106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 --2- 172.21.15.38:0/933253537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba4106f30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb94009900 tx=0x7fdb9402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:07.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- 172.21.15.38:0/933253537 shutdown_connections 2024-04-22T00:02:07.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 --2- 172.21.15.38:0/933253537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4107470 0x7fdba41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 --2- 172.21.15.38:0/933253537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba4106f30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 --2- 172.21.15.38:0/933253537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4102210 0x7fdba4104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- 172.21.15.38:0/933253537 >> 172.21.15.38:0/933253537 conn(0x7fdba40fbc20 msgr2=0x7fdba40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- 172.21.15.38:0/933253537 shutdown_connections 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- 172.21.15.38:0/933253537 wait complete. 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 Processor -- start 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.753+0000 7fdbac113640 1 -- start start 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 0x7fdba41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdba410d420 con 0x7fdba4102210 2024-04-22T00:02:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdba410d2a0 con 0x7fdba4104b40 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbac113640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdba410d5a0 con 0x7fdba4107470 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40038/0 (socket says 172.21.15.38:40038) 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 -- 172.21.15.38:0/4253966323 learned_addr learned my addr 172.21.15.38:0/4253966323 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba96ae640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbaa6b0640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 0x7fdba41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 -- 172.21.15.38:0/4253966323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 msgr2=0x7fdba41a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 0x7fdba41a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 -- 172.21.15.38:0/4253966323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 msgr2=0x7fdba419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 -- 172.21.15.38:0/4253966323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba41a1900 con 0x7fdba4102210 2024-04-22T00:02:07.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdbaa6b0640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 0x7fdba41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba96ae640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.754+0000 7fdba9eaf640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdba000e9f0 tx=0x7fdba000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.755+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba000cd80 con 0x7fdba4102210 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.755+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba0005500 con 0x7fdba4102210 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.755+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba41a1bf0 con 0x7fdba4102210 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.755+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0010690 con 0x7fdba4102210 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.755+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba41a20f0 con 0x7fdba4102210 2024-04-22T00:02:07.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.756+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba4108fc0 con 0x7fdba4102210 2024-04-22T00:02:07.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.758+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fdba00046c0 con 0x7fdba4102210 2024-04-22T00:02:07.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.759+0000 7fdb9affd640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 0x7fdb900a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:07.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.759+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fdba00cfef0 con 0x7fdba4102210 2024-04-22T00:02:07.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.759+0000 7fdba96ae640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 0x7fdb900a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:07.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.760+0000 7fdba96ae640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 0x7fdb900a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:07.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.764+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdba009c600 con 0x7fdba4102210 2024-04-22T00:02:07.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.957+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdba402d400 con 0x7fdba4102210 2024-04-22T00:02:07.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.958+0000 7fdb9affd640 1 -- 172.21.15.38:0/4253966323 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+25015 (secure 0 0 0) 0x7fdba009bf00 con 0x7fdba4102210 2024-04-22T00:02:07.951 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:07.951 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":92,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:02:04.818207+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:02:07.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 msgr2=0x7fdb900a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 0x7fdb900a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000c00 tx=0x7fdb94005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 msgr2=0x7fdba4199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdba000e9f0 tx=0x7fdba000edf0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 shutdown_connections 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb900a2d90 0x7fdb900a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdba4107470 0x7fdba41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba4104b40 0x7fdba419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 --2- 172.21.15.38:0/4253966323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdba4102210 0x7fdba4199cf0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 >> 172.21.15.38:0/4253966323 conn(0x7fdba40fbc20 msgr2=0x7fdba4101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:07.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 shutdown_connections 2024-04-22T00:02:07.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:07.961+0000 7fdbac113640 1 -- 172.21.15.38:0/4253966323 wait complete. 2024-04-22T00:02:07.957 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83620]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:07.957 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:07.957 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83620]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:07.957 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:02:07 smithi038 sudo[83620]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3575877984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/709100596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:07.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:07 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/406161962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:07.965 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:02:07.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:02:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T00:02:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3575877984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/709100596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:07 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/406161962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 -- 172.21.15.38:0/12000992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8100680 msgr2=0x7ff1d8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 --2- 172.21.15.38:0/12000992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8100680 0x7ff1d8100aa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009900 tx=0x7ff1d402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 -- 172.21.15.38:0/12000992 shutdown_connections 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 --2- 172.21.15.38:0/12000992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8102230 0x7ff1d8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 --2- 172.21.15.38:0/12000992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 --2- 172.21.15.38:0/12000992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8100680 0x7ff1d8100aa0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 -- 172.21.15.38:0/12000992 >> 172.21.15.38:0/12000992 conn(0x7ff1d80fbbf0 msgr2=0x7ff1d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 -- 172.21.15.38:0/12000992 shutdown_connections 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.084+0000 7ff1df927640 1 -- 172.21.15.38:0/12000992 wait complete. 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 Processor -- start 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 -- start start 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8100680 0x7ff1d8104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c2d0 con 0x7ff1d8102230 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c150 con 0x7ff1d8101870 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1df927640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810c450 con 0x7ff1d8100680 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1ddec4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1ddec4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40056/0 (socket says 172.21.15.38:40056) 2024-04-22T00:02:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1dd6c3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8100680 0x7ff1d8104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1dcec2640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.085+0000 7ff1ddec4640 1 -- 172.21.15.38:0/3969291755 learned_addr learned my addr 172.21.15.38:0/3969291755 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 -- 172.21.15.38:0/3969291755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8100680 msgr2=0x7ff1d8104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8100680 0x7ff1d8104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 -- 172.21.15.38:0/3969291755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 msgr2=0x7ff1d8102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 -- 172.21.15.38:0/3969291755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d8103a20 con 0x7ff1d8102230 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1dcec2640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ddec4640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009fd0 tx=0x7ff1d000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0017070 con 0x7ff1d8102230 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d0003e50 con 0x7ff1d8102230 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d81a1c80 con 0x7ff1d8102230 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00106c0 con 0x7ff1d8102230 2024-04-22T00:02:08.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.086+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d81a2180 con 0x7ff1d8102230 2024-04-22T00:02:08.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.087+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d81a2350 con 0x7ff1d8102230 2024-04-22T00:02:08.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.088+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7ff1d0003970 con 0x7ff1d8102230 2024-04-22T00:02:08.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.089+0000 7ff1ca7fc640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 0x7ff1c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.089+0000 7ff1dd6c3640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 0x7ff1c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.089+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff1d00cf590 con 0x7ff1d8102230 2024-04-22T00:02:08.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.090+0000 7ff1dd6c3640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 0x7ff1c40a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4000c00 tx=0x7ff1d4005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.094+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff1d009b6b0 con 0x7ff1d8102230 2024-04-22T00:02:08.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:08.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-22T00:02:08.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3575877984' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/709100596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:07 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/406161962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.281+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1d81a2520 con 0x7ff1d8102230 2024-04-22T00:02:08.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.282+0000 7ff1ca7fc640 1 -- 172.21.15.38:0/3969291755 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7ff1d009f9b0 con 0x7ff1d8102230 2024-04-22T00:02:08.275 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 msgr2=0x7ff1c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 0x7ff1c40a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4000c00 tx=0x7ff1d4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 msgr2=0x7ff1d8103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009fd0 tx=0x7ff1d000b260 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 shutdown_connections 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1c40a2d90 0x7ff1c40a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1d8102230 0x7ff1d8103190 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1d8101870 0x7ff1d8102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 --2- 172.21.15.38:0/3969291755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d8100680 0x7ff1d8104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 >> 172.21.15.38:0/3969291755 conn(0x7ff1d80fbbf0 msgr2=0x7ff1d8068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 shutdown_connections 2024-04-22T00:02:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.284+0000 7ff1df927640 1 -- 172.21.15.38:0/3969291755 wait complete. 2024-04-22T00:02:08.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 -- 172.21.15.38:0/1698139652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4101790 msgr2=0x7f36d4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 --2- 172.21.15.38:0/1698139652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4101790 0x7f36d4101bb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009900 tx=0x7f36c402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 -- 172.21.15.38:0/1698139652 shutdown_connections 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 --2- 172.21.15.38:0/1698139652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4103340 0x7f36d4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 --2- 172.21.15.38:0/1698139652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4102980 0x7f36d4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 --2- 172.21.15.38:0/1698139652 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4101790 0x7f36d4101bb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 -- 172.21.15.38:0/1698139652 >> 172.21.15.38:0/1698139652 conn(0x7f36d40fcd20 msgr2=0x7f36d40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 -- 172.21.15.38:0/1698139652 shutdown_connections 2024-04-22T00:02:08.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.405+0000 7f36d8c08640 1 -- 172.21.15.38:0/1698139652 wait complete. 2024-04-22T00:02:08.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 Processor -- start 2024-04-22T00:02:08.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 -- start start 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4101790 0x7f36d4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4102980 0x7f36d4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f36d410d3e0 con 0x7f36d4103340 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36d410d260 con 0x7f36d4102980 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d8c08640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36d410d560 con 0x7f36d4101790 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40082/0 (socket says 172.21.15.38:40082) 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 -- 172.21.15.38:0/2942935124 learned_addr learned my addr 172.21.15.38:0/2942935124 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36cbfff640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4102980 0x7f36d4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d259c640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4101790 0x7f36d4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 -- 172.21.15.38:0/2942935124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4101790 msgr2=0x7f36d4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4101790 0x7f36d4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 -- 172.21.15.38:0/2942935124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4102980 msgr2=0x7f36d4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4102980 0x7f36d4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.406+0000 7f36d2d9d640 1 -- 172.21.15.38:0/2942935124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d419f6c0 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36d2d9d640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00eb10 tx=0x7f36cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc004b20 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36cc004cc0 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d419f9b0 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc010760 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.407+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d419feb0 con 0x7f36d4103340 2024-04-22T00:02:08.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.408+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d4108b80 con 0x7f36d4103340 2024-04-22T00:02:08.403 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63622]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:08.403 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63622]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:08.403 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63622]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:08.403 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63622]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:08.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.410+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f36cc005000 con 0x7f36d4103340 2024-04-22T00:02:08.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.410+0000 7f36cb7fe640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 0x7f36c00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.411+0000 7f36d259c640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 0x7f36c00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.411+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f36cc0d00d0 con 0x7f36d4103340 2024-04-22T00:02:08.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.411+0000 7f36d259c640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 0x7f36c00a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f36c4000c00 tx=0x7f36c4005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.414+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f36cc0536f0 con 0x7f36d4103340 2024-04-22T00:02:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.618+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f36d402d320 con 0x7f36d4103340 2024-04-22T00:02:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.619+0000 7f36cb7fe640 1 -- 172.21.15.38:0/2942935124 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f36cc09c840 con 0x7f36d4103340 2024-04-22T00:02:08.611 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:08.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 msgr2=0x7f36c00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 0x7f36c00a5250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f36c4000c00 tx=0x7f36c4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 msgr2=0x7f36d419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f36cc00eb10 tx=0x7f36cc00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 shutdown_connections 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36c00a2d90 0x7f36c00a5250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36d4103340 0x7f36d419f050 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36d4102980 0x7f36d4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 --2- 172.21.15.38:0/2942935124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d4101790 0x7f36d4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 >> 172.21.15.38:0/2942935124 conn(0x7f36d40fcd20 msgr2=0x7f36d4105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 shutdown_connections 2024-04-22T00:02:08.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.621+0000 7f36d8c08640 1 -- 172.21.15.38:0/2942935124 wait complete. 2024-04-22T00:02:08.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:02:08.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 -- 172.21.15.38:0/2866364638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 msgr2=0x7f0a6c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 --2- 172.21.15.38:0/2866364638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c100aa0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009900 tx=0x7f0a6002bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 -- 172.21.15.38:0/2866364638 shutdown_connections 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 --2- 172.21.15.38:0/2866364638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 0x7f0a6c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 --2- 172.21.15.38:0/2866364638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 0x7f0a6c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 --2- 172.21.15.38:0/2866364638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c100aa0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 -- 172.21.15.38:0/2866364638 >> 172.21.15.38:0/2866364638 conn(0x7f0a6c0fbbf0 msgr2=0x7f0a6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 -- 172.21.15.38:0/2866364638 shutdown_connections 2024-04-22T00:02:08.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.743+0000 7f0a730c9640 1 -- 172.21.15.38:0/2866364638 wait complete. 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 Processor -- start 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 -- start start 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 0x7f0a6c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 0x7f0a6c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c10c2d0 con 0x7f0a6c100680 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c10c150 con 0x7f0a6c101870 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a730c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6c10c450 con 0x7f0a6c102230 2024-04-22T00:02:08.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a70e65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a70e65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40086/0 (socket says 172.21.15.38:40086) 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a70e65640 1 -- 172.21.15.38:0/1674172257 learned_addr learned my addr 172.21.15.38:0/1674172257 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a71666640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 0x7f0a6c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.744+0000 7f0a6bfff640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 0x7f0a6c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 -- 172.21.15.38:0/1674172257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 msgr2=0x7f0a6c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 0x7f0a6c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 -- 172.21.15.38:0/1674172257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 msgr2=0x7f0a6c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 0x7f0a6c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 -- 172.21.15.38:0/1674172257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6c103a20 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a70e65640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000c00 tx=0x7f0a6002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60041070 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a60004450 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6c1a1c20 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60003df0 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.745+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6c1a2170 con 0x7f0a6c100680 2024-04-22T00:02:08.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.746+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a6c1a2340 con 0x7f0a6c100680 2024-04-22T00:02:08.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.748+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f0a60044070 con 0x7f0a6c100680 2024-04-22T00:02:08.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.748+0000 7f0a69ffb640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 0x7f0a5c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:08.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.749+0000 7f0a6bfff640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 0x7f0a5c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:08.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.749+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0a600f1420 con 0x7f0a6c100680 2024-04-22T00:02:08.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.749+0000 7f0a6bfff640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 0x7f0a5c0a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c104380 tx=0x7f0a58005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:08.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.753+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0a600bddd0 con 0x7f0a6c100680 2024-04-22T00:02:08.826 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63705]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:08.826 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63705]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:08.826 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63705]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:08.827 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:02:08 smithi111 sudo[63705]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:08.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.945+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a6c1a2510 con 0x7f0a6c100680 2024-04-22T00:02:08.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.945+0000 7f0a69ffb640 1 -- 172.21.15.38:0/1674172257 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f0a600bd6d0 con 0x7f0a6c100680 2024-04-22T00:02:08.938 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:02:08.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.947+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 msgr2=0x7f0a5c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.947+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 0x7f0a5c0a5250 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c104380 tx=0x7f0a58005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 msgr2=0x7f0a6c104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0a60000c00 tx=0x7f0a6002c640 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 shutdown_connections 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0a5c0a2d90 0x7f0a5c0a5250 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a6c102230 0x7f0a6c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a6c101870 0x7f0a6c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 --2- 172.21.15.38:0/1674172257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a6c100680 0x7f0a6c104610 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 >> 172.21.15.38:0/1674172257 conn(0x7f0a6c0fbbf0 msgr2=0x7f0a6c068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 shutdown_connections 2024-04-22T00:02:08.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:08.948+0000 7f0a730c9640 1 -- 172.21.15.38:0/1674172257 wait complete. 2024-04-22T00:02:08.945 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[64963]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:08.945 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[64963]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:08.945 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[64963]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:08.945 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[64963]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:08.945 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.946 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4253966323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:08.946 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.946 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.946 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.947 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3969291755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:02:08.948 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:08.949 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:08 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2942935124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:08.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 -- 172.21.15.38:0/403428403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101280 msgr2=0x7fc4f8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/403428403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101280 0x7fc4f8108b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009ca0 tx=0x7fc4f002c140 comp rx=0 tx=0).stop 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 -- 172.21.15.38:0/403428403 shutdown_connections 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/403428403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101280 0x7fc4f8108b20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/403428403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f81008c0 0x7fc4f8100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/403428403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f8101d00 0x7fc4f8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 -- 172.21.15.38:0/403428403 >> 172.21.15.38:0/403428403 conn(0x7fc4f80fb370 msgr2=0x7fc4f80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 -- 172.21.15.38:0/403428403 shutdown_connections 2024-04-22T00:02:09.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.068+0000 7fc4fc90f640 1 -- 172.21.15.38:0/403428403 wait complete. 2024-04-22T00:02:09.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 Processor -- start 2024-04-22T00:02:09.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 -- start start 2024-04-22T00:02:09.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 0x7fc4f8199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 0x7fc4f819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810c2f0 con 0x7fc4f8101d00 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810c170 con 0x7fc4f8101280 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4fc90f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810c470 con 0x7fc4f81008c0 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4f659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 0x7fc4f8199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40104/0 (socket says 172.21.15.38:40104) 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4f6d9d640 1 -- 172.21.15.38:0/3502534689 learned_addr learned my addr 172.21.15.38:0/3502534689 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.069+0000 7fc4f5d9b640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 0x7fc4f819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 -- 172.21.15.38:0/3502534689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 msgr2=0x7fc4f8199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 0x7fc4f8199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 -- 172.21.15.38:0/3502534689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 msgr2=0x7fc4f819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 0x7fc4f819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 -- 172.21.15.38:0/3502534689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f81a18f0 con 0x7fc4f8101d00 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f5d9b640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 0x7fc4f819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f659c640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 0x7fc4f8199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4f6d9d640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0005cc0 tx=0x7fc4f0035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0039040 con 0x7fc4f8101d00 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f81a1b80 con 0x7fc4f8101d00 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f81a20c0 con 0x7fc4f8101d00 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4f0044070 con 0x7fc4f8101d00 2024-04-22T00:02:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.070+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f00043e0 con 0x7fc4f8101d00 2024-04-22T00:02:09.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.071+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f81a22f0 con 0x7fc4f8101d00 2024-04-22T00:02:09.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.073+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7fc4f0004580 con 0x7fc4f8101d00 2024-04-22T00:02:09.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.074+0000 7fc4e77fe640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 0x7fc4e00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.074+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc4f00f2360 con 0x7fc4f8101d00 2024-04-22T00:02:09.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.074+0000 7fc4f659c640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 0x7fc4e00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.074+0000 7fc4f659c640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 0x7fc4e00a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc4f819abf0 tx=0x7fc4e8005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.084+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc4f004f020 con 0x7fc4f8101d00 2024-04-22T00:02:09.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4253966323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:09.195 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3969291755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:09.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:09.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T00:02:09.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T00:02:09.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T00:02:09.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:08 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2942935124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.279 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[65167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:09.279 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[65167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:09.279 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[65167]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:09.279 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:02:08 smithi203 sudo[65167]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.303+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4f81a24c0 con 0x7fc4f8101d00 2024-04-22T00:02:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.304+0000 7fc4e77fe640 1 -- 172.21.15.38:0/3502534689 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7fc4f00bed90 con 0x7fc4f8101d00 2024-04-22T00:02:09.297 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 msgr2=0x7fc4e00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 0x7fc4e00a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc4f819abf0 tx=0x7fc4e8005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 msgr2=0x7fc4f81a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0005cc0 tx=0x7fc4f0035ae0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 shutdown_connections 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4e00a2d90 0x7fc4e00a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4f8101d00 0x7fc4f81a1250 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f8101280 0x7fc4f819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 --2- 172.21.15.38:0/3502534689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4f81008c0 0x7fc4f8199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 >> 172.21.15.38:0/3502534689 conn(0x7fc4f80fb370 msgr2=0x7fc4f80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 shutdown_connections 2024-04-22T00:02:09.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.306+0000 7fc4fc90f640 1 -- 172.21.15.38:0/3502534689 wait complete. 2024-04-22T00:02:09.312 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T00:02:09.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4253966323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:02:09.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3969291755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:08 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2942935124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 -- 172.21.15.38:0/1083829175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 msgr2=0x7f1e3c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/1083829175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c10d4f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1e34009cd0 tx=0x7f1e3402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 -- 172.21.15.38:0/1083829175 shutdown_connections 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/1083829175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c10d4f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/1083829175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c100620 0x7f1e3c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/1083829175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c106630 0x7f1e3c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 -- 172.21.15.38:0/1083829175 >> 172.21.15.38:0/1083829175 conn(0x7f1e3c077740 msgr2=0x7f1e3c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 -- 172.21.15.38:0/1083829175 shutdown_connections 2024-04-22T00:02:09.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.434+0000 7f1e43a3b640 1 -- 172.21.15.38:0/1083829175 wait complete. 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 Processor -- start 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 -- start start 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c1993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 0x7f1e3c1a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c110a40 con 0x7f1e3c100fe0 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c1108c0 con 0x7f1e3c100620 2024-04-22T00:02:09.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e43a3b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c110bc0 con 0x7f1e3c106630 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.435+0000 7f1e40fd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c1993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e41fd8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 0x7f1e3c1a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:50834/0 (socket says 172.21.15.38:50834) 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 -- 172.21.15.38:0/434275833 learned_addr learned my addr 172.21.15.38:0/434275833 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 -- 172.21.15.38:0/434275833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 msgr2=0x7f1e3c1a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 0x7f1e3c1a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 -- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 msgr2=0x7f1e3c1993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c1993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 -- 172.21.15.38:0/434275833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3c1a4440 con 0x7f1e3c100620 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e41fd8640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 0x7f1e3c1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e417d7640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800ea60 tx=0x7f1e3800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800cde0 con 0x7f1e3c100620 2024-04-22T00:02:09.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e380035f0 con 0x7f1e3c100620 2024-04-22T00:02:09.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e3c1a4610 con 0x7f1e3c100620 2024-04-22T00:02:09.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38010670 con 0x7f1e3c100620 2024-04-22T00:02:09.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.436+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e3c1a4b20 con 0x7f1e3c100620 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.438+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e3c068560 con 0x7f1e3c100620 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.438+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) ==== 138004+0+0 (secure 0 0 0) 0x7f1e38010810 con 0x7f1e3c100620 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.439+0000 7f1e327fc640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 0x7f1e280a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.439+0000 7f1e40fd6640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 0x7f1e280a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.439+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f1e380cfb10 con 0x7f1e3c100620 2024-04-22T00:02:09.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.440+0000 7f1e40fd6640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 0x7f1e280a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0693d0 tx=0x7f1e2c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.445+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e3809c280 con 0x7f1e3c100620 2024-04-22T00:02:09.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.450+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f1e380981f0 con 0x7f1e3c100620 2024-04-22T00:02:09.448 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:09.448 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:09.448 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84660]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:09.448 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84660]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:09.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.677+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e3c108bb0 con 0x7f1e3c100620 2024-04-22T00:02:09.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.678+0000 7f1e327fc640 1 -- 172.21.15.38:0/434275833 <== mon.2 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f1e38046780 con 0x7f1e3c100620 2024-04-22T00:02:09.670 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:09.670 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:02:03.216530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:02:09.671 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T00:02:09.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 msgr2=0x7f1e280a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 0x7f1e280a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0693d0 tx=0x7f1e2c005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:09.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 msgr2=0x7f1e3c198e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800ea60 tx=0x7f1e3800ef20 comp rx=0 tx=0).stop 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 shutdown_connections 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1e280a2e10 0x7f1e280a52d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e3c106630 0x7f1e3c1a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e3c100fe0 0x7f1e3c1993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 --2- 172.21.15.38:0/434275833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c100620 0x7f1e3c198e80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 >> 172.21.15.38:0/434275833 conn(0x7f1e3c077740 msgr2=0x7f1e3c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:09.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 shutdown_connections 2024-04-22T00:02:09.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.680+0000 7f1e43a3b640 1 -- 172.21.15.38:0/434275833 wait complete. 2024-04-22T00:02:09.684 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T00:02:09.684 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T00:02:09.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 -- 172.21.15.38:0/3219075440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 msgr2=0x7fe51810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 --2- 172.21.15.38:0/3219075440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe51810d510 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe510009cd0 tx=0x7fe51002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 -- 172.21.15.38:0/3219075440 shutdown_connections 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 --2- 172.21.15.38:0/3219075440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe51810d510 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 --2- 172.21.15.38:0/3219075440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 0x7fe518100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 --2- 172.21.15.38:0/3219075440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 0x7fe518106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 -- 172.21.15.38:0/3219075440 >> 172.21.15.38:0/3219075440 conn(0x7fe5180777a0 msgr2=0x7fe518077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 -- 172.21.15.38:0/3219075440 shutdown_connections 2024-04-22T00:02:09.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.795+0000 7fe51f07d640 1 -- 172.21.15.38:0/3219075440 wait complete. 2024-04-22T00:02:09.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 Processor -- start 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 -- start start 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 0x7fe518196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 0x7fe51819b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe518110a60 con 0x7fe518101050 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181108e0 con 0x7fe518106650 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.796+0000 7fe51f07d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe518110be0 con 0x7fe518100690 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40136/0 (socket says 172.21.15.38:40136) 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 -- 172.21.15.38:0/566766326 learned_addr learned my addr 172.21.15.38:0/566766326 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe51ce19640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 0x7fe518196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe51d61a640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 0x7fe51819b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 -- 172.21.15.38:0/566766326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 msgr2=0x7fe518196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 0x7fe518196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 -- 172.21.15.38:0/566766326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 msgr2=0x7fe51819b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 0x7fe51819b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 -- 172.21.15.38:0/566766326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819be90 con 0x7fe518101050 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50ffff640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe50800e9f0 tx=0x7fe50800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50800cd80 con 0x7fe518101050 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe508005500 con 0x7fe518101050 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819c180 con 0x7fe518101050 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.797+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508010690 con 0x7fe518101050 2024-04-22T00:02:09.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.798+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181a3a10 con 0x7fe518101050 2024-04-22T00:02:09.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.799+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe518067cd0 con 0x7fe518101050 2024-04-22T00:02:09.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.801+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe5080046c0 con 0x7fe518101050 2024-04-22T00:02:09.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.802+0000 7fe50dffb640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 0x7fe5000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:09.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.802+0000 7fe51ce19640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 0x7fe5000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:09.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.802+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe5080cfff0 con 0x7fe518101050 2024-04-22T00:02:09.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.802+0000 7fe51ce19640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 0x7fe5000a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe51400bb60 tx=0x7fe51400b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:09.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.812+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe5080536f0 con 0x7fe518101050 2024-04-22T00:02:09.957 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84786]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:09.958 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84786]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:09.958 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84786]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:09.958 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:02:09 smithi038 sudo[84786]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:09.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1674172257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3502534689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T00:02:09.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:09.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:09.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:09.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:09.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:09 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/434275833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:09.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:09.994+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe518061b80 con 0x7fe5000a2e20 2024-04-22T00:02:09.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.004+0000 7fe50dffb640 1 -- 172.21.15.38:0/566766326 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339434 (secure 0 0 0) 0x7fe518061b80 con 0x7fe5000a2e20 2024-04-22T00:02:10.003 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:10.006 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:02:10.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 msgr2=0x7fe5000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 0x7fe5000a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe51400bb60 tx=0x7fe51400b390 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 msgr2=0x7fe5181973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe50800e9f0 tx=0x7fe50800edf0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 shutdown_connections 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe5000a2e20 0x7fe5000a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe518106650 0x7fe51819b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe518101050 0x7fe5181973e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 --2- 172.21.15.38:0/566766326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe518100690 0x7fe518196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 >> 172.21.15.38:0/566766326 conn(0x7fe5180777a0 msgr2=0x7fe518103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 shutdown_connections 2024-04-22T00:02:10.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.008+0000 7fe51f07d640 1 -- 172.21.15.38:0/566766326 wait complete. 2024-04-22T00:02:10.012 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T00:02:08.838548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193430,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1482,"num_read_kb":911,"num_write":560,"num_write_kb":4414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181082,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365116,"kb_used_data":15924,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576708,"statfs":{"total":1151940427776,"available":1151566548992,"internally_reserved":0,"allocated":16306176,"data_stored":9610138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986866"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888347+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-22T00:02:04.888347+0000","last_peered":"2024-04-22T00:02:04.888347+0000","last_clean":"2024-04-22T00:02:04.888347+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-22T00:02:04.888347+0000","last_undegraded":"2024-04-22T00:02:04.888347+0000","last_fullsized":"2024-04-22T00:02:04.888347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888466+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-22T00:02:04.888466+0000","last_peered":"2024-04-22T00:02:04.888466+0000","last_clean":"2024-04-22T00:02:04.888466+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-22T00:02:04.888466+0000","last_undegraded":"2024-04-22T00:02:04.888466+0000","last_fullsized":"2024-04-22T00:02:04.888466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888688+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-22T00:02:04.888688+0000","last_peered":"2024-04-22T00:02:04.888688+0000","last_clean":"2024-04-22T00:02:04.888688+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-22T00:02:04.888688+0000","last_undegraded":"2024-04-22T00:02:04.888688+0000","last_fullsized":"2024-04-22T00:02:04.888688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825019+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-22T00:02:04.825019+0000","last_peered":"2024-04-22T00:02:04.825019+0000","last_clean":"2024-04-22T00:02:04.825019+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-22T00:02:04.825019+0000","last_undegraded":"2024-04-22T00:02:04.825019+0000","last_fullsized":"2024-04-22T00:02:04.825019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824719+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-22T00:02:04.824719+0000","last_peered":"2024-04-22T00:02:04.824719+0000","last_clean":"2024-04-22T00:02:04.824719+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-22T00:02:04.824719+0000","last_undegraded":"2024-04-22T00:02:04.824719+0000","last_fullsized":"2024-04-22T00:02:04.824719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887915+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-22T00:02:04.887915+0000","last_peered":"2024-04-22T00:02:04.887915+0000","last_clean":"2024-04-22T00:02:04.887915+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-22T00:02:04.887915+0000","last_undegraded":"2024-04-22T00:02:04.887915+0000","last_fullsized":"2024-04-22T00:02:04.887915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887595+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-22T00:02:04.887595+0000","last_peered":"2024-04-22T00:02:04.887595+0000","last_clean":"2024-04-22T00:02:04.887595+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-22T00:02:04.887595+0000","last_undegraded":"2024-04-22T00:02:04.887595+0000","last_fullsized":"2024-04-22T00:02:04.887595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887687+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-22T00:02:04.887687+0000","last_peered":"2024-04-22T00:02:04.887687+0000","last_clean":"2024-04-22T00:02:04.887687+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-22T00:02:04.887687+0000","last_undegraded":"2024-04-22T00:02:04.887687+0000","last_fullsized":"2024-04-22T00:02:04.887687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888505+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-22T00:02:04.888505+0000","last_peered":"2024-04-22T00:02:04.888505+0000","last_clean":"2024-04-22T00:02:04.888505+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-22T00:02:04.888505+0000","last_undegraded":"2024-04-22T00:02:04.888505+0000","last_fullsized":"2024-04-22T00:02:04.888505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888417+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-22T00:02:04.888417+0000","last_peered":"2024-04-22T00:02:04.888417+0000","last_clean":"2024-04-22T00:02:04.888417+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-22T00:02:04.888417+0000","last_undegraded":"2024-04-22T00:02:04.888417+0000","last_fullsized":"2024-04-22T00:02:04.888417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888811+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-22T00:02:04.888811+0000","last_peered":"2024-04-22T00:02:04.888811+0000","last_clean":"2024-04-22T00:02:04.888811+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-22T00:02:04.888811+0000","last_undegraded":"2024-04-22T00:02:04.888811+0000","last_fullsized":"2024-04-22T00:02:04.888811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824939+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-22T00:02:04.824939+0000","last_peered":"2024-04-22T00:02:04.824939+0000","last_clean":"2024-04-22T00:02:04.824939+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-22T00:02:04.824939+0000","last_undegraded":"2024-04-22T00:02:04.824939+0000","last_fullsized":"2024-04-22T00:02:04.824939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887447+0000","last_change":"2024-04-22T00:00:27.492882+0000","last_active":"2024-04-22T00:02:04.887447+0000","last_peered":"2024-04-22T00:02:04.887447+0000","last_clean":"2024-04-22T00:02:04.887447+0000","last_became_active":"2024-04-22T00:00:27.492451+0000","last_became_peered":"2024-04-22T00:00:27.492451+0000","last_unstale":"2024-04-22T00:02:04.887447+0000","last_undegraded":"2024-04-22T00:02:04.887447+0000","last_fullsized":"2024-04-22T00:02:04.887447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:10.283575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887109+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-22T00:02:04.887109+0000","last_peered":"2024-04-22T00:02:04.887109+0000","last_clean":"2024-04-22T00:02:04.887109+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-22T00:02:04.887109+0000","last_undegraded":"2024-04-22T00:02:04.887109+0000","last_fullsized":"2024-04-22T00:02:04.887109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887176+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-22T00:02:04.887176+0000","last_peered":"2024-04-22T00:02:04.887176+0000","last_clean":"2024-04-22T00:02:04.887176+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-22T00:02:04.887176+0000","last_undegraded":"2024-04-22T00:02:04.887176+0000","last_fullsized":"2024-04-22T00:02:04.887176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887188+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-22T00:02:04.887188+0000","last_peered":"2024-04-22T00:02:04.887188+0000","last_clean":"2024-04-22T00:02:04.887188+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-22T00:02:04.887188+0000","last_undegraded":"2024-04-22T00:02:04.887188+0000","last_fullsized":"2024-04-22T00:02:04.887188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888089+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-22T00:02:04.888089+0000","last_peered":"2024-04-22T00:02:04.888089+0000","last_clean":"2024-04-22T00:02:04.888089+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-22T00:02:04.888089+0000","last_undegraded":"2024-04-22T00:02:04.888089+0000","last_fullsized":"2024-04-22T00:02:04.888089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825401+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-22T00:02:04.825401+0000","last_peered":"2024-04-22T00:02:04.825401+0000","last_clean":"2024-04-22T00:02:04.825401+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-22T00:02:04.825401+0000","last_undegraded":"2024-04-22T00:02:04.825401+0000","last_fullsized":"2024-04-22T00:02:04.825401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887717+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-22T00:02:04.887717+0000","last_peered":"2024-04-22T00:02:04.887717+0000","last_clean":"2024-04-22T00:02:04.887717+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-22T00:02:04.887717+0000","last_undegraded":"2024-04-22T00:02:04.887717+0000","last_fullsized":"2024-04-22T00:02:04.887717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887268+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-22T00:02:04.887268+0000","last_peered":"2024-04-22T00:02:04.887268+0000","last_clean":"2024-04-22T00:02:04.887268+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-22T00:02:04.887268+0000","last_undegraded":"2024-04-22T00:02:04.887268+0000","last_fullsized":"2024-04-22T00:02:04.887268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":166,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065127+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:02:05.065127+0000","last_peered":"2024-04-22T00:02:05.065127+0000","last_clean":"2024-04-22T00:02:05.065127+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:02:05.065127+0000","last_undegraded":"2024-04-22T00:02:05.065127+0000","last_fullsized":"2024-04-22T00:02:05.065127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888904+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-22T00:02:04.888904+0000","last_peered":"2024-04-22T00:02:04.888904+0000","last_clean":"2024-04-22T00:02:04.888904+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-22T00:02:04.888904+0000","last_undegraded":"2024-04-22T00:02:04.888904+0000","last_fullsized":"2024-04-22T00:02:04.888904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824507+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-22T00:02:04.824507+0000","last_peered":"2024-04-22T00:02:04.824507+0000","last_clean":"2024-04-22T00:02:04.824507+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-22T00:02:04.824507+0000","last_undegraded":"2024-04-22T00:02:04.824507+0000","last_fullsized":"2024-04-22T00:02:04.824507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889096+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-22T00:02:04.889096+0000","last_peered":"2024-04-22T00:02:04.889096+0000","last_clean":"2024-04-22T00:02:04.889096+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-22T00:02:04.889096+0000","last_undegraded":"2024-04-22T00:02:04.889096+0000","last_fullsized":"2024-04-22T00:02:04.889096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888771+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-22T00:02:04.888771+0000","last_peered":"2024-04-22T00:02:04.888771+0000","last_clean":"2024-04-22T00:02:04.888771+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-22T00:02:04.888771+0000","last_undegraded":"2024-04-22T00:02:04.888771+0000","last_fullsized":"2024-04-22T00:02:04.888771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824795+0000","last_change":"2024-04-22T00:00:27.492457+0000","last_active":"2024-04-22T00:02:04.824795+0000","last_peered":"2024-04-22T00:02:04.824795+0000","last_clean":"2024-04-22T00:02:04.824795+0000","last_became_active":"2024-04-22T00:00:27.491828+0000","last_became_peered":"2024-04-22T00:00:27.491828+0000","last_unstale":"2024-04-22T00:02:04.824795+0000","last_undegraded":"2024-04-22T00:02:04.824795+0000","last_fullsized":"2024-04-22T00:02:04.824795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:09:18.512103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'13","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824854+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:02:04.824854+0000","last_peered":"2024-04-22T00:02:04.824854+0000","last_clean":"2024-04-22T00:02:04.824854+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:02:04.824854+0000","last_undegraded":"2024-04-22T00:02:04.824854+0000","last_fullsized":"2024-04-22T00:02:04.824854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887837+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-22T00:02:04.887837+0000","last_peered":"2024-04-22T00:02:04.887837+0000","last_clean":"2024-04-22T00:02:04.887837+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-22T00:02:04.887837+0000","last_undegraded":"2024-04-22T00:02:04.887837+0000","last_fullsized":"2024-04-22T00:02:04.887837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824567+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-22T00:02:04.824567+0000","last_peered":"2024-04-22T00:02:04.824567+0000","last_clean":"2024-04-22T00:02:04.824567+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-22T00:02:04.824567+0000","last_undegraded":"2024-04-22T00:02:04.824567+0000","last_fullsized":"2024-04-22T00:02:04.824567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824717+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-22T00:02:04.824717+0000","last_peered":"2024-04-22T00:02:04.824717+0000","last_clean":"2024-04-22T00:02:04.824717+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-22T00:02:04.824717+0000","last_undegraded":"2024-04-22T00:02:04.824717+0000","last_fullsized":"2024-04-22T00:02:04.824717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888097+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-22T00:02:04.888097+0000","last_peered":"2024-04-22T00:02:04.888097+0000","last_clean":"2024-04-22T00:02:04.888097+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-22T00:02:04.888097+0000","last_undegraded":"2024-04-22T00:02:04.888097+0000","last_fullsized":"2024-04-22T00:02:04.888097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887607+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-22T00:02:04.887607+0000","last_peered":"2024-04-22T00:02:04.887607+0000","last_clean":"2024-04-22T00:02:04.887607+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-22T00:02:04.887607+0000","last_undegraded":"2024-04-22T00:02:04.887607+0000","last_fullsized":"2024-04-22T00:02:04.887607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":158,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.444015+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:02:05.444015+0000","last_peered":"2024-04-22T00:02:05.444015+0000","last_clean":"2024-04-22T00:02:05.444015+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:02:05.444015+0000","last_undegraded":"2024-04-22T00:02:05.444015+0000","last_fullsized":"2024-04-22T00:02:05.444015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824620+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-22T00:02:04.824620+0000","last_peered":"2024-04-22T00:02:04.824620+0000","last_clean":"2024-04-22T00:02:04.824620+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-22T00:02:04.824620+0000","last_undegraded":"2024-04-22T00:02:04.824620+0000","last_fullsized":"2024-04-22T00:02:04.824620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887593+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-22T00:02:04.887593+0000","last_peered":"2024-04-22T00:02:04.887593+0000","last_clean":"2024-04-22T00:02:04.887593+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-22T00:02:04.887593+0000","last_undegraded":"2024-04-22T00:02:04.887593+0000","last_fullsized":"2024-04-22T00:02:04.887593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887797+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-22T00:02:04.887797+0000","last_peered":"2024-04-22T00:02:04.887797+0000","last_clean":"2024-04-22T00:02:04.887797+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-22T00:02:04.887797+0000","last_undegraded":"2024-04-22T00:02:04.887797+0000","last_fullsized":"2024-04-22T00:02:04.887797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887722+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-22T00:02:04.887722+0000","last_peered":"2024-04-22T00:02:04.887722+0000","last_clean":"2024-04-22T00:02:04.887722+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-22T00:02:04.887722+0000","last_undegraded":"2024-04-22T00:02:04.887722+0000","last_fullsized":"2024-04-22T00:02:04.887722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887618+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-22T00:02:04.887618+0000","last_peered":"2024-04-22T00:02:04.887618+0000","last_clean":"2024-04-22T00:02:04.887618+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-22T00:02:04.887618+0000","last_undegraded":"2024-04-22T00:02:04.887618+0000","last_fullsized":"2024-04-22T00:02:04.887618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887957+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-22T00:02:04.887957+0000","last_peered":"2024-04-22T00:02:04.887957+0000","last_clean":"2024-04-22T00:02:04.887957+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-22T00:02:04.887957+0000","last_undegraded":"2024-04-22T00:02:04.887957+0000","last_fullsized":"2024-04-22T00:02:04.887957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887527+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-22T00:02:04.887527+0000","last_peered":"2024-04-22T00:02:04.887527+0000","last_clean":"2024-04-22T00:02:04.887527+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-22T00:02:04.887527+0000","last_undegraded":"2024-04-22T00:02:04.887527+0000","last_fullsized":"2024-04-22T00:02:04.887527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888356+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-22T00:02:04.888356+0000","last_peered":"2024-04-22T00:02:04.888356+0000","last_clean":"2024-04-22T00:02:04.888356+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-22T00:02:04.888356+0000","last_undegraded":"2024-04-22T00:02:04.888356+0000","last_fullsized":"2024-04-22T00:02:04.888356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886900+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:02:04.886900+0000","last_peered":"2024-04-22T00:02:04.886900+0000","last_clean":"2024-04-22T00:02:04.886900+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:02:04.886900+0000","last_undegraded":"2024-04-22T00:02:04.886900+0000","last_fullsized":"2024-04-22T00:02:04.886900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824890+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-22T00:02:04.824890+0000","last_peered":"2024-04-22T00:02:04.824890+0000","last_clean":"2024-04-22T00:02:04.824890+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-22T00:02:04.824890+0000","last_undegraded":"2024-04-22T00:02:04.824890+0000","last_fullsized":"2024-04-22T00:02:04.824890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888253+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-22T00:02:04.888253+0000","last_peered":"2024-04-22T00:02:04.888253+0000","last_clean":"2024-04-22T00:02:04.888253+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-22T00:02:04.888253+0000","last_undegraded":"2024-04-22T00:02:04.888253+0000","last_fullsized":"2024-04-22T00:02:04.888253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":278,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.484350+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:02:05.484350+0000","last_peered":"2024-04-22T00:02:05.484350+0000","last_clean":"2024-04-22T00:02:05.484350+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:02:05.484350+0000","last_undegraded":"2024-04-22T00:02:05.484350+0000","last_fullsized":"2024-04-22T00:02:05.484350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888635+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-22T00:02:04.888635+0000","last_peered":"2024-04-22T00:02:04.888635+0000","last_clean":"2024-04-22T00:02:04.888635+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-22T00:02:04.888635+0000","last_undegraded":"2024-04-22T00:02:04.888635+0000","last_fullsized":"2024-04-22T00:02:04.888635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824672+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-22T00:02:04.824672+0000","last_peered":"2024-04-22T00:02:04.824672+0000","last_clean":"2024-04-22T00:02:04.824672+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-22T00:02:04.824672+0000","last_undegraded":"2024-04-22T00:02:04.824672+0000","last_fullsized":"2024-04-22T00:02:04.824672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888851+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:02:04.888851+0000","last_peered":"2024-04-22T00:02:04.888851+0000","last_clean":"2024-04-22T00:02:04.888851+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:02:04.888851+0000","last_undegraded":"2024-04-22T00:02:04.888851+0000","last_fullsized":"2024-04-22T00:02:04.888851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888718+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-22T00:02:04.888718+0000","last_peered":"2024-04-22T00:02:04.888718+0000","last_clean":"2024-04-22T00:02:04.888718+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-22T00:02:04.888718+0000","last_undegraded":"2024-04-22T00:02:04.888718+0000","last_fullsized":"2024-04-22T00:02:04.888718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824749+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-22T00:02:04.824749+0000","last_peered":"2024-04-22T00:02:04.824749+0000","last_clean":"2024-04-22T00:02:04.824749+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-22T00:02:04.824749+0000","last_undegraded":"2024-04-22T00:02:04.824749+0000","last_fullsized":"2024-04-22T00:02:04.824749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888091+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:02:04.888091+0000","last_peered":"2024-04-22T00:02:04.888091+0000","last_clean":"2024-04-22T00:02:04.888091+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:02:04.888091+0000","last_undegraded":"2024-04-22T00:02:04.888091+0000","last_fullsized":"2024-04-22T00:02:04.888091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888039+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-22T00:02:04.888039+0000","last_peered":"2024-04-22T00:02:04.888039+0000","last_clean":"2024-04-22T00:02:04.888039+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-22T00:02:04.888039+0000","last_undegraded":"2024-04-22T00:02:04.888039+0000","last_fullsized":"2024-04-22T00:02:04.888039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824846+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-22T00:02:04.824846+0000","last_peered":"2024-04-22T00:02:04.824846+0000","last_clean":"2024-04-22T00:02:04.824846+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-22T00:02:04.824846+0000","last_undegraded":"2024-04-22T00:02:04.824846+0000","last_fullsized":"2024-04-22T00:02:04.824846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'43","reported_seq":325,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443892+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:02:05.443892+0000","last_peered":"2024-04-22T00:02:05.443892+0000","last_clean":"2024-04-22T00:02:05.443892+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:02:05.443892+0000","last_undegraded":"2024-04-22T00:02:05.443892+0000","last_fullsized":"2024-04-22T00:02:05.443892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69045,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":45,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825061+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-22T00:02:04.825061+0000","last_peered":"2024-04-22T00:02:04.825061+0000","last_clean":"2024-04-22T00:02:04.825061+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-22T00:02:04.825061+0000","last_undegraded":"2024-04-22T00:02:04.825061+0000","last_fullsized":"2024-04-22T00:02:04.825061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888344+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-22T00:02:04.888344+0000","last_peered":"2024-04-22T00:02:04.888344+0000","last_clean":"2024-04-22T00:02:04.888344+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-22T00:02:04.888344+0000","last_undegraded":"2024-04-22T00:02:04.888344+0000","last_fullsized":"2024-04-22T00:02:04.888344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888192+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:02:04.888192+0000","last_peered":"2024-04-22T00:02:04.888192+0000","last_clean":"2024-04-22T00:02:04.888192+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:02:04.888192+0000","last_undegraded":"2024-04-22T00:02:04.888192+0000","last_fullsized":"2024-04-22T00:02:04.888192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887241+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-22T00:02:04.887241+0000","last_peered":"2024-04-22T00:02:04.887241+0000","last_clean":"2024-04-22T00:02:04.887241+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-22T00:02:04.887241+0000","last_undegraded":"2024-04-22T00:02:04.887241+0000","last_fullsized":"2024-04-22T00:02:04.887241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887354+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-22T00:02:04.887354+0000","last_peered":"2024-04-22T00:02:04.887354+0000","last_clean":"2024-04-22T00:02:04.887354+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-22T00:02:04.887354+0000","last_undegraded":"2024-04-22T00:02:04.887354+0000","last_fullsized":"2024-04-22T00:02:04.887354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887431+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:02:04.887431+0000","last_peered":"2024-04-22T00:02:04.887431+0000","last_clean":"2024-04-22T00:02:04.887431+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:02:04.887431+0000","last_undegraded":"2024-04-22T00:02:04.887431+0000","last_fullsized":"2024-04-22T00:02:04.887431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888547+0000","last_change":"2024-04-22T00:00:27.360979+0000","last_active":"2024-04-22T00:02:04.888547+0000","last_peered":"2024-04-22T00:02:04.888547+0000","last_clean":"2024-04-22T00:02:04.888547+0000","last_became_active":"2024-04-22T00:00:27.360589+0000","last_became_peered":"2024-04-22T00:00:27.360589+0000","last_unstale":"2024-04-22T00:02:04.888547+0000","last_undegraded":"2024-04-22T00:02:04.888547+0000","last_fullsized":"2024-04-22T00:02:04.888547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:25.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825130+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-22T00:02:04.825130+0000","last_peered":"2024-04-22T00:02:04.825130+0000","last_clean":"2024-04-22T00:02:04.825130+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-22T00:02:04.825130+0000","last_undegraded":"2024-04-22T00:02:04.825130+0000","last_fullsized":"2024-04-22T00:02:04.825130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443632+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-22T00:02:05.443632+0000","last_peered":"2024-04-22T00:02:05.443632+0000","last_clean":"2024-04-22T00:02:05.443632+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-22T00:02:05.443632+0000","last_undegraded":"2024-04-22T00:02:05.443632+0000","last_fullsized":"2024-04-22T00:02:05.443632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887550+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-22T00:02:04.887550+0000","last_peered":"2024-04-22T00:02:04.887550+0000","last_clean":"2024-04-22T00:02:04.887550+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-22T00:02:04.887550+0000","last_undegraded":"2024-04-22T00:02:04.887550+0000","last_fullsized":"2024-04-22T00:02:04.887550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888166+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-22T00:02:04.888166+0000","last_peered":"2024-04-22T00:02:04.888166+0000","last_clean":"2024-04-22T00:02:04.888166+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-22T00:02:04.888166+0000","last_undegraded":"2024-04-22T00:02:04.888166+0000","last_fullsized":"2024-04-22T00:02:04.888166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824529+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:02:04.824529+0000","last_peered":"2024-04-22T00:02:04.824529+0000","last_clean":"2024-04-22T00:02:04.824529+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:02:04.824529+0000","last_undegraded":"2024-04-22T00:02:04.824529+0000","last_fullsized":"2024-04-22T00:02:04.824529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887651+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-22T00:02:04.887651+0000","last_peered":"2024-04-22T00:02:04.887651+0000","last_clean":"2024-04-22T00:02:04.887651+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-22T00:02:04.887651+0000","last_undegraded":"2024-04-22T00:02:04.887651+0000","last_fullsized":"2024-04-22T00:02:04.887651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887494+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-22T00:02:04.887494+0000","last_peered":"2024-04-22T00:02:04.887494+0000","last_clean":"2024-04-22T00:02:04.887494+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-22T00:02:04.887494+0000","last_undegraded":"2024-04-22T00:02:04.887494+0000","last_fullsized":"2024-04-22T00:02:04.887494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824935+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-22T00:02:04.824935+0000","last_peered":"2024-04-22T00:02:04.824935+0000","last_clean":"2024-04-22T00:02:04.824935+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-22T00:02:04.824935+0000","last_undegraded":"2024-04-22T00:02:04.824935+0000","last_fullsized":"2024-04-22T00:02:04.824935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887393+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-22T00:02:04.887393+0000","last_peered":"2024-04-22T00:02:04.887393+0000","last_clean":"2024-04-22T00:02:04.887393+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-22T00:02:04.887393+0000","last_undegraded":"2024-04-22T00:02:04.887393+0000","last_fullsized":"2024-04-22T00:02:04.887393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824582+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-22T00:02:04.824582+0000","last_peered":"2024-04-22T00:02:04.824582+0000","last_clean":"2024-04-22T00:02:04.824582+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-22T00:02:04.824582+0000","last_undegraded":"2024-04-22T00:02:04.824582+0000","last_fullsized":"2024-04-22T00:02:04.824582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825063+0000","last_change":"2024-04-22T00:00:27.360956+0000","last_active":"2024-04-22T00:02:04.825063+0000","last_peered":"2024-04-22T00:02:04.825063+0000","last_clean":"2024-04-22T00:02:04.825063+0000","last_became_active":"2024-04-22T00:00:27.360481+0000","last_became_peered":"2024-04-22T00:00:27.360481+0000","last_unstale":"2024-04-22T00:02:04.825063+0000","last_undegraded":"2024-04-22T00:02:04.825063+0000","last_fullsized":"2024-04-22T00:02:04.825063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:16.624366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888234+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-22T00:02:04.888234+0000","last_peered":"2024-04-22T00:02:04.888234+0000","last_clean":"2024-04-22T00:02:04.888234+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-22T00:02:04.888234+0000","last_undegraded":"2024-04-22T00:02:04.888234+0000","last_fullsized":"2024-04-22T00:02:04.888234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887728+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-22T00:02:04.887728+0000","last_peered":"2024-04-22T00:02:04.887728+0000","last_clean":"2024-04-22T00:02:04.887728+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-22T00:02:04.887728+0000","last_undegraded":"2024-04-22T00:02:04.887728+0000","last_fullsized":"2024-04-22T00:02:04.887728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":160,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443457+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-22T00:02:05.443457+0000","last_peered":"2024-04-22T00:02:05.443457+0000","last_clean":"2024-04-22T00:02:05.443457+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-22T00:02:05.443457+0000","last_undegraded":"2024-04-22T00:02:05.443457+0000","last_fullsized":"2024-04-22T00:02:05.443457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887744+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-22T00:02:04.887744+0000","last_peered":"2024-04-22T00:02:04.887744+0000","last_clean":"2024-04-22T00:02:04.887744+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-22T00:02:04.887744+0000","last_undegraded":"2024-04-22T00:02:04.887744+0000","last_fullsized":"2024-04-22T00:02:04.887744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888339+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-22T00:02:04.888339+0000","last_peered":"2024-04-22T00:02:04.888339+0000","last_clean":"2024-04-22T00:02:04.888339+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-22T00:02:04.888339+0000","last_undegraded":"2024-04-22T00:02:04.888339+0000","last_fullsized":"2024-04-22T00:02:04.888339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886981+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-22T00:02:04.886981+0000","last_peered":"2024-04-22T00:02:04.886981+0000","last_clean":"2024-04-22T00:02:04.886981+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-22T00:02:04.886981+0000","last_undegraded":"2024-04-22T00:02:04.886981+0000","last_fullsized":"2024-04-22T00:02:04.886981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'7","reported_seq":159,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065164+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-22T00:02:05.065164+0000","last_peered":"2024-04-22T00:02:05.065164+0000","last_clean":"2024-04-22T00:02:05.065164+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-22T00:02:05.065164+0000","last_undegraded":"2024-04-22T00:02:05.065164+0000","last_fullsized":"2024-04-22T00:02:05.065164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888685+0000","last_change":"2024-04-22T00:00:27.369850+0000","last_active":"2024-04-22T00:02:04.888685+0000","last_peered":"2024-04-22T00:02:04.888685+0000","last_clean":"2024-04-22T00:02:04.888685+0000","last_became_active":"2024-04-22T00:00:27.361664+0000","last_became_peered":"2024-04-22T00:00:27.361664+0000","last_unstale":"2024-04-22T00:02:04.888685+0000","last_undegraded":"2024-04-22T00:02:04.888685+0000","last_fullsized":"2024-04-22T00:02:04.888685+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:42:13.794809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888320+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-22T00:02:04.888320+0000","last_peered":"2024-04-22T00:02:04.888320+0000","last_clean":"2024-04-22T00:02:04.888320+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-22T00:02:04.888320+0000","last_undegraded":"2024-04-22T00:02:04.888320+0000","last_fullsized":"2024-04-22T00:02:04.888320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824591+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-22T00:02:04.824591+0000","last_peered":"2024-04-22T00:02:04.824591+0000","last_clean":"2024-04-22T00:02:04.824591+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-22T00:02:04.824591+0000","last_undegraded":"2024-04-22T00:02:04.824591+0000","last_fullsized":"2024-04-22T00:02:04.824591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887243+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-22T00:02:04.887243+0000","last_peered":"2024-04-22T00:02:04.887243+0000","last_clean":"2024-04-22T00:02:04.887243+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-22T00:02:04.887243+0000","last_undegraded":"2024-04-22T00:02:04.887243+0000","last_fullsized":"2024-04-22T00:02:04.887243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888319+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-22T00:02:04.888319+0000","last_peered":"2024-04-22T00:02:04.888319+0000","last_clean":"2024-04-22T00:02:04.888319+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-22T00:02:04.888319+0000","last_undegraded":"2024-04-22T00:02:04.888319+0000","last_fullsized":"2024-04-22T00:02:04.888319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887578+0000","last_change":"2024-04-22T00:00:27.359209+0000","last_active":"2024-04-22T00:02:04.887578+0000","last_peered":"2024-04-22T00:02:04.887578+0000","last_clean":"2024-04-22T00:02:04.887578+0000","last_became_active":"2024-04-22T00:00:27.358750+0000","last_became_peered":"2024-04-22T00:00:27.358750+0000","last_unstale":"2024-04-22T00:02:04.887578+0000","last_undegraded":"2024-04-22T00:02:04.887578+0000","last_fullsized":"2024-04-22T00:02:04.887578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:05.053039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887357+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:02:04.887357+0000","last_peered":"2024-04-22T00:02:04.887357+0000","last_clean":"2024-04-22T00:02:04.887357+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:02:04.887357+0000","last_undegraded":"2024-04-22T00:02:04.887357+0000","last_fullsized":"2024-04-22T00:02:04.887357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887148+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-22T00:02:04.887148+0000","last_peered":"2024-04-22T00:02:04.887148+0000","last_clean":"2024-04-22T00:02:04.887148+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-22T00:02:04.887148+0000","last_undegraded":"2024-04-22T00:02:04.887148+0000","last_fullsized":"2024-04-22T00:02:04.887148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887086+0000","last_change":"2024-04-22T00:00:27.368767+0000","last_active":"2024-04-22T00:02:04.887086+0000","last_peered":"2024-04-22T00:02:04.887086+0000","last_clean":"2024-04-22T00:02:04.887086+0000","last_became_active":"2024-04-22T00:00:27.362160+0000","last_became_peered":"2024-04-22T00:00:27.362160+0000","last_unstale":"2024-04-22T00:02:04.887086+0000","last_undegraded":"2024-04-22T00:02:04.887086+0000","last_fullsized":"2024-04-22T00:02:04.887086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:38.606596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887917+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-22T00:02:04.887917+0000","last_peered":"2024-04-22T00:02:04.887917+0000","last_clean":"2024-04-22T00:02:04.887917+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-22T00:02:04.887917+0000","last_undegraded":"2024-04-22T00:02:04.887917+0000","last_fullsized":"2024-04-22T00:02:04.887917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887355+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-22T00:02:04.887355+0000","last_peered":"2024-04-22T00:02:04.887355+0000","last_clean":"2024-04-22T00:02:04.887355+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-22T00:02:04.887355+0000","last_undegraded":"2024-04-22T00:02:04.887355+0000","last_fullsized":"2024-04-22T00:02:04.887355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.029471+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:02:05.029471+0000","last_peered":"2024-04-22T00:02:05.029471+0000","last_clean":"2024-04-22T00:02:05.029471+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:02:05.029471+0000","last_undegraded":"2024-04-22T00:02:05.029471+0000","last_fullsized":"2024-04-22T00:02:05.029471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886568+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-22T00:02:04.886568+0000","last_peered":"2024-04-22T00:02:04.886568+0000","last_clean":"2024-04-22T00:02:04.886568+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-22T00:02:04.886568+0000","last_undegraded":"2024-04-22T00:02:04.886568+0000","last_fullsized":"2024-04-22T00:02:04.886568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825128+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-22T00:02:04.825128+0000","last_peered":"2024-04-22T00:02:04.825128+0000","last_clean":"2024-04-22T00:02:04.825128+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-22T00:02:04.825128+0000","last_undegraded":"2024-04-22T00:02:04.825128+0000","last_fullsized":"2024-04-22T00:02:04.825128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888601+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-22T00:02:04.888601+0000","last_peered":"2024-04-22T00:02:04.888601+0000","last_clean":"2024-04-22T00:02:04.888601+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-22T00:02:04.888601+0000","last_undegraded":"2024-04-22T00:02:04.888601+0000","last_fullsized":"2024-04-22T00:02:04.888601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824924+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-22T00:02:04.824924+0000","last_peered":"2024-04-22T00:02:04.824924+0000","last_clean":"2024-04-22T00:02:04.824924+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-22T00:02:04.824924+0000","last_undegraded":"2024-04-22T00:02:04.824924+0000","last_fullsized":"2024-04-22T00:02:04.824924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887300+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:02:04.887300+0000","last_peered":"2024-04-22T00:02:04.887300+0000","last_clean":"2024-04-22T00:02:04.887300+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:02:04.887300+0000","last_undegraded":"2024-04-22T00:02:04.887300+0000","last_fullsized":"2024-04-22T00:02:04.887300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824460+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-22T00:02:04.824460+0000","last_peered":"2024-04-22T00:02:04.824460+0000","last_clean":"2024-04-22T00:02:04.824460+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-22T00:02:04.824460+0000","last_undegraded":"2024-04-22T00:02:04.824460+0000","last_fullsized":"2024-04-22T00:02:04.824460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824869+0000","last_change":"2024-04-22T00:00:27.492100+0000","last_active":"2024-04-22T00:02:04.824869+0000","last_peered":"2024-04-22T00:02:04.824869+0000","last_clean":"2024-04-22T00:02:04.824869+0000","last_became_active":"2024-04-22T00:00:27.491414+0000","last_became_peered":"2024-04-22T00:00:27.491414+0000","last_unstale":"2024-04-22T00:02:04.824869+0000","last_undegraded":"2024-04-22T00:02:04.824869+0000","last_fullsized":"2024-04-22T00:02:04.824869+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:16:32.280602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888067+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:02:04.888067+0000","last_peered":"2024-04-22T00:02:04.888067+0000","last_clean":"2024-04-22T00:02:04.888067+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:02:04.888067+0000","last_undegraded":"2024-04-22T00:02:04.888067+0000","last_fullsized":"2024-04-22T00:02:04.888067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887057+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-22T00:02:04.887057+0000","last_peered":"2024-04-22T00:02:04.887057+0000","last_clean":"2024-04-22T00:02:04.887057+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-22T00:02:04.887057+0000","last_undegraded":"2024-04-22T00:02:04.887057+0000","last_fullsized":"2024-04-22T00:02:04.887057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824647+0000","last_change":"2024-04-22T00:00:27.497233+0000","last_active":"2024-04-22T00:02:04.824647+0000","last_peered":"2024-04-22T00:02:04.824647+0000","last_clean":"2024-04-22T00:02:04.824647+0000","last_became_active":"2024-04-22T00:00:27.492847+0000","last_became_peered":"2024-04-22T00:00:27.492847+0000","last_unstale":"2024-04-22T00:02:04.824647+0000","last_undegraded":"2024-04-22T00:02:04.824647+0000","last_fullsized":"2024-04-22T00:02:04.824647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:25.742746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887319+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:02:04.887319+0000","last_peered":"2024-04-22T00:02:04.887319+0000","last_clean":"2024-04-22T00:02:04.887319+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:02:04.887319+0000","last_undegraded":"2024-04-22T00:02:04.887319+0000","last_fullsized":"2024-04-22T00:02:04.887319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888732+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-22T00:02:04.888732+0000","last_peered":"2024-04-22T00:02:04.888732+0000","last_clean":"2024-04-22T00:02:04.888732+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-22T00:02:04.888732+0000","last_undegraded":"2024-04-22T00:02:04.888732+0000","last_fullsized":"2024-04-22T00:02:04.888732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824867+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-22T00:02:04.824867+0000","last_peered":"2024-04-22T00:02:04.824867+0000","last_clean":"2024-04-22T00:02:04.824867+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-22T00:02:04.824867+0000","last_undegraded":"2024-04-22T00:02:04.824867+0000","last_fullsized":"2024-04-22T00:02:04.824867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824650+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:02:04.824650+0000","last_peered":"2024-04-22T00:02:04.824650+0000","last_clean":"2024-04-22T00:02:04.824650+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:02:04.824650+0000","last_undegraded":"2024-04-22T00:02:04.824650+0000","last_fullsized":"2024-04-22T00:02:04.824650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888041+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-22T00:02:04.888041+0000","last_peered":"2024-04-22T00:02:04.888041+0000","last_clean":"2024-04-22T00:02:04.888041+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-22T00:02:04.888041+0000","last_undegraded":"2024-04-22T00:02:04.888041+0000","last_fullsized":"2024-04-22T00:02:04.888041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888383+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-22T00:02:04.888383+0000","last_peered":"2024-04-22T00:02:04.888383+0000","last_clean":"2024-04-22T00:02:04.888383+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-22T00:02:04.888383+0000","last_undegraded":"2024-04-22T00:02:04.888383+0000","last_fullsized":"2024-04-22T00:02:04.888383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888517+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:02:04.888517+0000","last_peered":"2024-04-22T00:02:04.888517+0000","last_clean":"2024-04-22T00:02:04.888517+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:02:04.888517+0000","last_undegraded":"2024-04-22T00:02:04.888517+0000","last_fullsized":"2024-04-22T00:02:04.888517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2010,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888876+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-22T00:02:04.888876+0000","last_peered":"2024-04-22T00:02:04.888876+0000","last_clean":"2024-04-22T00:02:04.888876+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-22T00:02:04.888876+0000","last_undegraded":"2024-04-22T00:02:04.888876+0000","last_fullsized":"2024-04-22T00:02:04.888876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886998+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-22T00:02:04.886998+0000","last_peered":"2024-04-22T00:02:04.886998+0000","last_clean":"2024-04-22T00:02:04.886998+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-22T00:02:04.886998+0000","last_undegraded":"2024-04-22T00:02:04.886998+0000","last_fullsized":"2024-04-22T00:02:04.886998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.027186+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:02:05.027186+0000","last_peered":"2024-04-22T00:02:05.027186+0000","last_clean":"2024-04-22T00:02:05.027186+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:02:05.027186+0000","last_undegraded":"2024-04-22T00:02:05.027186+0000","last_fullsized":"2024-04-22T00:02:05.027186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888652+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-22T00:02:04.888652+0000","last_peered":"2024-04-22T00:02:04.888652+0000","last_clean":"2024-04-22T00:02:04.888652+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-22T00:02:04.888652+0000","last_undegraded":"2024-04-22T00:02:04.888652+0000","last_fullsized":"2024-04-22T00:02:04.888652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886524+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-22T00:02:04.886524+0000","last_peered":"2024-04-22T00:02:04.886524+0000","last_clean":"2024-04-22T00:02:04.886524+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-22T00:02:04.886524+0000","last_undegraded":"2024-04-22T00:02:04.886524+0000","last_fullsized":"2024-04-22T00:02:04.886524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887998+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-22T00:02:04.887998+0000","last_peered":"2024-04-22T00:02:04.887998+0000","last_clean":"2024-04-22T00:02:04.887998+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-22T00:02:04.887998+0000","last_undegraded":"2024-04-22T00:02:04.887998+0000","last_fullsized":"2024-04-22T00:02:04.887998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887227+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-22T00:02:04.887227+0000","last_peered":"2024-04-22T00:02:04.887227+0000","last_clean":"2024-04-22T00:02:04.887227+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-22T00:02:04.887227+0000","last_undegraded":"2024-04-22T00:02:04.887227+0000","last_fullsized":"2024-04-22T00:02:04.887227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886684+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-22T00:02:04.886684+0000","last_peered":"2024-04-22T00:02:04.886684+0000","last_clean":"2024-04-22T00:02:04.886684+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-22T00:02:04.886684+0000","last_undegraded":"2024-04-22T00:02:04.886684+0000","last_fullsized":"2024-04-22T00:02:04.886684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886866+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-22T00:02:04.886866+0000","last_peered":"2024-04-22T00:02:04.886866+0000","last_clean":"2024-04-22T00:02:04.886866+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-22T00:02:04.886866+0000","last_undegraded":"2024-04-22T00:02:04.886866+0000","last_fullsized":"2024-04-22T00:02:04.886866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888285+0000","last_change":"2024-04-22T00:00:27.362810+0000","last_active":"2024-04-22T00:02:04.888285+0000","last_peered":"2024-04-22T00:02:04.888285+0000","last_clean":"2024-04-22T00:02:04.888285+0000","last_became_active":"2024-04-22T00:00:27.362380+0000","last_became_peered":"2024-04-22T00:00:27.362380+0000","last_unstale":"2024-04-22T00:02:04.888285+0000","last_undegraded":"2024-04-22T00:02:04.888285+0000","last_fullsized":"2024-04-22T00:02:04.888285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:09:30.391961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888096+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-22T00:02:04.888096+0000","last_peered":"2024-04-22T00:02:04.888096+0000","last_clean":"2024-04-22T00:02:04.888096+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-22T00:02:04.888096+0000","last_undegraded":"2024-04-22T00:02:04.888096+0000","last_fullsized":"2024-04-22T00:02:04.888096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887929+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-22T00:02:04.887929+0000","last_peered":"2024-04-22T00:02:04.887929+0000","last_clean":"2024-04-22T00:02:04.887929+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-22T00:02:04.887929+0000","last_undegraded":"2024-04-22T00:02:04.887929+0000","last_fullsized":"2024-04-22T00:02:04.887929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887246+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-22T00:02:04.887246+0000","last_peered":"2024-04-22T00:02:04.887246+0000","last_clean":"2024-04-22T00:02:04.887246+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-22T00:02:04.887246+0000","last_undegraded":"2024-04-22T00:02:04.887246+0000","last_fullsized":"2024-04-22T00:02:04.887246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889156+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-22T00:02:04.889156+0000","last_peered":"2024-04-22T00:02:04.889156+0000","last_clean":"2024-04-22T00:02:04.889156+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-22T00:02:04.889156+0000","last_undegraded":"2024-04-22T00:02:04.889156+0000","last_fullsized":"2024-04-22T00:02:04.889156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886857+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-22T00:02:04.886857+0000","last_peered":"2024-04-22T00:02:04.886857+0000","last_clean":"2024-04-22T00:02:04.886857+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-22T00:02:04.886857+0000","last_undegraded":"2024-04-22T00:02:04.886857+0000","last_fullsized":"2024-04-22T00:02:04.886857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887969+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-22T00:02:04.887969+0000","last_peered":"2024-04-22T00:02:04.887969+0000","last_clean":"2024-04-22T00:02:04.887969+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-22T00:02:04.887969+0000","last_undegraded":"2024-04-22T00:02:04.887969+0000","last_fullsized":"2024-04-22T00:02:04.887969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887229+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-22T00:02:04.887229+0000","last_peered":"2024-04-22T00:02:04.887229+0000","last_clean":"2024-04-22T00:02:04.887229+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-22T00:02:04.887229+0000","last_undegraded":"2024-04-22T00:02:04.887229+0000","last_fullsized":"2024-04-22T00:02:04.887229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887469+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-22T00:02:04.887469+0000","last_peered":"2024-04-22T00:02:04.887469+0000","last_clean":"2024-04-22T00:02:04.887469+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-22T00:02:04.887469+0000","last_undegraded":"2024-04-22T00:02:04.887469+0000","last_fullsized":"2024-04-22T00:02:04.887469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887415+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-22T00:02:04.887415+0000","last_peered":"2024-04-22T00:02:04.887415+0000","last_clean":"2024-04-22T00:02:04.887415+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-22T00:02:04.887415+0000","last_undegraded":"2024-04-22T00:02:04.887415+0000","last_fullsized":"2024-04-22T00:02:04.887415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602828,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1346,"num_read_kb":792,"num_write":377,"num_write_kb":1474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":512193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1888256,"data_stored":1303721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":904387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1458176,"data_stored":908749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28648,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716504,"statfs":{"total":95995035648,"available":95965700096,"internally_reserved":0,"allocated":1667072,"data_stored":1108187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":577532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":508971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":571655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1031199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":511672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1740800,"data_stored":1165197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:02:10.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.147 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1674172257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3502534689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.148 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.149 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:09 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/434275833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:10.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 -- 172.21.15.38:0/1565706173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0106650 msgr2=0x7f20f0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 --2- 172.21.15.38:0/1565706173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0106650 0x7f20f0106a30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009930 tx=0x7f20ec02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 -- 172.21.15.38:0/1565706173 shutdown_connections 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 --2- 172.21.15.38:0/1565706173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0101050 0x7f20f010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 --2- 172.21.15.38:0/1565706173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0100690 0x7f20f0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 --2- 172.21.15.38:0/1565706173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0106650 0x7f20f0106a30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 -- 172.21.15.38:0/1565706173 >> 172.21.15.38:0/1565706173 conn(0x7f20f00777a0 msgr2=0x7f20f0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 -- 172.21.15.38:0/1565706173 shutdown_connections 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.162+0000 7f20f73e5640 1 -- 172.21.15.38:0/1565706173 wait complete. 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 Processor -- start 2024-04-22T00:02:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 -- start start 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 0x7f20f010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0106650 0x7f20f010ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0111b60 con 0x7f20f0100690 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01119e0 con 0x7f20f0101050 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f73e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0111ce0 con 0x7f20f0106650 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40148/0 (socket says 172.21.15.38:40148) 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 -- 172.21.15.38:0/130389890 learned_addr learned my addr 172.21.15.38:0/130389890 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f4980640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 0x7f20f010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5982640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0106650 0x7f20f010ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 -- 172.21.15.38:0/130389890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0106650 msgr2=0x7f20f010ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0106650 0x7f20f010ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 -- 172.21.15.38:0/130389890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 msgr2=0x7f20f010e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 0x7f20f010e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 -- 172.21.15.38:0/130389890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f01a65c0 con 0x7f20f0100690 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f4980640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 0x7f20f010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.163+0000 7f20f5181640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009230 tx=0x7f20ec02cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.164+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec041070 con 0x7f20f0100690 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.164+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f01a6850 con 0x7f20f0100690 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.164+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ec004450 con 0x7f20f0100690 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.164+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec034440 con 0x7f20f0100690 2024-04-22T00:02:10.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.164+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f01a6d90 con 0x7f20f0100690 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.165+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f00689b0 con 0x7f20f0100690 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.166+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f20ec044070 con 0x7f20f0100690 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.166+0000 7f20e67fc640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 0x7f20dc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.167+0000 7f20f4980640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 0x7f20dc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.167+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f20ec0f1400 con 0x7f20f0100690 2024-04-22T00:02:10.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.167+0000 7f20f4980640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 0x7f20dc0a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20f010fa30 tx=0x7f20e0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.171+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f20ec0bdba0 con 0x7f20f0100690 2024-04-22T00:02:10.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1674172257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:10.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: Updating smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3502534689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:02:10.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:02:10.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:09 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/434275833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:10.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.351+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20f0061b80 con 0x7f20dc0a2e20 2024-04-22T00:02:10.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.360+0000 7f20e67fc640 1 -- 172.21.15.38:0/130389890 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339434 (secure 0 0 0) 0x7f20f0061b80 con 0x7f20dc0a2e20 2024-04-22T00:02:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:10.358 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 msgr2=0x7f20dc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 0x7f20dc0a52e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20f010fa30 tx=0x7f20e0005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 msgr2=0x7f20f010dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009230 tx=0x7f20ec02cba0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 shutdown_connections 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20dc0a2e20 0x7f20dc0a52e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f0106650 0x7f20f010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20f0101050 0x7f20f010e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 --2- 172.21.15.38:0/130389890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20f0100690 0x7f20f010dfa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 >> 172.21.15.38:0/130389890 conn(0x7f20f00777a0 msgr2=0x7f20f0103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 shutdown_connections 2024-04-22T00:02:10.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.364+0000 7f20f73e5640 1 -- 172.21.15.38:0/130389890 wait complete. 2024-04-22T00:02:10.367 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T00:02:08.838548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193430,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1482,"num_read_kb":911,"num_write":560,"num_write_kb":4414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181082,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365116,"kb_used_data":15924,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576708,"statfs":{"total":1151940427776,"available":1151566548992,"internally_reserved":0,"allocated":16306176,"data_stored":9610138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986866"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888347+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-22T00:02:04.888347+0000","last_peered":"2024-04-22T00:02:04.888347+0000","last_clean":"2024-04-22T00:02:04.888347+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-22T00:02:04.888347+0000","last_undegraded":"2024-04-22T00:02:04.888347+0000","last_fullsized":"2024-04-22T00:02:04.888347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888466+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-22T00:02:04.888466+0000","last_peered":"2024-04-22T00:02:04.888466+0000","last_clean":"2024-04-22T00:02:04.888466+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-22T00:02:04.888466+0000","last_undegraded":"2024-04-22T00:02:04.888466+0000","last_fullsized":"2024-04-22T00:02:04.888466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888688+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-22T00:02:04.888688+0000","last_peered":"2024-04-22T00:02:04.888688+0000","last_clean":"2024-04-22T00:02:04.888688+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-22T00:02:04.888688+0000","last_undegraded":"2024-04-22T00:02:04.888688+0000","last_fullsized":"2024-04-22T00:02:04.888688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825019+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-22T00:02:04.825019+0000","last_peered":"2024-04-22T00:02:04.825019+0000","last_clean":"2024-04-22T00:02:04.825019+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-22T00:02:04.825019+0000","last_undegraded":"2024-04-22T00:02:04.825019+0000","last_fullsized":"2024-04-22T00:02:04.825019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824719+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-22T00:02:04.824719+0000","last_peered":"2024-04-22T00:02:04.824719+0000","last_clean":"2024-04-22T00:02:04.824719+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-22T00:02:04.824719+0000","last_undegraded":"2024-04-22T00:02:04.824719+0000","last_fullsized":"2024-04-22T00:02:04.824719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887915+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-22T00:02:04.887915+0000","last_peered":"2024-04-22T00:02:04.887915+0000","last_clean":"2024-04-22T00:02:04.887915+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-22T00:02:04.887915+0000","last_undegraded":"2024-04-22T00:02:04.887915+0000","last_fullsized":"2024-04-22T00:02:04.887915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887595+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-22T00:02:04.887595+0000","last_peered":"2024-04-22T00:02:04.887595+0000","last_clean":"2024-04-22T00:02:04.887595+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-22T00:02:04.887595+0000","last_undegraded":"2024-04-22T00:02:04.887595+0000","last_fullsized":"2024-04-22T00:02:04.887595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887687+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-22T00:02:04.887687+0000","last_peered":"2024-04-22T00:02:04.887687+0000","last_clean":"2024-04-22T00:02:04.887687+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-22T00:02:04.887687+0000","last_undegraded":"2024-04-22T00:02:04.887687+0000","last_fullsized":"2024-04-22T00:02:04.887687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888505+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-22T00:02:04.888505+0000","last_peered":"2024-04-22T00:02:04.888505+0000","last_clean":"2024-04-22T00:02:04.888505+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-22T00:02:04.888505+0000","last_undegraded":"2024-04-22T00:02:04.888505+0000","last_fullsized":"2024-04-22T00:02:04.888505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888417+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-22T00:02:04.888417+0000","last_peered":"2024-04-22T00:02:04.888417+0000","last_clean":"2024-04-22T00:02:04.888417+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-22T00:02:04.888417+0000","last_undegraded":"2024-04-22T00:02:04.888417+0000","last_fullsized":"2024-04-22T00:02:04.888417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888811+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-22T00:02:04.888811+0000","last_peered":"2024-04-22T00:02:04.888811+0000","last_clean":"2024-04-22T00:02:04.888811+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-22T00:02:04.888811+0000","last_undegraded":"2024-04-22T00:02:04.888811+0000","last_fullsized":"2024-04-22T00:02:04.888811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824939+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-22T00:02:04.824939+0000","last_peered":"2024-04-22T00:02:04.824939+0000","last_clean":"2024-04-22T00:02:04.824939+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-22T00:02:04.824939+0000","last_undegraded":"2024-04-22T00:02:04.824939+0000","last_fullsized":"2024-04-22T00:02:04.824939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887447+0000","last_change":"2024-04-22T00:00:27.492882+0000","last_active":"2024-04-22T00:02:04.887447+0000","last_peered":"2024-04-22T00:02:04.887447+0000","last_clean":"2024-04-22T00:02:04.887447+0000","last_became_active":"2024-04-22T00:00:27.492451+0000","last_became_peered":"2024-04-22T00:00:27.492451+0000","last_unstale":"2024-04-22T00:02:04.887447+0000","last_undegraded":"2024-04-22T00:02:04.887447+0000","last_fullsized":"2024-04-22T00:02:04.887447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:10.283575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887109+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-22T00:02:04.887109+0000","last_peered":"2024-04-22T00:02:04.887109+0000","last_clean":"2024-04-22T00:02:04.887109+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-22T00:02:04.887109+0000","last_undegraded":"2024-04-22T00:02:04.887109+0000","last_fullsized":"2024-04-22T00:02:04.887109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887176+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-22T00:02:04.887176+0000","last_peered":"2024-04-22T00:02:04.887176+0000","last_clean":"2024-04-22T00:02:04.887176+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-22T00:02:04.887176+0000","last_undegraded":"2024-04-22T00:02:04.887176+0000","last_fullsized":"2024-04-22T00:02:04.887176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887188+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-22T00:02:04.887188+0000","last_peered":"2024-04-22T00:02:04.887188+0000","last_clean":"2024-04-22T00:02:04.887188+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-22T00:02:04.887188+0000","last_undegraded":"2024-04-22T00:02:04.887188+0000","last_fullsized":"2024-04-22T00:02:04.887188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888089+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-22T00:02:04.888089+0000","last_peered":"2024-04-22T00:02:04.888089+0000","last_clean":"2024-04-22T00:02:04.888089+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-22T00:02:04.888089+0000","last_undegraded":"2024-04-22T00:02:04.888089+0000","last_fullsized":"2024-04-22T00:02:04.888089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825401+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-22T00:02:04.825401+0000","last_peered":"2024-04-22T00:02:04.825401+0000","last_clean":"2024-04-22T00:02:04.825401+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-22T00:02:04.825401+0000","last_undegraded":"2024-04-22T00:02:04.825401+0000","last_fullsized":"2024-04-22T00:02:04.825401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887717+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-22T00:02:04.887717+0000","last_peered":"2024-04-22T00:02:04.887717+0000","last_clean":"2024-04-22T00:02:04.887717+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-22T00:02:04.887717+0000","last_undegraded":"2024-04-22T00:02:04.887717+0000","last_fullsized":"2024-04-22T00:02:04.887717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887268+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-22T00:02:04.887268+0000","last_peered":"2024-04-22T00:02:04.887268+0000","last_clean":"2024-04-22T00:02:04.887268+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-22T00:02:04.887268+0000","last_undegraded":"2024-04-22T00:02:04.887268+0000","last_fullsized":"2024-04-22T00:02:04.887268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":166,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065127+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:02:05.065127+0000","last_peered":"2024-04-22T00:02:05.065127+0000","last_clean":"2024-04-22T00:02:05.065127+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:02:05.065127+0000","last_undegraded":"2024-04-22T00:02:05.065127+0000","last_fullsized":"2024-04-22T00:02:05.065127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888904+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-22T00:02:04.888904+0000","last_peered":"2024-04-22T00:02:04.888904+0000","last_clean":"2024-04-22T00:02:04.888904+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-22T00:02:04.888904+0000","last_undegraded":"2024-04-22T00:02:04.888904+0000","last_fullsized":"2024-04-22T00:02:04.888904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824507+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-22T00:02:04.824507+0000","last_peered":"2024-04-22T00:02:04.824507+0000","last_clean":"2024-04-22T00:02:04.824507+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-22T00:02:04.824507+0000","last_undegraded":"2024-04-22T00:02:04.824507+0000","last_fullsized":"2024-04-22T00:02:04.824507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889096+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-22T00:02:04.889096+0000","last_peered":"2024-04-22T00:02:04.889096+0000","last_clean":"2024-04-22T00:02:04.889096+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-22T00:02:04.889096+0000","last_undegraded":"2024-04-22T00:02:04.889096+0000","last_fullsized":"2024-04-22T00:02:04.889096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888771+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-22T00:02:04.888771+0000","last_peered":"2024-04-22T00:02:04.888771+0000","last_clean":"2024-04-22T00:02:04.888771+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-22T00:02:04.888771+0000","last_undegraded":"2024-04-22T00:02:04.888771+0000","last_fullsized":"2024-04-22T00:02:04.888771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824795+0000","last_change":"2024-04-22T00:00:27.492457+0000","last_active":"2024-04-22T00:02:04.824795+0000","last_peered":"2024-04-22T00:02:04.824795+0000","last_clean":"2024-04-22T00:02:04.824795+0000","last_became_active":"2024-04-22T00:00:27.491828+0000","last_became_peered":"2024-04-22T00:00:27.491828+0000","last_unstale":"2024-04-22T00:02:04.824795+0000","last_undegraded":"2024-04-22T00:02:04.824795+0000","last_fullsized":"2024-04-22T00:02:04.824795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:09:18.512103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'13","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824854+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:02:04.824854+0000","last_peered":"2024-04-22T00:02:04.824854+0000","last_clean":"2024-04-22T00:02:04.824854+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:02:04.824854+0000","last_undegraded":"2024-04-22T00:02:04.824854+0000","last_fullsized":"2024-04-22T00:02:04.824854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887837+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-22T00:02:04.887837+0000","last_peered":"2024-04-22T00:02:04.887837+0000","last_clean":"2024-04-22T00:02:04.887837+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-22T00:02:04.887837+0000","last_undegraded":"2024-04-22T00:02:04.887837+0000","last_fullsized":"2024-04-22T00:02:04.887837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824567+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-22T00:02:04.824567+0000","last_peered":"2024-04-22T00:02:04.824567+0000","last_clean":"2024-04-22T00:02:04.824567+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-22T00:02:04.824567+0000","last_undegraded":"2024-04-22T00:02:04.824567+0000","last_fullsized":"2024-04-22T00:02:04.824567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824717+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-22T00:02:04.824717+0000","last_peered":"2024-04-22T00:02:04.824717+0000","last_clean":"2024-04-22T00:02:04.824717+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-22T00:02:04.824717+0000","last_undegraded":"2024-04-22T00:02:04.824717+0000","last_fullsized":"2024-04-22T00:02:04.824717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888097+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-22T00:02:04.888097+0000","last_peered":"2024-04-22T00:02:04.888097+0000","last_clean":"2024-04-22T00:02:04.888097+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-22T00:02:04.888097+0000","last_undegraded":"2024-04-22T00:02:04.888097+0000","last_fullsized":"2024-04-22T00:02:04.888097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887607+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-22T00:02:04.887607+0000","last_peered":"2024-04-22T00:02:04.887607+0000","last_clean":"2024-04-22T00:02:04.887607+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-22T00:02:04.887607+0000","last_undegraded":"2024-04-22T00:02:04.887607+0000","last_fullsized":"2024-04-22T00:02:04.887607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":158,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.444015+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:02:05.444015+0000","last_peered":"2024-04-22T00:02:05.444015+0000","last_clean":"2024-04-22T00:02:05.444015+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:02:05.444015+0000","last_undegraded":"2024-04-22T00:02:05.444015+0000","last_fullsized":"2024-04-22T00:02:05.444015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824620+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-22T00:02:04.824620+0000","last_peered":"2024-04-22T00:02:04.824620+0000","last_clean":"2024-04-22T00:02:04.824620+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-22T00:02:04.824620+0000","last_undegraded":"2024-04-22T00:02:04.824620+0000","last_fullsized":"2024-04-22T00:02:04.824620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887593+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-22T00:02:04.887593+0000","last_peered":"2024-04-22T00:02:04.887593+0000","last_clean":"2024-04-22T00:02:04.887593+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-22T00:02:04.887593+0000","last_undegraded":"2024-04-22T00:02:04.887593+0000","last_fullsized":"2024-04-22T00:02:04.887593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887797+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-22T00:02:04.887797+0000","last_peered":"2024-04-22T00:02:04.887797+0000","last_clean":"2024-04-22T00:02:04.887797+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-22T00:02:04.887797+0000","last_undegraded":"2024-04-22T00:02:04.887797+0000","last_fullsized":"2024-04-22T00:02:04.887797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887722+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-22T00:02:04.887722+0000","last_peered":"2024-04-22T00:02:04.887722+0000","last_clean":"2024-04-22T00:02:04.887722+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-22T00:02:04.887722+0000","last_undegraded":"2024-04-22T00:02:04.887722+0000","last_fullsized":"2024-04-22T00:02:04.887722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887618+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-22T00:02:04.887618+0000","last_peered":"2024-04-22T00:02:04.887618+0000","last_clean":"2024-04-22T00:02:04.887618+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-22T00:02:04.887618+0000","last_undegraded":"2024-04-22T00:02:04.887618+0000","last_fullsized":"2024-04-22T00:02:04.887618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887957+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-22T00:02:04.887957+0000","last_peered":"2024-04-22T00:02:04.887957+0000","last_clean":"2024-04-22T00:02:04.887957+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-22T00:02:04.887957+0000","last_undegraded":"2024-04-22T00:02:04.887957+0000","last_fullsized":"2024-04-22T00:02:04.887957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887527+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-22T00:02:04.887527+0000","last_peered":"2024-04-22T00:02:04.887527+0000","last_clean":"2024-04-22T00:02:04.887527+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-22T00:02:04.887527+0000","last_undegraded":"2024-04-22T00:02:04.887527+0000","last_fullsized":"2024-04-22T00:02:04.887527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888356+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-22T00:02:04.888356+0000","last_peered":"2024-04-22T00:02:04.888356+0000","last_clean":"2024-04-22T00:02:04.888356+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-22T00:02:04.888356+0000","last_undegraded":"2024-04-22T00:02:04.888356+0000","last_fullsized":"2024-04-22T00:02:04.888356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886900+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:02:04.886900+0000","last_peered":"2024-04-22T00:02:04.886900+0000","last_clean":"2024-04-22T00:02:04.886900+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:02:04.886900+0000","last_undegraded":"2024-04-22T00:02:04.886900+0000","last_fullsized":"2024-04-22T00:02:04.886900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824890+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-22T00:02:04.824890+0000","last_peered":"2024-04-22T00:02:04.824890+0000","last_clean":"2024-04-22T00:02:04.824890+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-22T00:02:04.824890+0000","last_undegraded":"2024-04-22T00:02:04.824890+0000","last_fullsized":"2024-04-22T00:02:04.824890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888253+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-22T00:02:04.888253+0000","last_peered":"2024-04-22T00:02:04.888253+0000","last_clean":"2024-04-22T00:02:04.888253+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-22T00:02:04.888253+0000","last_undegraded":"2024-04-22T00:02:04.888253+0000","last_fullsized":"2024-04-22T00:02:04.888253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":278,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.484350+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:02:05.484350+0000","last_peered":"2024-04-22T00:02:05.484350+0000","last_clean":"2024-04-22T00:02:05.484350+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:02:05.484350+0000","last_undegraded":"2024-04-22T00:02:05.484350+0000","last_fullsized":"2024-04-22T00:02:05.484350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888635+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-22T00:02:04.888635+0000","last_peered":"2024-04-22T00:02:04.888635+0000","last_clean":"2024-04-22T00:02:04.888635+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-22T00:02:04.888635+0000","last_undegraded":"2024-04-22T00:02:04.888635+0000","last_fullsized":"2024-04-22T00:02:04.888635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824672+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-22T00:02:04.824672+0000","last_peered":"2024-04-22T00:02:04.824672+0000","last_clean":"2024-04-22T00:02:04.824672+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-22T00:02:04.824672+0000","last_undegraded":"2024-04-22T00:02:04.824672+0000","last_fullsized":"2024-04-22T00:02:04.824672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888851+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:02:04.888851+0000","last_peered":"2024-04-22T00:02:04.888851+0000","last_clean":"2024-04-22T00:02:04.888851+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:02:04.888851+0000","last_undegraded":"2024-04-22T00:02:04.888851+0000","last_fullsized":"2024-04-22T00:02:04.888851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888718+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-22T00:02:04.888718+0000","last_peered":"2024-04-22T00:02:04.888718+0000","last_clean":"2024-04-22T00:02:04.888718+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-22T00:02:04.888718+0000","last_undegraded":"2024-04-22T00:02:04.888718+0000","last_fullsized":"2024-04-22T00:02:04.888718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824749+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-22T00:02:04.824749+0000","last_peered":"2024-04-22T00:02:04.824749+0000","last_clean":"2024-04-22T00:02:04.824749+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-22T00:02:04.824749+0000","last_undegraded":"2024-04-22T00:02:04.824749+0000","last_fullsized":"2024-04-22T00:02:04.824749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888091+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:02:04.888091+0000","last_peered":"2024-04-22T00:02:04.888091+0000","last_clean":"2024-04-22T00:02:04.888091+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:02:04.888091+0000","last_undegraded":"2024-04-22T00:02:04.888091+0000","last_fullsized":"2024-04-22T00:02:04.888091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888039+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-22T00:02:04.888039+0000","last_peered":"2024-04-22T00:02:04.888039+0000","last_clean":"2024-04-22T00:02:04.888039+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-22T00:02:04.888039+0000","last_undegraded":"2024-04-22T00:02:04.888039+0000","last_fullsized":"2024-04-22T00:02:04.888039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824846+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-22T00:02:04.824846+0000","last_peered":"2024-04-22T00:02:04.824846+0000","last_clean":"2024-04-22T00:02:04.824846+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-22T00:02:04.824846+0000","last_undegraded":"2024-04-22T00:02:04.824846+0000","last_fullsized":"2024-04-22T00:02:04.824846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'43","reported_seq":325,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443892+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:02:05.443892+0000","last_peered":"2024-04-22T00:02:05.443892+0000","last_clean":"2024-04-22T00:02:05.443892+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:02:05.443892+0000","last_undegraded":"2024-04-22T00:02:05.443892+0000","last_fullsized":"2024-04-22T00:02:05.443892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69045,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":45,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825061+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-22T00:02:04.825061+0000","last_peered":"2024-04-22T00:02:04.825061+0000","last_clean":"2024-04-22T00:02:04.825061+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-22T00:02:04.825061+0000","last_undegraded":"2024-04-22T00:02:04.825061+0000","last_fullsized":"2024-04-22T00:02:04.825061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888344+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-22T00:02:04.888344+0000","last_peered":"2024-04-22T00:02:04.888344+0000","last_clean":"2024-04-22T00:02:04.888344+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-22T00:02:04.888344+0000","last_undegraded":"2024-04-22T00:02:04.888344+0000","last_fullsized":"2024-04-22T00:02:04.888344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888192+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:02:04.888192+0000","last_peered":"2024-04-22T00:02:04.888192+0000","last_clean":"2024-04-22T00:02:04.888192+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:02:04.888192+0000","last_undegraded":"2024-04-22T00:02:04.888192+0000","last_fullsized":"2024-04-22T00:02:04.888192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887241+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-22T00:02:04.887241+0000","last_peered":"2024-04-22T00:02:04.887241+0000","last_clean":"2024-04-22T00:02:04.887241+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-22T00:02:04.887241+0000","last_undegraded":"2024-04-22T00:02:04.887241+0000","last_fullsized":"2024-04-22T00:02:04.887241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887354+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-22T00:02:04.887354+0000","last_peered":"2024-04-22T00:02:04.887354+0000","last_clean":"2024-04-22T00:02:04.887354+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-22T00:02:04.887354+0000","last_undegraded":"2024-04-22T00:02:04.887354+0000","last_fullsized":"2024-04-22T00:02:04.887354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887431+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:02:04.887431+0000","last_peered":"2024-04-22T00:02:04.887431+0000","last_clean":"2024-04-22T00:02:04.887431+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:02:04.887431+0000","last_undegraded":"2024-04-22T00:02:04.887431+0000","last_fullsized":"2024-04-22T00:02:04.887431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888547+0000","last_change":"2024-04-22T00:00:27.360979+0000","last_active":"2024-04-22T00:02:04.888547+0000","last_peered":"2024-04-22T00:02:04.888547+0000","last_clean":"2024-04-22T00:02:04.888547+0000","last_became_active":"2024-04-22T00:00:27.360589+0000","last_became_peered":"2024-04-22T00:00:27.360589+0000","last_unstale":"2024-04-22T00:02:04.888547+0000","last_undegraded":"2024-04-22T00:02:04.888547+0000","last_fullsized":"2024-04-22T00:02:04.888547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:25.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825130+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-22T00:02:04.825130+0000","last_peered":"2024-04-22T00:02:04.825130+0000","last_clean":"2024-04-22T00:02:04.825130+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-22T00:02:04.825130+0000","last_undegraded":"2024-04-22T00:02:04.825130+0000","last_fullsized":"2024-04-22T00:02:04.825130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443632+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-22T00:02:05.443632+0000","last_peered":"2024-04-22T00:02:05.443632+0000","last_clean":"2024-04-22T00:02:05.443632+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-22T00:02:05.443632+0000","last_undegraded":"2024-04-22T00:02:05.443632+0000","last_fullsized":"2024-04-22T00:02:05.443632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887550+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-22T00:02:04.887550+0000","last_peered":"2024-04-22T00:02:04.887550+0000","last_clean":"2024-04-22T00:02:04.887550+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-22T00:02:04.887550+0000","last_undegraded":"2024-04-22T00:02:04.887550+0000","last_fullsized":"2024-04-22T00:02:04.887550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888166+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-22T00:02:04.888166+0000","last_peered":"2024-04-22T00:02:04.888166+0000","last_clean":"2024-04-22T00:02:04.888166+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-22T00:02:04.888166+0000","last_undegraded":"2024-04-22T00:02:04.888166+0000","last_fullsized":"2024-04-22T00:02:04.888166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824529+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:02:04.824529+0000","last_peered":"2024-04-22T00:02:04.824529+0000","last_clean":"2024-04-22T00:02:04.824529+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:02:04.824529+0000","last_undegraded":"2024-04-22T00:02:04.824529+0000","last_fullsized":"2024-04-22T00:02:04.824529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887651+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-22T00:02:04.887651+0000","last_peered":"2024-04-22T00:02:04.887651+0000","last_clean":"2024-04-22T00:02:04.887651+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-22T00:02:04.887651+0000","last_undegraded":"2024-04-22T00:02:04.887651+0000","last_fullsized":"2024-04-22T00:02:04.887651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887494+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-22T00:02:04.887494+0000","last_peered":"2024-04-22T00:02:04.887494+0000","last_clean":"2024-04-22T00:02:04.887494+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-22T00:02:04.887494+0000","last_undegraded":"2024-04-22T00:02:04.887494+0000","last_fullsized":"2024-04-22T00:02:04.887494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824935+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-22T00:02:04.824935+0000","last_peered":"2024-04-22T00:02:04.824935+0000","last_clean":"2024-04-22T00:02:04.824935+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-22T00:02:04.824935+0000","last_undegraded":"2024-04-22T00:02:04.824935+0000","last_fullsized":"2024-04-22T00:02:04.824935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887393+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-22T00:02:04.887393+0000","last_peered":"2024-04-22T00:02:04.887393+0000","last_clean":"2024-04-22T00:02:04.887393+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-22T00:02:04.887393+0000","last_undegraded":"2024-04-22T00:02:04.887393+0000","last_fullsized":"2024-04-22T00:02:04.887393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824582+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-22T00:02:04.824582+0000","last_peered":"2024-04-22T00:02:04.824582+0000","last_clean":"2024-04-22T00:02:04.824582+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-22T00:02:04.824582+0000","last_undegraded":"2024-04-22T00:02:04.824582+0000","last_fullsized":"2024-04-22T00:02:04.824582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825063+0000","last_change":"2024-04-22T00:00:27.360956+0000","last_active":"2024-04-22T00:02:04.825063+0000","last_peered":"2024-04-22T00:02:04.825063+0000","last_clean":"2024-04-22T00:02:04.825063+0000","last_became_active":"2024-04-22T00:00:27.360481+0000","last_became_peered":"2024-04-22T00:00:27.360481+0000","last_unstale":"2024-04-22T00:02:04.825063+0000","last_undegraded":"2024-04-22T00:02:04.825063+0000","last_fullsized":"2024-04-22T00:02:04.825063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:16.624366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888234+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-22T00:02:04.888234+0000","last_peered":"2024-04-22T00:02:04.888234+0000","last_clean":"2024-04-22T00:02:04.888234+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-22T00:02:04.888234+0000","last_undegraded":"2024-04-22T00:02:04.888234+0000","last_fullsized":"2024-04-22T00:02:04.888234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887728+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-22T00:02:04.887728+0000","last_peered":"2024-04-22T00:02:04.887728+0000","last_clean":"2024-04-22T00:02:04.887728+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-22T00:02:04.887728+0000","last_undegraded":"2024-04-22T00:02:04.887728+0000","last_fullsized":"2024-04-22T00:02:04.887728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":160,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443457+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-22T00:02:05.443457+0000","last_peered":"2024-04-22T00:02:05.443457+0000","last_clean":"2024-04-22T00:02:05.443457+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-22T00:02:05.443457+0000","last_undegraded":"2024-04-22T00:02:05.443457+0000","last_fullsized":"2024-04-22T00:02:05.443457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887744+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-22T00:02:04.887744+0000","last_peered":"2024-04-22T00:02:04.887744+0000","last_clean":"2024-04-22T00:02:04.887744+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-22T00:02:04.887744+0000","last_undegraded":"2024-04-22T00:02:04.887744+0000","last_fullsized":"2024-04-22T00:02:04.887744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888339+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-22T00:02:04.888339+0000","last_peered":"2024-04-22T00:02:04.888339+0000","last_clean":"2024-04-22T00:02:04.888339+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-22T00:02:04.888339+0000","last_undegraded":"2024-04-22T00:02:04.888339+0000","last_fullsized":"2024-04-22T00:02:04.888339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886981+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-22T00:02:04.886981+0000","last_peered":"2024-04-22T00:02:04.886981+0000","last_clean":"2024-04-22T00:02:04.886981+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-22T00:02:04.886981+0000","last_undegraded":"2024-04-22T00:02:04.886981+0000","last_fullsized":"2024-04-22T00:02:04.886981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'7","reported_seq":159,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065164+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-22T00:02:05.065164+0000","last_peered":"2024-04-22T00:02:05.065164+0000","last_clean":"2024-04-22T00:02:05.065164+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-22T00:02:05.065164+0000","last_undegraded":"2024-04-22T00:02:05.065164+0000","last_fullsized":"2024-04-22T00:02:05.065164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888685+0000","last_change":"2024-04-22T00:00:27.369850+0000","last_active":"2024-04-22T00:02:04.888685+0000","last_peered":"2024-04-22T00:02:04.888685+0000","last_clean":"2024-04-22T00:02:04.888685+0000","last_became_active":"2024-04-22T00:00:27.361664+0000","last_became_peered":"2024-04-22T00:00:27.361664+0000","last_unstale":"2024-04-22T00:02:04.888685+0000","last_undegraded":"2024-04-22T00:02:04.888685+0000","last_fullsized":"2024-04-22T00:02:04.888685+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:42:13.794809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888320+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-22T00:02:04.888320+0000","last_peered":"2024-04-22T00:02:04.888320+0000","last_clean":"2024-04-22T00:02:04.888320+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-22T00:02:04.888320+0000","last_undegraded":"2024-04-22T00:02:04.888320+0000","last_fullsized":"2024-04-22T00:02:04.888320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824591+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-22T00:02:04.824591+0000","last_peered":"2024-04-22T00:02:04.824591+0000","last_clean":"2024-04-22T00:02:04.824591+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-22T00:02:04.824591+0000","last_undegraded":"2024-04-22T00:02:04.824591+0000","last_fullsized":"2024-04-22T00:02:04.824591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887243+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-22T00:02:04.887243+0000","last_peered":"2024-04-22T00:02:04.887243+0000","last_clean":"2024-04-22T00:02:04.887243+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-22T00:02:04.887243+0000","last_undegraded":"2024-04-22T00:02:04.887243+0000","last_fullsized":"2024-04-22T00:02:04.887243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888319+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-22T00:02:04.888319+0000","last_peered":"2024-04-22T00:02:04.888319+0000","last_clean":"2024-04-22T00:02:04.888319+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-22T00:02:04.888319+0000","last_undegraded":"2024-04-22T00:02:04.888319+0000","last_fullsized":"2024-04-22T00:02:04.888319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887578+0000","last_change":"2024-04-22T00:00:27.359209+0000","last_active":"2024-04-22T00:02:04.887578+0000","last_peered":"2024-04-22T00:02:04.887578+0000","last_clean":"2024-04-22T00:02:04.887578+0000","last_became_active":"2024-04-22T00:00:27.358750+0000","last_became_peered":"2024-04-22T00:00:27.358750+0000","last_unstale":"2024-04-22T00:02:04.887578+0000","last_undegraded":"2024-04-22T00:02:04.887578+0000","last_fullsized":"2024-04-22T00:02:04.887578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:05.053039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887357+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:02:04.887357+0000","last_peered":"2024-04-22T00:02:04.887357+0000","last_clean":"2024-04-22T00:02:04.887357+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:02:04.887357+0000","last_undegraded":"2024-04-22T00:02:04.887357+0000","last_fullsized":"2024-04-22T00:02:04.887357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887148+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-22T00:02:04.887148+0000","last_peered":"2024-04-22T00:02:04.887148+0000","last_clean":"2024-04-22T00:02:04.887148+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-22T00:02:04.887148+0000","last_undegraded":"2024-04-22T00:02:04.887148+0000","last_fullsized":"2024-04-22T00:02:04.887148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887086+0000","last_change":"2024-04-22T00:00:27.368767+0000","last_active":"2024-04-22T00:02:04.887086+0000","last_peered":"2024-04-22T00:02:04.887086+0000","last_clean":"2024-04-22T00:02:04.887086+0000","last_became_active":"2024-04-22T00:00:27.362160+0000","last_became_peered":"2024-04-22T00:00:27.362160+0000","last_unstale":"2024-04-22T00:02:04.887086+0000","last_undegraded":"2024-04-22T00:02:04.887086+0000","last_fullsized":"2024-04-22T00:02:04.887086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:38.606596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887917+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-22T00:02:04.887917+0000","last_peered":"2024-04-22T00:02:04.887917+0000","last_clean":"2024-04-22T00:02:04.887917+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-22T00:02:04.887917+0000","last_undegraded":"2024-04-22T00:02:04.887917+0000","last_fullsized":"2024-04-22T00:02:04.887917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887355+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-22T00:02:04.887355+0000","last_peered":"2024-04-22T00:02:04.887355+0000","last_clean":"2024-04-22T00:02:04.887355+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-22T00:02:04.887355+0000","last_undegraded":"2024-04-22T00:02:04.887355+0000","last_fullsized":"2024-04-22T00:02:04.887355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.029471+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:02:05.029471+0000","last_peered":"2024-04-22T00:02:05.029471+0000","last_clean":"2024-04-22T00:02:05.029471+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:02:05.029471+0000","last_undegraded":"2024-04-22T00:02:05.029471+0000","last_fullsized":"2024-04-22T00:02:05.029471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886568+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-22T00:02:04.886568+0000","last_peered":"2024-04-22T00:02:04.886568+0000","last_clean":"2024-04-22T00:02:04.886568+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-22T00:02:04.886568+0000","last_undegraded":"2024-04-22T00:02:04.886568+0000","last_fullsized":"2024-04-22T00:02:04.886568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825128+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-22T00:02:04.825128+0000","last_peered":"2024-04-22T00:02:04.825128+0000","last_clean":"2024-04-22T00:02:04.825128+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-22T00:02:04.825128+0000","last_undegraded":"2024-04-22T00:02:04.825128+0000","last_fullsized":"2024-04-22T00:02:04.825128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888601+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-22T00:02:04.888601+0000","last_peered":"2024-04-22T00:02:04.888601+0000","last_clean":"2024-04-22T00:02:04.888601+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-22T00:02:04.888601+0000","last_undegraded":"2024-04-22T00:02:04.888601+0000","last_fullsized":"2024-04-22T00:02:04.888601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824924+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-22T00:02:04.824924+0000","last_peered":"2024-04-22T00:02:04.824924+0000","last_clean":"2024-04-22T00:02:04.824924+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-22T00:02:04.824924+0000","last_undegraded":"2024-04-22T00:02:04.824924+0000","last_fullsized":"2024-04-22T00:02:04.824924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887300+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:02:04.887300+0000","last_peered":"2024-04-22T00:02:04.887300+0000","last_clean":"2024-04-22T00:02:04.887300+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:02:04.887300+0000","last_undegraded":"2024-04-22T00:02:04.887300+0000","last_fullsized":"2024-04-22T00:02:04.887300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824460+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-22T00:02:04.824460+0000","last_peered":"2024-04-22T00:02:04.824460+0000","last_clean":"2024-04-22T00:02:04.824460+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-22T00:02:04.824460+0000","last_undegraded":"2024-04-22T00:02:04.824460+0000","last_fullsized":"2024-04-22T00:02:04.824460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824869+0000","last_change":"2024-04-22T00:00:27.492100+0000","last_active":"2024-04-22T00:02:04.824869+0000","last_peered":"2024-04-22T00:02:04.824869+0000","last_clean":"2024-04-22T00:02:04.824869+0000","last_became_active":"2024-04-22T00:00:27.491414+0000","last_became_peered":"2024-04-22T00:00:27.491414+0000","last_unstale":"2024-04-22T00:02:04.824869+0000","last_undegraded":"2024-04-22T00:02:04.824869+0000","last_fullsized":"2024-04-22T00:02:04.824869+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:16:32.280602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888067+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:02:04.888067+0000","last_peered":"2024-04-22T00:02:04.888067+0000","last_clean":"2024-04-22T00:02:04.888067+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:02:04.888067+0000","last_undegraded":"2024-04-22T00:02:04.888067+0000","last_fullsized":"2024-04-22T00:02:04.888067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887057+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-22T00:02:04.887057+0000","last_peered":"2024-04-22T00:02:04.887057+0000","last_clean":"2024-04-22T00:02:04.887057+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-22T00:02:04.887057+0000","last_undegraded":"2024-04-22T00:02:04.887057+0000","last_fullsized":"2024-04-22T00:02:04.887057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824647+0000","last_change":"2024-04-22T00:00:27.497233+0000","last_active":"2024-04-22T00:02:04.824647+0000","last_peered":"2024-04-22T00:02:04.824647+0000","last_clean":"2024-04-22T00:02:04.824647+0000","last_became_active":"2024-04-22T00:00:27.492847+0000","last_became_peered":"2024-04-22T00:00:27.492847+0000","last_unstale":"2024-04-22T00:02:04.824647+0000","last_undegraded":"2024-04-22T00:02:04.824647+0000","last_fullsized":"2024-04-22T00:02:04.824647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:25.742746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887319+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:02:04.887319+0000","last_peered":"2024-04-22T00:02:04.887319+0000","last_clean":"2024-04-22T00:02:04.887319+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:02:04.887319+0000","last_undegraded":"2024-04-22T00:02:04.887319+0000","last_fullsized":"2024-04-22T00:02:04.887319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888732+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-22T00:02:04.888732+0000","last_peered":"2024-04-22T00:02:04.888732+0000","last_clean":"2024-04-22T00:02:04.888732+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-22T00:02:04.888732+0000","last_undegraded":"2024-04-22T00:02:04.888732+0000","last_fullsized":"2024-04-22T00:02:04.888732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824867+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-22T00:02:04.824867+0000","last_peered":"2024-04-22T00:02:04.824867+0000","last_clean":"2024-04-22T00:02:04.824867+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-22T00:02:04.824867+0000","last_undegraded":"2024-04-22T00:02:04.824867+0000","last_fullsized":"2024-04-22T00:02:04.824867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824650+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:02:04.824650+0000","last_peered":"2024-04-22T00:02:04.824650+0000","last_clean":"2024-04-22T00:02:04.824650+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:02:04.824650+0000","last_undegraded":"2024-04-22T00:02:04.824650+0000","last_fullsized":"2024-04-22T00:02:04.824650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888041+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-22T00:02:04.888041+0000","last_peered":"2024-04-22T00:02:04.888041+0000","last_clean":"2024-04-22T00:02:04.888041+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-22T00:02:04.888041+0000","last_undegraded":"2024-04-22T00:02:04.888041+0000","last_fullsized":"2024-04-22T00:02:04.888041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888383+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-22T00:02:04.888383+0000","last_peered":"2024-04-22T00:02:04.888383+0000","last_clean":"2024-04-22T00:02:04.888383+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-22T00:02:04.888383+0000","last_undegraded":"2024-04-22T00:02:04.888383+0000","last_fullsized":"2024-04-22T00:02:04.888383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888517+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:02:04.888517+0000","last_peered":"2024-04-22T00:02:04.888517+0000","last_clean":"2024-04-22T00:02:04.888517+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:02:04.888517+0000","last_undegraded":"2024-04-22T00:02:04.888517+0000","last_fullsized":"2024-04-22T00:02:04.888517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2010,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888876+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-22T00:02:04.888876+0000","last_peered":"2024-04-22T00:02:04.888876+0000","last_clean":"2024-04-22T00:02:04.888876+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-22T00:02:04.888876+0000","last_undegraded":"2024-04-22T00:02:04.888876+0000","last_fullsized":"2024-04-22T00:02:04.888876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886998+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-22T00:02:04.886998+0000","last_peered":"2024-04-22T00:02:04.886998+0000","last_clean":"2024-04-22T00:02:04.886998+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-22T00:02:04.886998+0000","last_undegraded":"2024-04-22T00:02:04.886998+0000","last_fullsized":"2024-04-22T00:02:04.886998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.027186+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:02:05.027186+0000","last_peered":"2024-04-22T00:02:05.027186+0000","last_clean":"2024-04-22T00:02:05.027186+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:02:05.027186+0000","last_undegraded":"2024-04-22T00:02:05.027186+0000","last_fullsized":"2024-04-22T00:02:05.027186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888652+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-22T00:02:04.888652+0000","last_peered":"2024-04-22T00:02:04.888652+0000","last_clean":"2024-04-22T00:02:04.888652+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-22T00:02:04.888652+0000","last_undegraded":"2024-04-22T00:02:04.888652+0000","last_fullsized":"2024-04-22T00:02:04.888652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886524+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-22T00:02:04.886524+0000","last_peered":"2024-04-22T00:02:04.886524+0000","last_clean":"2024-04-22T00:02:04.886524+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-22T00:02:04.886524+0000","last_undegraded":"2024-04-22T00:02:04.886524+0000","last_fullsized":"2024-04-22T00:02:04.886524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887998+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-22T00:02:04.887998+0000","last_peered":"2024-04-22T00:02:04.887998+0000","last_clean":"2024-04-22T00:02:04.887998+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-22T00:02:04.887998+0000","last_undegraded":"2024-04-22T00:02:04.887998+0000","last_fullsized":"2024-04-22T00:02:04.887998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887227+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-22T00:02:04.887227+0000","last_peered":"2024-04-22T00:02:04.887227+0000","last_clean":"2024-04-22T00:02:04.887227+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-22T00:02:04.887227+0000","last_undegraded":"2024-04-22T00:02:04.887227+0000","last_fullsized":"2024-04-22T00:02:04.887227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886684+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-22T00:02:04.886684+0000","last_peered":"2024-04-22T00:02:04.886684+0000","last_clean":"2024-04-22T00:02:04.886684+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-22T00:02:04.886684+0000","last_undegraded":"2024-04-22T00:02:04.886684+0000","last_fullsized":"2024-04-22T00:02:04.886684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886866+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-22T00:02:04.886866+0000","last_peered":"2024-04-22T00:02:04.886866+0000","last_clean":"2024-04-22T00:02:04.886866+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-22T00:02:04.886866+0000","last_undegraded":"2024-04-22T00:02:04.886866+0000","last_fullsized":"2024-04-22T00:02:04.886866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888285+0000","last_change":"2024-04-22T00:00:27.362810+0000","last_active":"2024-04-22T00:02:04.888285+0000","last_peered":"2024-04-22T00:02:04.888285+0000","last_clean":"2024-04-22T00:02:04.888285+0000","last_became_active":"2024-04-22T00:00:27.362380+0000","last_became_peered":"2024-04-22T00:00:27.362380+0000","last_unstale":"2024-04-22T00:02:04.888285+0000","last_undegraded":"2024-04-22T00:02:04.888285+0000","last_fullsized":"2024-04-22T00:02:04.888285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:09:30.391961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888096+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-22T00:02:04.888096+0000","last_peered":"2024-04-22T00:02:04.888096+0000","last_clean":"2024-04-22T00:02:04.888096+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-22T00:02:04.888096+0000","last_undegraded":"2024-04-22T00:02:04.888096+0000","last_fullsized":"2024-04-22T00:02:04.888096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887929+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-22T00:02:04.887929+0000","last_peered":"2024-04-22T00:02:04.887929+0000","last_clean":"2024-04-22T00:02:04.887929+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-22T00:02:04.887929+0000","last_undegraded":"2024-04-22T00:02:04.887929+0000","last_fullsized":"2024-04-22T00:02:04.887929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887246+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-22T00:02:04.887246+0000","last_peered":"2024-04-22T00:02:04.887246+0000","last_clean":"2024-04-22T00:02:04.887246+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-22T00:02:04.887246+0000","last_undegraded":"2024-04-22T00:02:04.887246+0000","last_fullsized":"2024-04-22T00:02:04.887246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889156+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-22T00:02:04.889156+0000","last_peered":"2024-04-22T00:02:04.889156+0000","last_clean":"2024-04-22T00:02:04.889156+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-22T00:02:04.889156+0000","last_undegraded":"2024-04-22T00:02:04.889156+0000","last_fullsized":"2024-04-22T00:02:04.889156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886857+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-22T00:02:04.886857+0000","last_peered":"2024-04-22T00:02:04.886857+0000","last_clean":"2024-04-22T00:02:04.886857+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-22T00:02:04.886857+0000","last_undegraded":"2024-04-22T00:02:04.886857+0000","last_fullsized":"2024-04-22T00:02:04.886857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887969+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-22T00:02:04.887969+0000","last_peered":"2024-04-22T00:02:04.887969+0000","last_clean":"2024-04-22T00:02:04.887969+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-22T00:02:04.887969+0000","last_undegraded":"2024-04-22T00:02:04.887969+0000","last_fullsized":"2024-04-22T00:02:04.887969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887229+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-22T00:02:04.887229+0000","last_peered":"2024-04-22T00:02:04.887229+0000","last_clean":"2024-04-22T00:02:04.887229+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-22T00:02:04.887229+0000","last_undegraded":"2024-04-22T00:02:04.887229+0000","last_fullsized":"2024-04-22T00:02:04.887229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887469+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-22T00:02:04.887469+0000","last_peered":"2024-04-22T00:02:04.887469+0000","last_clean":"2024-04-22T00:02:04.887469+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-22T00:02:04.887469+0000","last_undegraded":"2024-04-22T00:02:04.887469+0000","last_fullsized":"2024-04-22T00:02:04.887469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887415+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-22T00:02:04.887415+0000","last_peered":"2024-04-22T00:02:04.887415+0000","last_clean":"2024-04-22T00:02:04.887415+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-22T00:02:04.887415+0000","last_undegraded":"2024-04-22T00:02:04.887415+0000","last_fullsized":"2024-04-22T00:02:04.887415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602828,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1346,"num_read_kb":792,"num_write":377,"num_write_kb":1474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":512193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1888256,"data_stored":1303721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":904387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1458176,"data_stored":908749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28648,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716504,"statfs":{"total":95995035648,"available":95965700096,"internally_reserved":0,"allocated":1667072,"data_stored":1108187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":577532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":508971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":571655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1031199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":511672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1740800,"data_stored":1165197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:02:10.372 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T00:02:10.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T00:02:10.404 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64524]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:10.405 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:10.405 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64524]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:10.405 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64524]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:10.405 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64527]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.517+0000 7efef85b8640 1 -- 172.21.15.38:0/3222549625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef010bd10 msgr2=0x7efef010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.517+0000 7efef85b8640 1 --2- 172.21.15.38:0/3222549625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef010bd10 0x7efef010e1b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efee8009ca0 tx=0x7efee802c230 comp rx=0 tx=0).stop 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- 172.21.15.38:0/3222549625 shutdown_connections 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- 172.21.15.38:0/3222549625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef010bd10 0x7efef010e1b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- 172.21.15.38:0/3222549625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01032f0 0x7efef010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- 172.21.15.38:0/3222549625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef01029d0 0x7efef0102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- 172.21.15.38:0/3222549625 >> 172.21.15.38:0/3222549625 conn(0x7efef0077740 msgr2=0x7efef0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- 172.21.15.38:0/3222549625 shutdown_connections 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- 172.21.15.38:0/3222549625 wait complete. 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 Processor -- start 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- start start 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01029d0 0x7efef0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef010bd10 0x7efef019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efef0111b10 con 0x7efef01032f0 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efef0111990 con 0x7efef01029d0 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.518+0000 7efef85b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efef0111c90 con 0x7efef010bd10 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40156/0 (socket says 172.21.15.38:40156) 2024-04-22T00:02:10.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 -- 172.21.15.38:0/2906680691 learned_addr learned my addr 172.21.15.38:0/2906680691 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef6b55640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef010bd10 0x7efef019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef6354640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01029d0 0x7efef0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 -- 172.21.15.38:0/2906680691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef010bd10 msgr2=0x7efef019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef010bd10 0x7efef019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 -- 172.21.15.38:0/2906680691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01029d0 msgr2=0x7efef0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01029d0 0x7efef0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 -- 172.21.15.38:0/2906680691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef019cf70 con 0x7efef01032f0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef5b53640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9f0 tx=0x7efee000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee000cd80 con 0x7efef01032f0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efee0010910 con 0x7efef01032f0 2024-04-22T00:02:10.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef019d260 con 0x7efef01032f0 2024-04-22T00:02:10.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0018990 con 0x7efef01032f0 2024-04-22T00:02:10.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.519+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef01a4af0 con 0x7efef01032f0 2024-04-22T00:02:10.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.520+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef0068560 con 0x7efef01032f0 2024-04-22T00:02:10.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.521+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7efee0010ac0 con 0x7efef01032f0 2024-04-22T00:02:10.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.522+0000 7efee77fe640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 0x7efedc0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.522+0000 7efef6354640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 0x7efedc0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.523+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efee00d0110 con 0x7efef01032f0 2024-04-22T00:02:10.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.523+0000 7efef6354640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 0x7efedc0a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efeec009b60 tx=0x7efeec005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.526+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efee009c7a0 con 0x7efef01032f0 2024-04-22T00:02:10.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.713+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efef0061b80 con 0x7efedc0a2ea0 2024-04-22T00:02:10.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.723+0000 7efee77fe640 1 -- 172.21.15.38:0/2906680691 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339434 (secure 0 0 0) 0x7efef0061b80 con 0x7efedc0a2ea0 2024-04-22T00:02:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:10.721 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.726+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 msgr2=0x7efedc0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.726+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 0x7efedc0a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efeec009b60 tx=0x7efeec005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.726+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 msgr2=0x7efef0198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.726+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9f0 tx=0x7efee000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 shutdown_connections 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efedc0a2ea0 0x7efedc0a5360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef010bd10 0x7efef019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef01032f0 0x7efef0198510 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 --2- 172.21.15.38:0/2906680691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef01029d0 0x7efef0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 >> 172.21.15.38:0/2906680691 conn(0x7efef0077740 msgr2=0x7efef00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 shutdown_connections 2024-04-22T00:02:10.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.727+0000 7efef85b8640 1 -- 172.21.15.38:0/2906680691 wait complete. 2024-04-22T00:02:10.730 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T00:02:08.838548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193430,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1482,"num_read_kb":911,"num_write":560,"num_write_kb":4414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181082,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":945,"ondisk_log_size":945,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365116,"kb_used_data":15924,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576708,"statfs":{"total":1151940427776,"available":1151566548992,"internally_reserved":0,"allocated":16306176,"data_stored":9610138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986866"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888347+0000","last_change":"2024-04-21T23:59:36.263369+0000","last_active":"2024-04-22T00:02:04.888347+0000","last_peered":"2024-04-22T00:02:04.888347+0000","last_clean":"2024-04-22T00:02:04.888347+0000","last_became_active":"2024-04-21T23:59:36.262840+0000","last_became_peered":"2024-04-21T23:59:36.262840+0000","last_unstale":"2024-04-22T00:02:04.888347+0000","last_undegraded":"2024-04-22T00:02:04.888347+0000","last_fullsized":"2024-04-22T00:02:04.888347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:56:00.972937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888466+0000","last_change":"2024-04-21T23:59:36.252781+0000","last_active":"2024-04-22T00:02:04.888466+0000","last_peered":"2024-04-22T00:02:04.888466+0000","last_clean":"2024-04-22T00:02:04.888466+0000","last_became_active":"2024-04-21T23:59:36.252471+0000","last_became_peered":"2024-04-21T23:59:36.252471+0000","last_unstale":"2024-04-22T00:02:04.888466+0000","last_undegraded":"2024-04-22T00:02:04.888466+0000","last_fullsized":"2024-04-22T00:02:04.888466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:26.482638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888688+0000","last_change":"2024-04-21T23:59:36.256200+0000","last_active":"2024-04-22T00:02:04.888688+0000","last_peered":"2024-04-22T00:02:04.888688+0000","last_clean":"2024-04-22T00:02:04.888688+0000","last_became_active":"2024-04-21T23:59:36.255907+0000","last_became_peered":"2024-04-21T23:59:36.255907+0000","last_unstale":"2024-04-22T00:02:04.888688+0000","last_undegraded":"2024-04-22T00:02:04.888688+0000","last_fullsized":"2024-04-22T00:02:04.888688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:27:12.643970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825019+0000","last_change":"2024-04-21T23:59:36.257895+0000","last_active":"2024-04-22T00:02:04.825019+0000","last_peered":"2024-04-22T00:02:04.825019+0000","last_clean":"2024-04-22T00:02:04.825019+0000","last_became_active":"2024-04-21T23:59:36.257593+0000","last_became_peered":"2024-04-21T23:59:36.257593+0000","last_unstale":"2024-04-22T00:02:04.825019+0000","last_undegraded":"2024-04-22T00:02:04.825019+0000","last_fullsized":"2024-04-22T00:02:04.825019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:08:53.137906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824719+0000","last_change":"2024-04-21T23:59:36.252589+0000","last_active":"2024-04-22T00:02:04.824719+0000","last_peered":"2024-04-22T00:02:04.824719+0000","last_clean":"2024-04-22T00:02:04.824719+0000","last_became_active":"2024-04-21T23:59:36.252254+0000","last_became_peered":"2024-04-21T23:59:36.252254+0000","last_unstale":"2024-04-22T00:02:04.824719+0000","last_undegraded":"2024-04-22T00:02:04.824719+0000","last_fullsized":"2024-04-22T00:02:04.824719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:05.782165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887915+0000","last_change":"2024-04-21T23:59:36.261834+0000","last_active":"2024-04-22T00:02:04.887915+0000","last_peered":"2024-04-22T00:02:04.887915+0000","last_clean":"2024-04-22T00:02:04.887915+0000","last_became_active":"2024-04-21T23:59:36.261482+0000","last_became_peered":"2024-04-21T23:59:36.261482+0000","last_unstale":"2024-04-22T00:02:04.887915+0000","last_undegraded":"2024-04-22T00:02:04.887915+0000","last_fullsized":"2024-04-22T00:02:04.887915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:39:43.492484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887595+0000","last_change":"2024-04-21T23:59:36.263776+0000","last_active":"2024-04-22T00:02:04.887595+0000","last_peered":"2024-04-22T00:02:04.887595+0000","last_clean":"2024-04-22T00:02:04.887595+0000","last_became_active":"2024-04-21T23:59:36.261422+0000","last_became_peered":"2024-04-21T23:59:36.261422+0000","last_unstale":"2024-04-22T00:02:04.887595+0000","last_undegraded":"2024-04-22T00:02:04.887595+0000","last_fullsized":"2024-04-22T00:02:04.887595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:33:26.657847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887687+0000","last_change":"2024-04-21T23:59:36.260170+0000","last_active":"2024-04-22T00:02:04.887687+0000","last_peered":"2024-04-22T00:02:04.887687+0000","last_clean":"2024-04-22T00:02:04.887687+0000","last_became_active":"2024-04-21T23:59:36.259723+0000","last_became_peered":"2024-04-21T23:59:36.259723+0000","last_unstale":"2024-04-22T00:02:04.887687+0000","last_undegraded":"2024-04-22T00:02:04.887687+0000","last_fullsized":"2024-04-22T00:02:04.887687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:56:13.942343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888505+0000","last_change":"2024-04-21T23:59:36.259330+0000","last_active":"2024-04-22T00:02:04.888505+0000","last_peered":"2024-04-22T00:02:04.888505+0000","last_clean":"2024-04-22T00:02:04.888505+0000","last_became_active":"2024-04-21T23:59:36.258999+0000","last_became_peered":"2024-04-21T23:59:36.258999+0000","last_unstale":"2024-04-22T00:02:04.888505+0000","last_undegraded":"2024-04-22T00:02:04.888505+0000","last_fullsized":"2024-04-22T00:02:04.888505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:48:24.812509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888417+0000","last_change":"2024-04-21T23:59:36.262910+0000","last_active":"2024-04-22T00:02:04.888417+0000","last_peered":"2024-04-22T00:02:04.888417+0000","last_clean":"2024-04-22T00:02:04.888417+0000","last_became_active":"2024-04-21T23:59:36.262093+0000","last_became_peered":"2024-04-21T23:59:36.262093+0000","last_unstale":"2024-04-22T00:02:04.888417+0000","last_undegraded":"2024-04-22T00:02:04.888417+0000","last_fullsized":"2024-04-22T00:02:04.888417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:22:06.649156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888811+0000","last_change":"2024-04-21T23:59:36.260867+0000","last_active":"2024-04-22T00:02:04.888811+0000","last_peered":"2024-04-22T00:02:04.888811+0000","last_clean":"2024-04-22T00:02:04.888811+0000","last_became_active":"2024-04-21T23:59:36.260320+0000","last_became_peered":"2024-04-21T23:59:36.260320+0000","last_unstale":"2024-04-22T00:02:04.888811+0000","last_undegraded":"2024-04-22T00:02:04.888811+0000","last_fullsized":"2024-04-22T00:02:04.888811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:38:51.392912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824939+0000","last_change":"2024-04-21T23:59:36.256067+0000","last_active":"2024-04-22T00:02:04.824939+0000","last_peered":"2024-04-22T00:02:04.824939+0000","last_clean":"2024-04-22T00:02:04.824939+0000","last_became_active":"2024-04-21T23:59:36.255748+0000","last_became_peered":"2024-04-21T23:59:36.255748+0000","last_unstale":"2024-04-22T00:02:04.824939+0000","last_undegraded":"2024-04-22T00:02:04.824939+0000","last_fullsized":"2024-04-22T00:02:04.824939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:35.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887447+0000","last_change":"2024-04-22T00:00:27.492882+0000","last_active":"2024-04-22T00:02:04.887447+0000","last_peered":"2024-04-22T00:02:04.887447+0000","last_clean":"2024-04-22T00:02:04.887447+0000","last_became_active":"2024-04-22T00:00:27.492451+0000","last_became_peered":"2024-04-22T00:00:27.492451+0000","last_unstale":"2024-04-22T00:02:04.887447+0000","last_undegraded":"2024-04-22T00:02:04.887447+0000","last_fullsized":"2024-04-22T00:02:04.887447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:37:10.283575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887109+0000","last_change":"2024-04-21T23:59:36.259461+0000","last_active":"2024-04-22T00:02:04.887109+0000","last_peered":"2024-04-22T00:02:04.887109+0000","last_clean":"2024-04-22T00:02:04.887109+0000","last_became_active":"2024-04-21T23:59:36.259153+0000","last_became_peered":"2024-04-21T23:59:36.259153+0000","last_unstale":"2024-04-22T00:02:04.887109+0000","last_undegraded":"2024-04-22T00:02:04.887109+0000","last_fullsized":"2024-04-22T00:02:04.887109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:11.588539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887176+0000","last_change":"2024-04-21T23:59:36.256916+0000","last_active":"2024-04-22T00:02:04.887176+0000","last_peered":"2024-04-22T00:02:04.887176+0000","last_clean":"2024-04-22T00:02:04.887176+0000","last_became_active":"2024-04-21T23:59:36.256619+0000","last_became_peered":"2024-04-21T23:59:36.256619+0000","last_unstale":"2024-04-22T00:02:04.887176+0000","last_undegraded":"2024-04-22T00:02:04.887176+0000","last_fullsized":"2024-04-22T00:02:04.887176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:50:42.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887188+0000","last_change":"2024-04-21T23:59:36.262969+0000","last_active":"2024-04-22T00:02:04.887188+0000","last_peered":"2024-04-22T00:02:04.887188+0000","last_clean":"2024-04-22T00:02:04.887188+0000","last_became_active":"2024-04-21T23:59:36.262630+0000","last_became_peered":"2024-04-21T23:59:36.262630+0000","last_unstale":"2024-04-22T00:02:04.887188+0000","last_undegraded":"2024-04-22T00:02:04.887188+0000","last_fullsized":"2024-04-22T00:02:04.887188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:17:11.901652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888089+0000","last_change":"2024-04-21T23:59:36.263741+0000","last_active":"2024-04-22T00:02:04.888089+0000","last_peered":"2024-04-22T00:02:04.888089+0000","last_clean":"2024-04-22T00:02:04.888089+0000","last_became_active":"2024-04-21T23:59:36.263416+0000","last_became_peered":"2024-04-21T23:59:36.263416+0000","last_unstale":"2024-04-22T00:02:04.888089+0000","last_undegraded":"2024-04-22T00:02:04.888089+0000","last_fullsized":"2024-04-22T00:02:04.888089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:12:10.217896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825401+0000","last_change":"2024-04-21T23:59:36.251736+0000","last_active":"2024-04-22T00:02:04.825401+0000","last_peered":"2024-04-22T00:02:04.825401+0000","last_clean":"2024-04-22T00:02:04.825401+0000","last_became_active":"2024-04-21T23:59:36.251309+0000","last_became_peered":"2024-04-21T23:59:36.251309+0000","last_unstale":"2024-04-22T00:02:04.825401+0000","last_undegraded":"2024-04-22T00:02:04.825401+0000","last_fullsized":"2024-04-22T00:02:04.825401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:55:34.867782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887717+0000","last_change":"2024-04-21T23:59:36.261820+0000","last_active":"2024-04-22T00:02:04.887717+0000","last_peered":"2024-04-22T00:02:04.887717+0000","last_clean":"2024-04-22T00:02:04.887717+0000","last_became_active":"2024-04-21T23:59:36.260101+0000","last_became_peered":"2024-04-21T23:59:36.260101+0000","last_unstale":"2024-04-22T00:02:04.887717+0000","last_undegraded":"2024-04-22T00:02:04.887717+0000","last_fullsized":"2024-04-22T00:02:04.887717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:53:39.498934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887268+0000","last_change":"2024-04-21T23:59:40.418635+0000","last_active":"2024-04-22T00:02:04.887268+0000","last_peered":"2024-04-22T00:02:04.887268+0000","last_clean":"2024-04-22T00:02:04.887268+0000","last_became_active":"2024-04-21T23:59:40.418099+0000","last_became_peered":"2024-04-21T23:59:40.418099+0000","last_unstale":"2024-04-22T00:02:04.887268+0000","last_undegraded":"2024-04-22T00:02:04.887268+0000","last_fullsized":"2024-04-22T00:02:04.887268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:49.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":166,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065127+0000","last_change":"2024-04-21T23:59:35.613684+0000","last_active":"2024-04-22T00:02:05.065127+0000","last_peered":"2024-04-22T00:02:05.065127+0000","last_clean":"2024-04-22T00:02:05.065127+0000","last_became_active":"2024-04-21T23:59:35.613312+0000","last_became_peered":"2024-04-21T23:59:35.613312+0000","last_unstale":"2024-04-22T00:02:05.065127+0000","last_undegraded":"2024-04-22T00:02:05.065127+0000","last_fullsized":"2024-04-22T00:02:05.065127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:15:56.952612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888904+0000","last_change":"2024-04-21T23:59:36.258090+0000","last_active":"2024-04-22T00:02:04.888904+0000","last_peered":"2024-04-22T00:02:04.888904+0000","last_clean":"2024-04-22T00:02:04.888904+0000","last_became_active":"2024-04-21T23:59:36.257762+0000","last_became_peered":"2024-04-21T23:59:36.257762+0000","last_unstale":"2024-04-22T00:02:04.888904+0000","last_undegraded":"2024-04-22T00:02:04.888904+0000","last_fullsized":"2024-04-22T00:02:04.888904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:41:04.150289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824507+0000","last_change":"2024-04-21T23:59:40.412256+0000","last_active":"2024-04-22T00:02:04.824507+0000","last_peered":"2024-04-22T00:02:04.824507+0000","last_clean":"2024-04-22T00:02:04.824507+0000","last_became_active":"2024-04-21T23:59:40.411825+0000","last_became_peered":"2024-04-21T23:59:40.411825+0000","last_unstale":"2024-04-22T00:02:04.824507+0000","last_undegraded":"2024-04-22T00:02:04.824507+0000","last_fullsized":"2024-04-22T00:02:04.824507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:00:54.654238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889096+0000","last_change":"2024-04-21T23:59:35.621164+0000","last_active":"2024-04-22T00:02:04.889096+0000","last_peered":"2024-04-22T00:02:04.889096+0000","last_clean":"2024-04-22T00:02:04.889096+0000","last_became_active":"2024-04-21T23:59:35.619651+0000","last_became_peered":"2024-04-21T23:59:35.619651+0000","last_unstale":"2024-04-22T00:02:04.889096+0000","last_undegraded":"2024-04-22T00:02:04.889096+0000","last_fullsized":"2024-04-22T00:02:04.889096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:00:09.709804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888771+0000","last_change":"2024-04-21T23:59:36.249482+0000","last_active":"2024-04-22T00:02:04.888771+0000","last_peered":"2024-04-22T00:02:04.888771+0000","last_clean":"2024-04-22T00:02:04.888771+0000","last_became_active":"2024-04-21T23:59:36.249116+0000","last_became_peered":"2024-04-21T23:59:36.249116+0000","last_unstale":"2024-04-22T00:02:04.888771+0000","last_undegraded":"2024-04-22T00:02:04.888771+0000","last_fullsized":"2024-04-22T00:02:04.888771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:00:00.755306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824795+0000","last_change":"2024-04-22T00:00:27.492457+0000","last_active":"2024-04-22T00:02:04.824795+0000","last_peered":"2024-04-22T00:02:04.824795+0000","last_clean":"2024-04-22T00:02:04.824795+0000","last_became_active":"2024-04-22T00:00:27.491828+0000","last_became_peered":"2024-04-22T00:00:27.491828+0000","last_unstale":"2024-04-22T00:02:04.824795+0000","last_undegraded":"2024-04-22T00:02:04.824795+0000","last_fullsized":"2024-04-22T00:02:04.824795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:09:18.512103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'13","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824854+0000","last_change":"2024-04-21T23:59:35.617535+0000","last_active":"2024-04-22T00:02:04.824854+0000","last_peered":"2024-04-22T00:02:04.824854+0000","last_clean":"2024-04-22T00:02:04.824854+0000","last_became_active":"2024-04-21T23:59:35.617053+0000","last_became_peered":"2024-04-21T23:59:35.617053+0000","last_unstale":"2024-04-22T00:02:04.824854+0000","last_undegraded":"2024-04-22T00:02:04.824854+0000","last_fullsized":"2024-04-22T00:02:04.824854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:12.904974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887837+0000","last_change":"2024-04-21T23:59:36.250205+0000","last_active":"2024-04-22T00:02:04.887837+0000","last_peered":"2024-04-22T00:02:04.887837+0000","last_clean":"2024-04-22T00:02:04.887837+0000","last_became_active":"2024-04-21T23:59:36.249384+0000","last_became_peered":"2024-04-21T23:59:36.249384+0000","last_unstale":"2024-04-22T00:02:04.887837+0000","last_undegraded":"2024-04-22T00:02:04.887837+0000","last_fullsized":"2024-04-22T00:02:04.887837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:15:05.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824567+0000","last_change":"2024-04-21T23:59:40.413754+0000","last_active":"2024-04-22T00:02:04.824567+0000","last_peered":"2024-04-22T00:02:04.824567+0000","last_clean":"2024-04-22T00:02:04.824567+0000","last_became_active":"2024-04-21T23:59:40.412964+0000","last_became_peered":"2024-04-21T23:59:40.412964+0000","last_unstale":"2024-04-22T00:02:04.824567+0000","last_undegraded":"2024-04-22T00:02:04.824567+0000","last_fullsized":"2024-04-22T00:02:04.824567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:59.391991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824717+0000","last_change":"2024-04-21T23:59:35.610383+0000","last_active":"2024-04-22T00:02:04.824717+0000","last_peered":"2024-04-22T00:02:04.824717+0000","last_clean":"2024-04-22T00:02:04.824717+0000","last_became_active":"2024-04-21T23:59:35.610069+0000","last_became_peered":"2024-04-21T23:59:35.610069+0000","last_unstale":"2024-04-22T00:02:04.824717+0000","last_undegraded":"2024-04-22T00:02:04.824717+0000","last_fullsized":"2024-04-22T00:02:04.824717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:46:38.481632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888097+0000","last_change":"2024-04-21T23:59:36.246771+0000","last_active":"2024-04-22T00:02:04.888097+0000","last_peered":"2024-04-22T00:02:04.888097+0000","last_clean":"2024-04-22T00:02:04.888097+0000","last_became_active":"2024-04-21T23:59:36.246460+0000","last_became_peered":"2024-04-21T23:59:36.246460+0000","last_unstale":"2024-04-22T00:02:04.888097+0000","last_undegraded":"2024-04-22T00:02:04.888097+0000","last_fullsized":"2024-04-22T00:02:04.888097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:21:29.153220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887607+0000","last_change":"2024-04-21T23:59:40.413400+0000","last_active":"2024-04-22T00:02:04.887607+0000","last_peered":"2024-04-22T00:02:04.887607+0000","last_clean":"2024-04-22T00:02:04.887607+0000","last_became_active":"2024-04-21T23:59:40.412849+0000","last_became_peered":"2024-04-21T23:59:40.412849+0000","last_unstale":"2024-04-22T00:02:04.887607+0000","last_undegraded":"2024-04-22T00:02:04.887607+0000","last_fullsized":"2024-04-22T00:02:04.887607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:41:33.341511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":158,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.444015+0000","last_change":"2024-04-21T23:59:35.615014+0000","last_active":"2024-04-22T00:02:05.444015+0000","last_peered":"2024-04-22T00:02:05.444015+0000","last_clean":"2024-04-22T00:02:05.444015+0000","last_became_active":"2024-04-21T23:59:35.614713+0000","last_became_peered":"2024-04-21T23:59:35.614713+0000","last_unstale":"2024-04-22T00:02:05.444015+0000","last_undegraded":"2024-04-22T00:02:05.444015+0000","last_fullsized":"2024-04-22T00:02:05.444015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:18:14.881321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824620+0000","last_change":"2024-04-21T23:59:36.259892+0000","last_active":"2024-04-22T00:02:04.824620+0000","last_peered":"2024-04-22T00:02:04.824620+0000","last_clean":"2024-04-22T00:02:04.824620+0000","last_became_active":"2024-04-21T23:59:36.259576+0000","last_became_peered":"2024-04-21T23:59:36.259576+0000","last_unstale":"2024-04-22T00:02:04.824620+0000","last_undegraded":"2024-04-22T00:02:04.824620+0000","last_fullsized":"2024-04-22T00:02:04.824620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:33:01.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887593+0000","last_change":"2024-04-21T23:59:40.421112+0000","last_active":"2024-04-22T00:02:04.887593+0000","last_peered":"2024-04-22T00:02:04.887593+0000","last_clean":"2024-04-22T00:02:04.887593+0000","last_became_active":"2024-04-21T23:59:40.420666+0000","last_became_peered":"2024-04-21T23:59:40.420666+0000","last_unstale":"2024-04-22T00:02:04.887593+0000","last_undegraded":"2024-04-22T00:02:04.887593+0000","last_fullsized":"2024-04-22T00:02:04.887593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:48:16.579001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887797+0000","last_change":"2024-04-21T23:59:35.611819+0000","last_active":"2024-04-22T00:02:04.887797+0000","last_peered":"2024-04-22T00:02:04.887797+0000","last_clean":"2024-04-22T00:02:04.887797+0000","last_became_active":"2024-04-21T23:59:35.611494+0000","last_became_peered":"2024-04-21T23:59:35.611494+0000","last_unstale":"2024-04-22T00:02:04.887797+0000","last_undegraded":"2024-04-22T00:02:04.887797+0000","last_fullsized":"2024-04-22T00:02:04.887797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:07:23.390838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887722+0000","last_change":"2024-04-21T23:59:36.253691+0000","last_active":"2024-04-22T00:02:04.887722+0000","last_peered":"2024-04-22T00:02:04.887722+0000","last_clean":"2024-04-22T00:02:04.887722+0000","last_became_active":"2024-04-21T23:59:36.253254+0000","last_became_peered":"2024-04-21T23:59:36.253254+0000","last_unstale":"2024-04-22T00:02:04.887722+0000","last_undegraded":"2024-04-22T00:02:04.887722+0000","last_fullsized":"2024-04-22T00:02:04.887722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:15:27.199574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887618+0000","last_change":"2024-04-21T23:59:40.415053+0000","last_active":"2024-04-22T00:02:04.887618+0000","last_peered":"2024-04-22T00:02:04.887618+0000","last_clean":"2024-04-22T00:02:04.887618+0000","last_became_active":"2024-04-21T23:59:40.414621+0000","last_became_peered":"2024-04-21T23:59:40.414621+0000","last_unstale":"2024-04-22T00:02:04.887618+0000","last_undegraded":"2024-04-22T00:02:04.887618+0000","last_fullsized":"2024-04-22T00:02:04.887618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:37.132416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887957+0000","last_change":"2024-04-21T23:59:35.611100+0000","last_active":"2024-04-22T00:02:04.887957+0000","last_peered":"2024-04-22T00:02:04.887957+0000","last_clean":"2024-04-22T00:02:04.887957+0000","last_became_active":"2024-04-21T23:59:35.610686+0000","last_became_peered":"2024-04-21T23:59:35.610686+0000","last_unstale":"2024-04-22T00:02:04.887957+0000","last_undegraded":"2024-04-22T00:02:04.887957+0000","last_fullsized":"2024-04-22T00:02:04.887957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:44:17.211209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887527+0000","last_change":"2024-04-21T23:59:36.260573+0000","last_active":"2024-04-22T00:02:04.887527+0000","last_peered":"2024-04-22T00:02:04.887527+0000","last_clean":"2024-04-22T00:02:04.887527+0000","last_became_active":"2024-04-21T23:59:36.260218+0000","last_became_peered":"2024-04-21T23:59:36.260218+0000","last_unstale":"2024-04-22T00:02:04.887527+0000","last_undegraded":"2024-04-22T00:02:04.887527+0000","last_fullsized":"2024-04-22T00:02:04.887527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:41.702073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888356+0000","last_change":"2024-04-21T23:59:40.421455+0000","last_active":"2024-04-22T00:02:04.888356+0000","last_peered":"2024-04-22T00:02:04.888356+0000","last_clean":"2024-04-22T00:02:04.888356+0000","last_became_active":"2024-04-21T23:59:40.421006+0000","last_became_peered":"2024-04-21T23:59:40.421006+0000","last_unstale":"2024-04-22T00:02:04.888356+0000","last_undegraded":"2024-04-22T00:02:04.888356+0000","last_fullsized":"2024-04-22T00:02:04.888356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:30:17.572159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886900+0000","last_change":"2024-04-21T23:59:35.614165+0000","last_active":"2024-04-22T00:02:04.886900+0000","last_peered":"2024-04-22T00:02:04.886900+0000","last_clean":"2024-04-22T00:02:04.886900+0000","last_became_active":"2024-04-21T23:59:35.613853+0000","last_became_peered":"2024-04-21T23:59:35.613853+0000","last_unstale":"2024-04-22T00:02:04.886900+0000","last_undegraded":"2024-04-22T00:02:04.886900+0000","last_fullsized":"2024-04-22T00:02:04.886900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:27:01.680023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824890+0000","last_change":"2024-04-21T23:59:36.251657+0000","last_active":"2024-04-22T00:02:04.824890+0000","last_peered":"2024-04-22T00:02:04.824890+0000","last_clean":"2024-04-22T00:02:04.824890+0000","last_became_active":"2024-04-21T23:59:36.251258+0000","last_became_peered":"2024-04-21T23:59:36.251258+0000","last_unstale":"2024-04-22T00:02:04.824890+0000","last_undegraded":"2024-04-22T00:02:04.824890+0000","last_fullsized":"2024-04-22T00:02:04.824890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:24:24.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888253+0000","last_change":"2024-04-21T23:59:40.416718+0000","last_active":"2024-04-22T00:02:04.888253+0000","last_peered":"2024-04-22T00:02:04.888253+0000","last_clean":"2024-04-22T00:02:04.888253+0000","last_became_active":"2024-04-21T23:59:40.416315+0000","last_became_peered":"2024-04-21T23:59:40.416315+0000","last_unstale":"2024-04-22T00:02:04.888253+0000","last_undegraded":"2024-04-22T00:02:04.888253+0000","last_fullsized":"2024-04-22T00:02:04.888253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:57.143478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":278,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.484350+0000","last_change":"2024-04-21T23:59:35.610731+0000","last_active":"2024-04-22T00:02:05.484350+0000","last_peered":"2024-04-22T00:02:05.484350+0000","last_clean":"2024-04-22T00:02:05.484350+0000","last_became_active":"2024-04-21T23:59:35.610391+0000","last_became_peered":"2024-04-21T23:59:35.610391+0000","last_unstale":"2024-04-22T00:02:05.484350+0000","last_undegraded":"2024-04-22T00:02:05.484350+0000","last_fullsized":"2024-04-22T00:02:05.484350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:31:26.535373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888635+0000","last_change":"2024-04-21T23:59:36.249788+0000","last_active":"2024-04-22T00:02:04.888635+0000","last_peered":"2024-04-22T00:02:04.888635+0000","last_clean":"2024-04-22T00:02:04.888635+0000","last_became_active":"2024-04-21T23:59:36.248523+0000","last_became_peered":"2024-04-21T23:59:36.248523+0000","last_unstale":"2024-04-22T00:02:04.888635+0000","last_undegraded":"2024-04-22T00:02:04.888635+0000","last_fullsized":"2024-04-22T00:02:04.888635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:59:53.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824672+0000","last_change":"2024-04-21T23:59:40.406904+0000","last_active":"2024-04-22T00:02:04.824672+0000","last_peered":"2024-04-22T00:02:04.824672+0000","last_clean":"2024-04-22T00:02:04.824672+0000","last_became_active":"2024-04-21T23:59:40.406508+0000","last_became_peered":"2024-04-21T23:59:40.406508+0000","last_unstale":"2024-04-22T00:02:04.824672+0000","last_undegraded":"2024-04-22T00:02:04.824672+0000","last_fullsized":"2024-04-22T00:02:04.824672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:13:23.077819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888851+0000","last_change":"2024-04-21T23:59:35.615593+0000","last_active":"2024-04-22T00:02:04.888851+0000","last_peered":"2024-04-22T00:02:04.888851+0000","last_clean":"2024-04-22T00:02:04.888851+0000","last_became_active":"2024-04-21T23:59:35.615050+0000","last_became_peered":"2024-04-21T23:59:35.615050+0000","last_unstale":"2024-04-22T00:02:04.888851+0000","last_undegraded":"2024-04-22T00:02:04.888851+0000","last_fullsized":"2024-04-22T00:02:04.888851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:17:24.257753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888718+0000","last_change":"2024-04-21T23:59:36.252276+0000","last_active":"2024-04-22T00:02:04.888718+0000","last_peered":"2024-04-22T00:02:04.888718+0000","last_clean":"2024-04-22T00:02:04.888718+0000","last_became_active":"2024-04-21T23:59:36.251915+0000","last_became_peered":"2024-04-21T23:59:36.251915+0000","last_unstale":"2024-04-22T00:02:04.888718+0000","last_undegraded":"2024-04-22T00:02:04.888718+0000","last_fullsized":"2024-04-22T00:02:04.888718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:58:49.336340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824749+0000","last_change":"2024-04-21T23:59:40.409229+0000","last_active":"2024-04-22T00:02:04.824749+0000","last_peered":"2024-04-22T00:02:04.824749+0000","last_clean":"2024-04-22T00:02:04.824749+0000","last_became_active":"2024-04-21T23:59:40.408786+0000","last_became_peered":"2024-04-21T23:59:40.408786+0000","last_unstale":"2024-04-22T00:02:04.824749+0000","last_undegraded":"2024-04-22T00:02:04.824749+0000","last_fullsized":"2024-04-22T00:02:04.824749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:46:27.699956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888091+0000","last_change":"2024-04-21T23:59:35.612459+0000","last_active":"2024-04-22T00:02:04.888091+0000","last_peered":"2024-04-22T00:02:04.888091+0000","last_clean":"2024-04-22T00:02:04.888091+0000","last_became_active":"2024-04-21T23:59:35.612034+0000","last_became_peered":"2024-04-21T23:59:35.612034+0000","last_unstale":"2024-04-22T00:02:04.888091+0000","last_undegraded":"2024-04-22T00:02:04.888091+0000","last_fullsized":"2024-04-22T00:02:04.888091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:54.311643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888039+0000","last_change":"2024-04-21T23:59:36.255947+0000","last_active":"2024-04-22T00:02:04.888039+0000","last_peered":"2024-04-22T00:02:04.888039+0000","last_clean":"2024-04-22T00:02:04.888039+0000","last_became_active":"2024-04-21T23:59:36.255653+0000","last_became_peered":"2024-04-21T23:59:36.255653+0000","last_unstale":"2024-04-22T00:02:04.888039+0000","last_undegraded":"2024-04-22T00:02:04.888039+0000","last_fullsized":"2024-04-22T00:02:04.888039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:31.367104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824846+0000","last_change":"2024-04-21T23:59:40.410044+0000","last_active":"2024-04-22T00:02:04.824846+0000","last_peered":"2024-04-22T00:02:04.824846+0000","last_clean":"2024-04-22T00:02:04.824846+0000","last_became_active":"2024-04-21T23:59:40.409580+0000","last_became_peered":"2024-04-21T23:59:40.409580+0000","last_unstale":"2024-04-22T00:02:04.824846+0000","last_undegraded":"2024-04-22T00:02:04.824846+0000","last_fullsized":"2024-04-22T00:02:04.824846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:15:23.837275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'43","reported_seq":325,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443892+0000","last_change":"2024-04-21T23:59:35.609463+0000","last_active":"2024-04-22T00:02:05.443892+0000","last_peered":"2024-04-22T00:02:05.443892+0000","last_clean":"2024-04-22T00:02:05.443892+0000","last_became_active":"2024-04-21T23:59:35.609132+0000","last_became_peered":"2024-04-21T23:59:35.609132+0000","last_unstale":"2024-04-22T00:02:05.443892+0000","last_undegraded":"2024-04-22T00:02:05.443892+0000","last_fullsized":"2024-04-22T00:02:05.443892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:07.609642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69045,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":45,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825061+0000","last_change":"2024-04-21T23:59:36.256992+0000","last_active":"2024-04-22T00:02:04.825061+0000","last_peered":"2024-04-22T00:02:04.825061+0000","last_clean":"2024-04-22T00:02:04.825061+0000","last_became_active":"2024-04-21T23:59:36.256602+0000","last_became_peered":"2024-04-21T23:59:36.256602+0000","last_unstale":"2024-04-22T00:02:04.825061+0000","last_undegraded":"2024-04-22T00:02:04.825061+0000","last_fullsized":"2024-04-22T00:02:04.825061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:02:35.877693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888344+0000","last_change":"2024-04-21T23:59:40.415516+0000","last_active":"2024-04-22T00:02:04.888344+0000","last_peered":"2024-04-22T00:02:04.888344+0000","last_clean":"2024-04-22T00:02:04.888344+0000","last_became_active":"2024-04-21T23:59:40.415092+0000","last_became_peered":"2024-04-21T23:59:40.415092+0000","last_unstale":"2024-04-22T00:02:04.888344+0000","last_undegraded":"2024-04-22T00:02:04.888344+0000","last_fullsized":"2024-04-22T00:02:04.888344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:19.618776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888192+0000","last_change":"2024-04-21T23:59:35.620713+0000","last_active":"2024-04-22T00:02:04.888192+0000","last_peered":"2024-04-22T00:02:04.888192+0000","last_clean":"2024-04-22T00:02:04.888192+0000","last_became_active":"2024-04-21T23:59:35.620266+0000","last_became_peered":"2024-04-21T23:59:35.620266+0000","last_unstale":"2024-04-22T00:02:04.888192+0000","last_undegraded":"2024-04-22T00:02:04.888192+0000","last_fullsized":"2024-04-22T00:02:04.888192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:55:37.656298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887241+0000","last_change":"2024-04-21T23:59:36.258627+0000","last_active":"2024-04-22T00:02:04.887241+0000","last_peered":"2024-04-22T00:02:04.887241+0000","last_clean":"2024-04-22T00:02:04.887241+0000","last_became_active":"2024-04-21T23:59:36.258025+0000","last_became_peered":"2024-04-21T23:59:36.258025+0000","last_unstale":"2024-04-22T00:02:04.887241+0000","last_undegraded":"2024-04-22T00:02:04.887241+0000","last_fullsized":"2024-04-22T00:02:04.887241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:26:22.233947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887354+0000","last_change":"2024-04-21T23:59:40.412014+0000","last_active":"2024-04-22T00:02:04.887354+0000","last_peered":"2024-04-22T00:02:04.887354+0000","last_clean":"2024-04-22T00:02:04.887354+0000","last_became_active":"2024-04-21T23:59:40.411556+0000","last_became_peered":"2024-04-21T23:59:40.411556+0000","last_unstale":"2024-04-22T00:02:04.887354+0000","last_undegraded":"2024-04-22T00:02:04.887354+0000","last_fullsized":"2024-04-22T00:02:04.887354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:50:35.026284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887431+0000","last_change":"2024-04-21T23:59:35.622198+0000","last_active":"2024-04-22T00:02:04.887431+0000","last_peered":"2024-04-22T00:02:04.887431+0000","last_clean":"2024-04-22T00:02:04.887431+0000","last_became_active":"2024-04-21T23:59:35.621887+0000","last_became_peered":"2024-04-21T23:59:35.621887+0000","last_unstale":"2024-04-22T00:02:04.887431+0000","last_undegraded":"2024-04-22T00:02:04.887431+0000","last_fullsized":"2024-04-22T00:02:04.887431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.766645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888547+0000","last_change":"2024-04-22T00:00:27.360979+0000","last_active":"2024-04-22T00:02:04.888547+0000","last_peered":"2024-04-22T00:02:04.888547+0000","last_clean":"2024-04-22T00:02:04.888547+0000","last_became_active":"2024-04-22T00:00:27.360589+0000","last_became_peered":"2024-04-22T00:00:27.360589+0000","last_unstale":"2024-04-22T00:02:04.888547+0000","last_undegraded":"2024-04-22T00:02:04.888547+0000","last_fullsized":"2024-04-22T00:02:04.888547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:10:25.020527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825130+0000","last_change":"2024-04-21T23:59:40.417397+0000","last_active":"2024-04-22T00:02:04.825130+0000","last_peered":"2024-04-22T00:02:04.825130+0000","last_clean":"2024-04-22T00:02:04.825130+0000","last_became_active":"2024-04-21T23:59:40.416974+0000","last_became_peered":"2024-04-21T23:59:40.416974+0000","last_unstale":"2024-04-22T00:02:04.825130+0000","last_undegraded":"2024-04-22T00:02:04.825130+0000","last_fullsized":"2024-04-22T00:02:04.825130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:16:06.280337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443632+0000","last_change":"2024-04-21T23:59:35.613313+0000","last_active":"2024-04-22T00:02:05.443632+0000","last_peered":"2024-04-22T00:02:05.443632+0000","last_clean":"2024-04-22T00:02:05.443632+0000","last_became_active":"2024-04-21T23:59:35.612999+0000","last_became_peered":"2024-04-21T23:59:35.612999+0000","last_unstale":"2024-04-22T00:02:05.443632+0000","last_undegraded":"2024-04-22T00:02:05.443632+0000","last_fullsized":"2024-04-22T00:02:05.443632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:31:06.910057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887550+0000","last_change":"2024-04-21T23:59:36.265632+0000","last_active":"2024-04-22T00:02:04.887550+0000","last_peered":"2024-04-22T00:02:04.887550+0000","last_clean":"2024-04-22T00:02:04.887550+0000","last_became_active":"2024-04-21T23:59:36.265295+0000","last_became_peered":"2024-04-21T23:59:36.265295+0000","last_unstale":"2024-04-22T00:02:04.887550+0000","last_undegraded":"2024-04-22T00:02:04.887550+0000","last_fullsized":"2024-04-22T00:02:04.887550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:33:40.809533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888166+0000","last_change":"2024-04-21T23:59:40.422076+0000","last_active":"2024-04-22T00:02:04.888166+0000","last_peered":"2024-04-22T00:02:04.888166+0000","last_clean":"2024-04-22T00:02:04.888166+0000","last_became_active":"2024-04-21T23:59:40.420914+0000","last_became_peered":"2024-04-21T23:59:40.420914+0000","last_unstale":"2024-04-22T00:02:04.888166+0000","last_undegraded":"2024-04-22T00:02:04.888166+0000","last_fullsized":"2024-04-22T00:02:04.888166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:59:58.475586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824529+0000","last_change":"2024-04-21T23:59:35.611962+0000","last_active":"2024-04-22T00:02:04.824529+0000","last_peered":"2024-04-22T00:02:04.824529+0000","last_clean":"2024-04-22T00:02:04.824529+0000","last_became_active":"2024-04-21T23:59:35.611659+0000","last_became_peered":"2024-04-21T23:59:35.611659+0000","last_unstale":"2024-04-22T00:02:04.824529+0000","last_undegraded":"2024-04-22T00:02:04.824529+0000","last_fullsized":"2024-04-22T00:02:04.824529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:44:15.802226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887651+0000","last_change":"2024-04-21T23:59:36.249244+0000","last_active":"2024-04-22T00:02:04.887651+0000","last_peered":"2024-04-22T00:02:04.887651+0000","last_clean":"2024-04-22T00:02:04.887651+0000","last_became_active":"2024-04-21T23:59:36.248933+0000","last_became_peered":"2024-04-21T23:59:36.248933+0000","last_unstale":"2024-04-22T00:02:04.887651+0000","last_undegraded":"2024-04-22T00:02:04.887651+0000","last_fullsized":"2024-04-22T00:02:04.887651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:23:51.971889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887494+0000","last_change":"2024-04-21T23:59:40.415679+0000","last_active":"2024-04-22T00:02:04.887494+0000","last_peered":"2024-04-22T00:02:04.887494+0000","last_clean":"2024-04-22T00:02:04.887494+0000","last_became_active":"2024-04-21T23:59:40.415182+0000","last_became_peered":"2024-04-21T23:59:40.415182+0000","last_unstale":"2024-04-22T00:02:04.887494+0000","last_undegraded":"2024-04-22T00:02:04.887494+0000","last_fullsized":"2024-04-22T00:02:04.887494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:51.218426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824935+0000","last_change":"2024-04-21T23:59:35.610290+0000","last_active":"2024-04-22T00:02:04.824935+0000","last_peered":"2024-04-22T00:02:04.824935+0000","last_clean":"2024-04-22T00:02:04.824935+0000","last_became_active":"2024-04-21T23:59:35.609937+0000","last_became_peered":"2024-04-21T23:59:35.609937+0000","last_unstale":"2024-04-22T00:02:04.824935+0000","last_undegraded":"2024-04-22T00:02:04.824935+0000","last_fullsized":"2024-04-22T00:02:04.824935+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:18:46.168304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887393+0000","last_change":"2024-04-21T23:59:36.250303+0000","last_active":"2024-04-22T00:02:04.887393+0000","last_peered":"2024-04-22T00:02:04.887393+0000","last_clean":"2024-04-22T00:02:04.887393+0000","last_became_active":"2024-04-21T23:59:36.249989+0000","last_became_peered":"2024-04-21T23:59:36.249989+0000","last_unstale":"2024-04-22T00:02:04.887393+0000","last_undegraded":"2024-04-22T00:02:04.887393+0000","last_fullsized":"2024-04-22T00:02:04.887393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:50.994549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824582+0000","last_change":"2024-04-21T23:59:40.499099+0000","last_active":"2024-04-22T00:02:04.824582+0000","last_peered":"2024-04-22T00:02:04.824582+0000","last_clean":"2024-04-22T00:02:04.824582+0000","last_became_active":"2024-04-21T23:59:40.498562+0000","last_became_peered":"2024-04-21T23:59:40.498562+0000","last_unstale":"2024-04-22T00:02:04.824582+0000","last_undegraded":"2024-04-22T00:02:04.824582+0000","last_fullsized":"2024-04-22T00:02:04.824582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:47.318140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825063+0000","last_change":"2024-04-22T00:00:27.360956+0000","last_active":"2024-04-22T00:02:04.825063+0000","last_peered":"2024-04-22T00:02:04.825063+0000","last_clean":"2024-04-22T00:02:04.825063+0000","last_became_active":"2024-04-22T00:00:27.360481+0000","last_became_peered":"2024-04-22T00:00:27.360481+0000","last_unstale":"2024-04-22T00:02:04.825063+0000","last_undegraded":"2024-04-22T00:02:04.825063+0000","last_fullsized":"2024-04-22T00:02:04.825063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:16.624366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888234+0000","last_change":"2024-04-21T23:59:36.255619+0000","last_active":"2024-04-22T00:02:04.888234+0000","last_peered":"2024-04-22T00:02:04.888234+0000","last_clean":"2024-04-22T00:02:04.888234+0000","last_became_active":"2024-04-21T23:59:36.255295+0000","last_became_peered":"2024-04-21T23:59:36.255295+0000","last_unstale":"2024-04-22T00:02:04.888234+0000","last_undegraded":"2024-04-22T00:02:04.888234+0000","last_fullsized":"2024-04-22T00:02:04.888234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:01:23.513079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887728+0000","last_change":"2024-04-21T23:59:40.421876+0000","last_active":"2024-04-22T00:02:04.887728+0000","last_peered":"2024-04-22T00:02:04.887728+0000","last_clean":"2024-04-22T00:02:04.887728+0000","last_became_active":"2024-04-21T23:59:40.421451+0000","last_became_peered":"2024-04-21T23:59:40.421451+0000","last_unstale":"2024-04-22T00:02:04.887728+0000","last_undegraded":"2024-04-22T00:02:04.887728+0000","last_fullsized":"2024-04-22T00:02:04.887728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:47.331970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":160,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.443457+0000","last_change":"2024-04-21T23:59:35.616857+0000","last_active":"2024-04-22T00:02:05.443457+0000","last_peered":"2024-04-22T00:02:05.443457+0000","last_clean":"2024-04-22T00:02:05.443457+0000","last_became_active":"2024-04-21T23:59:35.615789+0000","last_became_peered":"2024-04-21T23:59:35.615789+0000","last_unstale":"2024-04-22T00:02:05.443457+0000","last_undegraded":"2024-04-22T00:02:05.443457+0000","last_fullsized":"2024-04-22T00:02:05.443457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:09:18.478836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887744+0000","last_change":"2024-04-21T23:59:36.260613+0000","last_active":"2024-04-22T00:02:04.887744+0000","last_peered":"2024-04-22T00:02:04.887744+0000","last_clean":"2024-04-22T00:02:04.887744+0000","last_became_active":"2024-04-21T23:59:36.260254+0000","last_became_peered":"2024-04-21T23:59:36.260254+0000","last_unstale":"2024-04-22T00:02:04.887744+0000","last_undegraded":"2024-04-22T00:02:04.887744+0000","last_fullsized":"2024-04-22T00:02:04.887744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:23.787004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888339+0000","last_change":"2024-04-21T23:59:40.499455+0000","last_active":"2024-04-22T00:02:04.888339+0000","last_peered":"2024-04-22T00:02:04.888339+0000","last_clean":"2024-04-22T00:02:04.888339+0000","last_became_active":"2024-04-21T23:59:40.498770+0000","last_became_peered":"2024-04-21T23:59:40.498770+0000","last_unstale":"2024-04-22T00:02:04.888339+0000","last_undegraded":"2024-04-22T00:02:04.888339+0000","last_fullsized":"2024-04-22T00:02:04.888339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:58:26.745186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886981+0000","last_change":"2024-04-21T23:59:36.254411+0000","last_active":"2024-04-22T00:02:04.886981+0000","last_peered":"2024-04-22T00:02:04.886981+0000","last_clean":"2024-04-22T00:02:04.886981+0000","last_became_active":"2024-04-21T23:59:36.253290+0000","last_became_peered":"2024-04-21T23:59:36.253290+0000","last_unstale":"2024-04-22T00:02:04.886981+0000","last_undegraded":"2024-04-22T00:02:04.886981+0000","last_fullsized":"2024-04-22T00:02:04.886981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:03:17.125833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'7","reported_seq":159,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.065164+0000","last_change":"2024-04-21T23:59:35.622117+0000","last_active":"2024-04-22T00:02:05.065164+0000","last_peered":"2024-04-22T00:02:05.065164+0000","last_clean":"2024-04-22T00:02:05.065164+0000","last_became_active":"2024-04-21T23:59:35.621721+0000","last_became_peered":"2024-04-21T23:59:35.621721+0000","last_unstale":"2024-04-22T00:02:05.065164+0000","last_undegraded":"2024-04-22T00:02:05.065164+0000","last_fullsized":"2024-04-22T00:02:05.065164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:01:02.648151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888685+0000","last_change":"2024-04-22T00:00:27.369850+0000","last_active":"2024-04-22T00:02:04.888685+0000","last_peered":"2024-04-22T00:02:04.888685+0000","last_clean":"2024-04-22T00:02:04.888685+0000","last_became_active":"2024-04-22T00:00:27.361664+0000","last_became_peered":"2024-04-22T00:00:27.361664+0000","last_unstale":"2024-04-22T00:02:04.888685+0000","last_undegraded":"2024-04-22T00:02:04.888685+0000","last_fullsized":"2024-04-22T00:02:04.888685+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:42:13.794809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888320+0000","last_change":"2024-04-21T23:59:36.259666+0000","last_active":"2024-04-22T00:02:04.888320+0000","last_peered":"2024-04-22T00:02:04.888320+0000","last_clean":"2024-04-22T00:02:04.888320+0000","last_became_active":"2024-04-21T23:59:36.259343+0000","last_became_peered":"2024-04-21T23:59:36.259343+0000","last_unstale":"2024-04-22T00:02:04.888320+0000","last_undegraded":"2024-04-22T00:02:04.888320+0000","last_fullsized":"2024-04-22T00:02:04.888320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:29:11.946268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824591+0000","last_change":"2024-04-21T23:59:35.603689+0000","last_active":"2024-04-22T00:02:04.824591+0000","last_peered":"2024-04-22T00:02:04.824591+0000","last_clean":"2024-04-22T00:02:04.824591+0000","last_became_active":"2024-04-21T23:59:35.603357+0000","last_became_peered":"2024-04-21T23:59:35.603357+0000","last_unstale":"2024-04-22T00:02:04.824591+0000","last_undegraded":"2024-04-22T00:02:04.824591+0000","last_fullsized":"2024-04-22T00:02:04.824591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:24:20.073341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887243+0000","last_change":"2024-04-21T23:58:24.306054+0000","last_active":"2024-04-22T00:02:04.887243+0000","last_peered":"2024-04-22T00:02:04.887243+0000","last_clean":"2024-04-22T00:02:04.887243+0000","last_became_active":"2024-04-21T23:58:24.305490+0000","last_became_peered":"2024-04-21T23:58:24.305490+0000","last_unstale":"2024-04-22T00:02:04.887243+0000","last_undegraded":"2024-04-22T00:02:04.887243+0000","last_fullsized":"2024-04-22T00:02:04.887243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:54:54.664156+0000","last_clean_scrub_stamp":"2024-04-21T23:54:54.664156+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:51:22.728646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888319+0000","last_change":"2024-04-21T23:59:40.499339+0000","last_active":"2024-04-22T00:02:04.888319+0000","last_peered":"2024-04-22T00:02:04.888319+0000","last_clean":"2024-04-22T00:02:04.888319+0000","last_became_active":"2024-04-21T23:59:40.498898+0000","last_became_peered":"2024-04-21T23:59:40.498898+0000","last_unstale":"2024-04-22T00:02:04.888319+0000","last_undegraded":"2024-04-22T00:02:04.888319+0000","last_fullsized":"2024-04-22T00:02:04.888319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:44:23.220374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887578+0000","last_change":"2024-04-22T00:00:27.359209+0000","last_active":"2024-04-22T00:02:04.887578+0000","last_peered":"2024-04-22T00:02:04.887578+0000","last_clean":"2024-04-22T00:02:04.887578+0000","last_became_active":"2024-04-22T00:00:27.358750+0000","last_became_peered":"2024-04-22T00:00:27.358750+0000","last_unstale":"2024-04-22T00:02:04.887578+0000","last_undegraded":"2024-04-22T00:02:04.887578+0000","last_fullsized":"2024-04-22T00:02:04.887578+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:42:05.053039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887357+0000","last_change":"2024-04-21T23:59:35.615662+0000","last_active":"2024-04-22T00:02:04.887357+0000","last_peered":"2024-04-22T00:02:04.887357+0000","last_clean":"2024-04-22T00:02:04.887357+0000","last_became_active":"2024-04-21T23:59:35.615297+0000","last_became_peered":"2024-04-21T23:59:35.615297+0000","last_unstale":"2024-04-22T00:02:04.887357+0000","last_undegraded":"2024-04-22T00:02:04.887357+0000","last_fullsized":"2024-04-22T00:02:04.887357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:40:43.332606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887148+0000","last_change":"2024-04-21T23:59:40.412327+0000","last_active":"2024-04-22T00:02:04.887148+0000","last_peered":"2024-04-22T00:02:04.887148+0000","last_clean":"2024-04-22T00:02:04.887148+0000","last_became_active":"2024-04-21T23:59:40.411890+0000","last_became_peered":"2024-04-21T23:59:40.411890+0000","last_unstale":"2024-04-22T00:02:04.887148+0000","last_undegraded":"2024-04-22T00:02:04.887148+0000","last_fullsized":"2024-04-22T00:02:04.887148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:50.643151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887086+0000","last_change":"2024-04-22T00:00:27.368767+0000","last_active":"2024-04-22T00:02:04.887086+0000","last_peered":"2024-04-22T00:02:04.887086+0000","last_clean":"2024-04-22T00:02:04.887086+0000","last_became_active":"2024-04-22T00:00:27.362160+0000","last_became_peered":"2024-04-22T00:00:27.362160+0000","last_unstale":"2024-04-22T00:02:04.887086+0000","last_undegraded":"2024-04-22T00:02:04.887086+0000","last_fullsized":"2024-04-22T00:02:04.887086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:38.606596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887917+0000","last_change":"2024-04-21T23:59:36.254716+0000","last_active":"2024-04-22T00:02:04.887917+0000","last_peered":"2024-04-22T00:02:04.887917+0000","last_clean":"2024-04-22T00:02:04.887917+0000","last_became_active":"2024-04-21T23:59:36.254337+0000","last_became_peered":"2024-04-21T23:59:36.254337+0000","last_unstale":"2024-04-22T00:02:04.887917+0000","last_undegraded":"2024-04-22T00:02:04.887917+0000","last_fullsized":"2024-04-22T00:02:04.887917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:12:05.541456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887355+0000","last_change":"2024-04-21T23:59:40.499773+0000","last_active":"2024-04-22T00:02:04.887355+0000","last_peered":"2024-04-22T00:02:04.887355+0000","last_clean":"2024-04-22T00:02:04.887355+0000","last_became_active":"2024-04-21T23:59:40.498823+0000","last_became_peered":"2024-04-21T23:59:40.498823+0000","last_unstale":"2024-04-22T00:02:04.887355+0000","last_undegraded":"2024-04-22T00:02:04.887355+0000","last_fullsized":"2024-04-22T00:02:04.887355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:04:19.760690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.029471+0000","last_change":"2024-04-21T23:59:35.615731+0000","last_active":"2024-04-22T00:02:05.029471+0000","last_peered":"2024-04-22T00:02:05.029471+0000","last_clean":"2024-04-22T00:02:05.029471+0000","last_became_active":"2024-04-21T23:59:35.615422+0000","last_became_peered":"2024-04-21T23:59:35.615422+0000","last_unstale":"2024-04-22T00:02:05.029471+0000","last_undegraded":"2024-04-22T00:02:05.029471+0000","last_fullsized":"2024-04-22T00:02:05.029471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:26.423319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.256728+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.256393+0000","last_became_peered":"2024-04-21T23:59:36.256393+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:38:53.296515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886568+0000","last_change":"2024-04-21T23:59:40.407779+0000","last_active":"2024-04-22T00:02:04.886568+0000","last_peered":"2024-04-22T00:02:04.886568+0000","last_clean":"2024-04-22T00:02:04.886568+0000","last_became_active":"2024-04-21T23:59:40.407333+0000","last_became_peered":"2024-04-21T23:59:40.407333+0000","last_unstale":"2024-04-22T00:02:04.886568+0000","last_undegraded":"2024-04-22T00:02:04.886568+0000","last_fullsized":"2024-04-22T00:02:04.886568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:52:22.415566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.825128+0000","last_change":"2024-04-21T23:59:35.611804+0000","last_active":"2024-04-22T00:02:04.825128+0000","last_peered":"2024-04-22T00:02:04.825128+0000","last_clean":"2024-04-22T00:02:04.825128+0000","last_became_active":"2024-04-21T23:59:35.611451+0000","last_became_peered":"2024-04-21T23:59:35.611451+0000","last_unstale":"2024-04-22T00:02:04.825128+0000","last_undegraded":"2024-04-22T00:02:04.825128+0000","last_fullsized":"2024-04-22T00:02:04.825128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:09.529643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888601+0000","last_change":"2024-04-21T23:59:36.249983+0000","last_active":"2024-04-22T00:02:04.888601+0000","last_peered":"2024-04-22T00:02:04.888601+0000","last_clean":"2024-04-22T00:02:04.888601+0000","last_became_active":"2024-04-21T23:59:36.249627+0000","last_became_peered":"2024-04-21T23:59:36.249627+0000","last_unstale":"2024-04-22T00:02:04.888601+0000","last_undegraded":"2024-04-22T00:02:04.888601+0000","last_fullsized":"2024-04-22T00:02:04.888601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:52.133802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824924+0000","last_change":"2024-04-21T23:59:40.499619+0000","last_active":"2024-04-22T00:02:04.824924+0000","last_peered":"2024-04-22T00:02:04.824924+0000","last_clean":"2024-04-22T00:02:04.824924+0000","last_became_active":"2024-04-21T23:59:40.499144+0000","last_became_peered":"2024-04-21T23:59:40.499144+0000","last_unstale":"2024-04-22T00:02:04.824924+0000","last_undegraded":"2024-04-22T00:02:04.824924+0000","last_fullsized":"2024-04-22T00:02:04.824924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:30.075315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887300+0000","last_change":"2024-04-21T23:59:35.613475+0000","last_active":"2024-04-22T00:02:04.887300+0000","last_peered":"2024-04-22T00:02:04.887300+0000","last_clean":"2024-04-22T00:02:04.887300+0000","last_became_active":"2024-04-21T23:59:35.613178+0000","last_became_peered":"2024-04-21T23:59:35.613178+0000","last_unstale":"2024-04-22T00:02:04.887300+0000","last_undegraded":"2024-04-22T00:02:04.887300+0000","last_fullsized":"2024-04-22T00:02:04.887300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:33:09.094987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824460+0000","last_change":"2024-04-21T23:59:36.251832+0000","last_active":"2024-04-22T00:02:04.824460+0000","last_peered":"2024-04-22T00:02:04.824460+0000","last_clean":"2024-04-22T00:02:04.824460+0000","last_became_active":"2024-04-21T23:59:36.251520+0000","last_became_peered":"2024-04-21T23:59:36.251520+0000","last_unstale":"2024-04-22T00:02:04.824460+0000","last_undegraded":"2024-04-22T00:02:04.824460+0000","last_fullsized":"2024-04-22T00:02:04.824460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:47:14.785483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824869+0000","last_change":"2024-04-22T00:00:27.492100+0000","last_active":"2024-04-22T00:02:04.824869+0000","last_peered":"2024-04-22T00:02:04.824869+0000","last_clean":"2024-04-22T00:02:04.824869+0000","last_became_active":"2024-04-22T00:00:27.491414+0000","last_became_peered":"2024-04-22T00:00:27.491414+0000","last_unstale":"2024-04-22T00:02:04.824869+0000","last_undegraded":"2024-04-22T00:02:04.824869+0000","last_fullsized":"2024-04-22T00:02:04.824869+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:16:32.280602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888067+0000","last_change":"2024-04-21T23:59:35.622617+0000","last_active":"2024-04-22T00:02:04.888067+0000","last_peered":"2024-04-22T00:02:04.888067+0000","last_clean":"2024-04-22T00:02:04.888067+0000","last_became_active":"2024-04-21T23:59:35.622203+0000","last_became_peered":"2024-04-21T23:59:35.622203+0000","last_unstale":"2024-04-22T00:02:04.888067+0000","last_undegraded":"2024-04-22T00:02:04.888067+0000","last_fullsized":"2024-04-22T00:02:04.888067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:35:49.062179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887057+0000","last_change":"2024-04-21T23:59:36.253614+0000","last_active":"2024-04-22T00:02:04.887057+0000","last_peered":"2024-04-22T00:02:04.887057+0000","last_clean":"2024-04-22T00:02:04.887057+0000","last_became_active":"2024-04-21T23:59:36.252935+0000","last_became_peered":"2024-04-21T23:59:36.252935+0000","last_unstale":"2024-04-22T00:02:04.887057+0000","last_undegraded":"2024-04-22T00:02:04.887057+0000","last_fullsized":"2024-04-22T00:02:04.887057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:54:26.384656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824647+0000","last_change":"2024-04-22T00:00:27.497233+0000","last_active":"2024-04-22T00:02:04.824647+0000","last_peered":"2024-04-22T00:02:04.824647+0000","last_clean":"2024-04-22T00:02:04.824647+0000","last_became_active":"2024-04-22T00:00:27.492847+0000","last_became_peered":"2024-04-22T00:00:27.492847+0000","last_unstale":"2024-04-22T00:02:04.824647+0000","last_undegraded":"2024-04-22T00:02:04.824647+0000","last_fullsized":"2024-04-22T00:02:04.824647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:25.742746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887319+0000","last_change":"2024-04-21T23:59:35.620126+0000","last_active":"2024-04-22T00:02:04.887319+0000","last_peered":"2024-04-22T00:02:04.887319+0000","last_clean":"2024-04-22T00:02:04.887319+0000","last_became_active":"2024-04-21T23:59:35.619771+0000","last_became_peered":"2024-04-21T23:59:35.619771+0000","last_unstale":"2024-04-22T00:02:04.887319+0000","last_undegraded":"2024-04-22T00:02:04.887319+0000","last_fullsized":"2024-04-22T00:02:04.887319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:01:25.930923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888732+0000","last_change":"2024-04-21T23:59:36.264820+0000","last_active":"2024-04-22T00:02:04.888732+0000","last_peered":"2024-04-22T00:02:04.888732+0000","last_clean":"2024-04-22T00:02:04.888732+0000","last_became_active":"2024-04-21T23:59:36.264474+0000","last_became_peered":"2024-04-21T23:59:36.264474+0000","last_unstale":"2024-04-22T00:02:04.888732+0000","last_undegraded":"2024-04-22T00:02:04.888732+0000","last_fullsized":"2024-04-22T00:02:04.888732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:07:10.786334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824867+0000","last_change":"2024-04-21T23:59:40.499906+0000","last_active":"2024-04-22T00:02:04.824867+0000","last_peered":"2024-04-22T00:02:04.824867+0000","last_clean":"2024-04-22T00:02:04.824867+0000","last_became_active":"2024-04-21T23:59:40.499249+0000","last_became_peered":"2024-04-21T23:59:40.499249+0000","last_unstale":"2024-04-22T00:02:04.824867+0000","last_undegraded":"2024-04-22T00:02:04.824867+0000","last_fullsized":"2024-04-22T00:02:04.824867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:05.293118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.824650+0000","last_change":"2024-04-21T23:59:35.617405+0000","last_active":"2024-04-22T00:02:04.824650+0000","last_peered":"2024-04-22T00:02:04.824650+0000","last_clean":"2024-04-22T00:02:04.824650+0000","last_became_active":"2024-04-21T23:59:35.617086+0000","last_became_peered":"2024-04-21T23:59:35.617086+0000","last_unstale":"2024-04-22T00:02:04.824650+0000","last_undegraded":"2024-04-22T00:02:04.824650+0000","last_fullsized":"2024-04-22T00:02:04.824650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:03.701220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888041+0000","last_change":"2024-04-21T23:59:36.257085+0000","last_active":"2024-04-22T00:02:04.888041+0000","last_peered":"2024-04-22T00:02:04.888041+0000","last_clean":"2024-04-22T00:02:04.888041+0000","last_became_active":"2024-04-21T23:59:36.256598+0000","last_became_peered":"2024-04-21T23:59:36.256598+0000","last_unstale":"2024-04-22T00:02:04.888041+0000","last_undegraded":"2024-04-22T00:02:04.888041+0000","last_fullsized":"2024-04-22T00:02:04.888041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:10:40.300887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888383+0000","last_change":"2024-04-21T23:59:40.411455+0000","last_active":"2024-04-22T00:02:04.888383+0000","last_peered":"2024-04-22T00:02:04.888383+0000","last_clean":"2024-04-22T00:02:04.888383+0000","last_became_active":"2024-04-21T23:59:40.411012+0000","last_became_peered":"2024-04-21T23:59:40.411012+0000","last_unstale":"2024-04-22T00:02:04.888383+0000","last_undegraded":"2024-04-22T00:02:04.888383+0000","last_fullsized":"2024-04-22T00:02:04.888383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:17:51.013684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'8","reported_seq":163,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888517+0000","last_change":"2024-04-21T23:59:35.615869+0000","last_active":"2024-04-22T00:02:04.888517+0000","last_peered":"2024-04-22T00:02:04.888517+0000","last_clean":"2024-04-22T00:02:04.888517+0000","last_became_active":"2024-04-21T23:59:35.615513+0000","last_became_peered":"2024-04-21T23:59:35.615513+0000","last_unstale":"2024-04-22T00:02:04.888517+0000","last_undegraded":"2024-04-22T00:02:04.888517+0000","last_fullsized":"2024-04-22T00:02:04.888517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:20.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2010,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888876+0000","last_change":"2024-04-21T23:59:36.259757+0000","last_active":"2024-04-22T00:02:04.888876+0000","last_peered":"2024-04-22T00:02:04.888876+0000","last_clean":"2024-04-22T00:02:04.888876+0000","last_became_active":"2024-04-21T23:59:36.256919+0000","last_became_peered":"2024-04-21T23:59:36.256919+0000","last_unstale":"2024-04-22T00:02:04.888876+0000","last_undegraded":"2024-04-22T00:02:04.888876+0000","last_fullsized":"2024-04-22T00:02:04.888876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:25:20.657589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886998+0000","last_change":"2024-04-21T23:59:40.418932+0000","last_active":"2024-04-22T00:02:04.886998+0000","last_peered":"2024-04-22T00:02:04.886998+0000","last_clean":"2024-04-22T00:02:04.886998+0000","last_became_active":"2024-04-21T23:59:40.418500+0000","last_became_peered":"2024-04-21T23:59:40.418500+0000","last_unstale":"2024-04-22T00:02:04.886998+0000","last_undegraded":"2024-04-22T00:02:04.886998+0000","last_fullsized":"2024-04-22T00:02:04.886998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:12:38.818928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:05.027186+0000","last_change":"2024-04-21T23:59:35.624429+0000","last_active":"2024-04-22T00:02:05.027186+0000","last_peered":"2024-04-22T00:02:05.027186+0000","last_clean":"2024-04-22T00:02:05.027186+0000","last_became_active":"2024-04-21T23:59:35.624172+0000","last_became_peered":"2024-04-21T23:59:35.624172+0000","last_unstale":"2024-04-22T00:02:05.027186+0000","last_undegraded":"2024-04-22T00:02:05.027186+0000","last_fullsized":"2024-04-22T00:02:05.027186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:16:54.895996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888652+0000","last_change":"2024-04-21T23:59:36.260918+0000","last_active":"2024-04-22T00:02:04.888652+0000","last_peered":"2024-04-22T00:02:04.888652+0000","last_clean":"2024-04-22T00:02:04.888652+0000","last_became_active":"2024-04-21T23:59:36.260577+0000","last_became_peered":"2024-04-21T23:59:36.260577+0000","last_unstale":"2024-04-22T00:02:04.888652+0000","last_undegraded":"2024-04-22T00:02:04.888652+0000","last_fullsized":"2024-04-22T00:02:04.888652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:48:44.232906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886524+0000","last_change":"2024-04-21T23:59:40.409543+0000","last_active":"2024-04-22T00:02:04.886524+0000","last_peered":"2024-04-22T00:02:04.886524+0000","last_clean":"2024-04-22T00:02:04.886524+0000","last_became_active":"2024-04-21T23:59:40.408914+0000","last_became_peered":"2024-04-21T23:59:40.408914+0000","last_unstale":"2024-04-22T00:02:04.886524+0000","last_undegraded":"2024-04-22T00:02:04.886524+0000","last_fullsized":"2024-04-22T00:02:04.886524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:39.371543+0000","last_clean_scrub_stamp":"2024-04-21T23:59:39.371543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:41.968607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887998+0000","last_change":"2024-04-21T23:59:35.622164+0000","last_active":"2024-04-22T00:02:04.887998+0000","last_peered":"2024-04-22T00:02:04.887998+0000","last_clean":"2024-04-22T00:02:04.887998+0000","last_became_active":"2024-04-21T23:59:35.621816+0000","last_became_peered":"2024-04-21T23:59:35.621816+0000","last_unstale":"2024-04-22T00:02:04.887998+0000","last_undegraded":"2024-04-22T00:02:04.887998+0000","last_fullsized":"2024-04-22T00:02:04.887998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:34.578794+0000","last_clean_scrub_stamp":"2024-04-21T23:59:34.578794+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:25.920670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889013+0000","last_change":"2024-04-21T23:59:36.255349+0000","last_active":"2024-04-22T00:02:04.889013+0000","last_peered":"2024-04-22T00:02:04.889013+0000","last_clean":"2024-04-22T00:02:04.889013+0000","last_became_active":"2024-04-21T23:59:36.255017+0000","last_became_peered":"2024-04-21T23:59:36.255017+0000","last_unstale":"2024-04-22T00:02:04.889013+0000","last_undegraded":"2024-04-22T00:02:04.889013+0000","last_fullsized":"2024-04-22T00:02:04.889013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:35:30.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887227+0000","last_change":"2024-04-21T23:59:36.254860+0000","last_active":"2024-04-22T00:02:04.887227+0000","last_peered":"2024-04-22T00:02:04.887227+0000","last_clean":"2024-04-22T00:02:04.887227+0000","last_became_active":"2024-04-21T23:59:36.253354+0000","last_became_peered":"2024-04-21T23:59:36.253354+0000","last_unstale":"2024-04-22T00:02:04.887227+0000","last_undegraded":"2024-04-22T00:02:04.887227+0000","last_fullsized":"2024-04-22T00:02:04.887227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:39:07.079941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886684+0000","last_change":"2024-04-21T23:59:36.263689+0000","last_active":"2024-04-22T00:02:04.886684+0000","last_peered":"2024-04-22T00:02:04.886684+0000","last_clean":"2024-04-22T00:02:04.886684+0000","last_became_active":"2024-04-21T23:59:36.263228+0000","last_became_peered":"2024-04-21T23:59:36.263228+0000","last_unstale":"2024-04-22T00:02:04.886684+0000","last_undegraded":"2024-04-22T00:02:04.886684+0000","last_fullsized":"2024-04-22T00:02:04.886684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:05:03.573172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886866+0000","last_change":"2024-04-21T23:59:36.260884+0000","last_active":"2024-04-22T00:02:04.886866+0000","last_peered":"2024-04-22T00:02:04.886866+0000","last_clean":"2024-04-22T00:02:04.886866+0000","last_became_active":"2024-04-21T23:59:36.260546+0000","last_became_peered":"2024-04-21T23:59:36.260546+0000","last_unstale":"2024-04-22T00:02:04.886866+0000","last_undegraded":"2024-04-22T00:02:04.886866+0000","last_fullsized":"2024-04-22T00:02:04.886866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:26:01.428972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888285+0000","last_change":"2024-04-22T00:00:27.362810+0000","last_active":"2024-04-22T00:02:04.888285+0000","last_peered":"2024-04-22T00:02:04.888285+0000","last_clean":"2024-04-22T00:02:04.888285+0000","last_became_active":"2024-04-22T00:00:27.362380+0000","last_became_peered":"2024-04-22T00:00:27.362380+0000","last_unstale":"2024-04-22T00:02:04.888285+0000","last_undegraded":"2024-04-22T00:02:04.888285+0000","last_fullsized":"2024-04-22T00:02:04.888285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:09:30.391961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.888096+0000","last_change":"2024-04-21T23:59:36.249508+0000","last_active":"2024-04-22T00:02:04.888096+0000","last_peered":"2024-04-22T00:02:04.888096+0000","last_clean":"2024-04-22T00:02:04.888096+0000","last_became_active":"2024-04-21T23:59:36.249183+0000","last_became_peered":"2024-04-21T23:59:36.249183+0000","last_unstale":"2024-04-22T00:02:04.888096+0000","last_undegraded":"2024-04-22T00:02:04.888096+0000","last_fullsized":"2024-04-22T00:02:04.888096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:38:20.506593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887929+0000","last_change":"2024-04-21T23:59:36.252273+0000","last_active":"2024-04-22T00:02:04.887929+0000","last_peered":"2024-04-22T00:02:04.887929+0000","last_clean":"2024-04-22T00:02:04.887929+0000","last_became_active":"2024-04-21T23:59:36.251969+0000","last_became_peered":"2024-04-21T23:59:36.251969+0000","last_unstale":"2024-04-22T00:02:04.887929+0000","last_undegraded":"2024-04-22T00:02:04.887929+0000","last_fullsized":"2024-04-22T00:02:04.887929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:41:28.774041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887246+0000","last_change":"2024-04-21T23:59:36.258095+0000","last_active":"2024-04-22T00:02:04.887246+0000","last_peered":"2024-04-22T00:02:04.887246+0000","last_clean":"2024-04-22T00:02:04.887246+0000","last_became_active":"2024-04-21T23:59:36.257736+0000","last_became_peered":"2024-04-21T23:59:36.257736+0000","last_unstale":"2024-04-22T00:02:04.887246+0000","last_undegraded":"2024-04-22T00:02:04.887246+0000","last_fullsized":"2024-04-22T00:02:04.887246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:40.566070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.889156+0000","last_change":"2024-04-21T23:59:36.255383+0000","last_active":"2024-04-22T00:02:04.889156+0000","last_peered":"2024-04-22T00:02:04.889156+0000","last_clean":"2024-04-22T00:02:04.889156+0000","last_became_active":"2024-04-21T23:59:36.255052+0000","last_became_peered":"2024-04-21T23:59:36.255052+0000","last_unstale":"2024-04-22T00:02:04.889156+0000","last_undegraded":"2024-04-22T00:02:04.889156+0000","last_fullsized":"2024-04-22T00:02:04.889156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:10.014795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.886857+0000","last_change":"2024-04-21T23:59:36.254507+0000","last_active":"2024-04-22T00:02:04.886857+0000","last_peered":"2024-04-22T00:02:04.886857+0000","last_clean":"2024-04-22T00:02:04.886857+0000","last_became_active":"2024-04-21T23:59:36.253956+0000","last_became_peered":"2024-04-21T23:59:36.253956+0000","last_unstale":"2024-04-22T00:02:04.886857+0000","last_undegraded":"2024-04-22T00:02:04.886857+0000","last_fullsized":"2024-04-22T00:02:04.886857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:37:39.444500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887969+0000","last_change":"2024-04-21T23:59:36.260446+0000","last_active":"2024-04-22T00:02:04.887969+0000","last_peered":"2024-04-22T00:02:04.887969+0000","last_clean":"2024-04-22T00:02:04.887969+0000","last_became_active":"2024-04-21T23:59:36.259626+0000","last_became_peered":"2024-04-21T23:59:36.259626+0000","last_unstale":"2024-04-22T00:02:04.887969+0000","last_undegraded":"2024-04-22T00:02:04.887969+0000","last_fullsized":"2024-04-22T00:02:04.887969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:50:12.623530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887229+0000","last_change":"2024-04-21T23:59:36.254807+0000","last_active":"2024-04-22T00:02:04.887229+0000","last_peered":"2024-04-22T00:02:04.887229+0000","last_clean":"2024-04-22T00:02:04.887229+0000","last_became_active":"2024-04-21T23:59:36.254523+0000","last_became_peered":"2024-04-21T23:59:36.254523+0000","last_unstale":"2024-04-22T00:02:04.887229+0000","last_undegraded":"2024-04-22T00:02:04.887229+0000","last_fullsized":"2024-04-22T00:02:04.887229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:35:45.242690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887469+0000","last_change":"2024-04-21T23:59:36.248732+0000","last_active":"2024-04-22T00:02:04.887469+0000","last_peered":"2024-04-22T00:02:04.887469+0000","last_clean":"2024-04-22T00:02:04.887469+0000","last_became_active":"2024-04-21T23:59:36.248410+0000","last_became_peered":"2024-04-21T23:59:36.248410+0000","last_unstale":"2024-04-22T00:02:04.887469+0000","last_undegraded":"2024-04-22T00:02:04.887469+0000","last_fullsized":"2024-04-22T00:02:04.887469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:13:09.606136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T00:02:04.887415+0000","last_change":"2024-04-21T23:59:36.251821+0000","last_active":"2024-04-22T00:02:04.887415+0000","last_peered":"2024-04-22T00:02:04.887415+0000","last_clean":"2024-04-22T00:02:04.887415+0000","last_became_active":"2024-04-21T23:59:36.251496+0000","last_became_peered":"2024-04-21T23:59:36.251496+0000","last_unstale":"2024-04-22T00:02:04.887415+0000","last_undegraded":"2024-04-22T00:02:04.887415+0000","last_fullsized":"2024-04-22T00:02:04.887415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:59:35.589113+0000","last_clean_scrub_stamp":"2024-04-21T23:59:35.589113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:01.193598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602828,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1346,"num_read_kb":792,"num_write":377,"num_write_kb":1474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1808484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":512193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28860,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716292,"statfs":{"total":95995035648,"available":95965483008,"internally_reserved":0,"allocated":1888256,"data_stored":1303721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1466368,"data_stored":904387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1458176,"data_stored":908749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28648,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716504,"statfs":{"total":95995035648,"available":95965700096,"internally_reserved":0,"allocated":1667072,"data_stored":1108187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":577532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":508971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1138688,"data_stored":571655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1031199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":511672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1740800,"data_stored":1165197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":203304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T00:02:10.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:02:10.826 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64527]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:10.826 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64527]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:10.826 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:02:10 smithi111 sudo[64527]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:10.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 -- 172.21.15.38:0/3564918511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101f30 msgr2=0x7f14a410e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 --2- 172.21.15.38:0/3564918511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101f30 0x7f14a410e3f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f149c009ca0 tx=0x7f149c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 -- 172.21.15.38:0/3564918511 shutdown_connections 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 --2- 172.21.15.38:0/3564918511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101f30 0x7f14a410e3f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 --2- 172.21.15.38:0/3564918511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4101570 0x7f14a41019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 --2- 172.21.15.38:0/3564918511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4107630 0x7f14a4107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 -- 172.21.15.38:0/3564918511 >> 172.21.15.38:0/3564918511 conn(0x7f14a40fd0a0 msgr2=0x7f14a40ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 -- 172.21.15.38:0/3564918511 shutdown_connections 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.881+0000 7f14a9201640 1 -- 172.21.15.38:0/3564918511 wait complete. 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.882+0000 7f14a9201640 1 Processor -- start 2024-04-22T00:02:10.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.882+0000 7f14a9201640 1 -- start start 2024-04-22T00:02:10.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.882+0000 7f14a9201640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 0x7f14a419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4107630 0x7f14a4197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14a41118f0 con 0x7f14a4101570 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f14a4111770 con 0x7f14a4101f30 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14a4111a70 con 0x7f14a4107630 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40166/0 (socket says 172.21.15.38:40166) 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 -- 172.21.15.38:0/3847291181 learned_addr learned my addr 172.21.15.38:0/3847291181 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a359e640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4107630 0x7f14a4197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a259c640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 0x7f14a419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 -- 172.21.15.38:0/3847291181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4107630 msgr2=0x7f14a4197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4107630 0x7f14a4197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 -- 172.21.15.38:0/3847291181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 msgr2=0x7f14a419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 0x7f14a419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 -- 172.21.15.38:0/3847291181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a4198430 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a259c640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 0x7f14a419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a2d9d640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f149400ea60 tx=0x7f149400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149400cde0 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a4198720 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14940035f0 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494010670 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.883+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a41a49e0 con 0x7f14a4101570 2024-04-22T00:02:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.884+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14a4102c90 con 0x7f14a4101570 2024-04-22T00:02:10.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.886+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f14940037a0 con 0x7f14a4101570 2024-04-22T00:02:10.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.886+0000 7f148bfff640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 0x7f148c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:10.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.886+0000 7f14a259c640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 0x7f148c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:10.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.886+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f14940cfe90 con 0x7f14a4101570 2024-04-22T00:02:10.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.887+0000 7f14a259c640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 0x7f148c0a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14a4199300 tx=0x7f1498005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:10.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:10.890+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f149409c5b0 con 0x7f14a4101570 2024-04-22T00:02:10.906 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:10 smithi203 ceph-mon[31431]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:10.907 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:10 smithi203 ceph-mon[31431]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:10.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65569]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:10.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:10.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65569]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:10.907 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65569]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:11.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.076+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14a41099a0 con 0x7f14a4101570 2024-04-22T00:02:11.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.080+0000 7f148bfff640 1 -- 172.21.15.38:0/3847291181 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+25015 (secure 0 0 0) 0x7f149409beb0 con 0x7f14a4101570 2024-04-22T00:02:11.073 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:11.073 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":92,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:02:04.818207+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:02:11.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 msgr2=0x7f148c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 0x7f148c0a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f14a4199300 tx=0x7f1498005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 msgr2=0x7f14a419da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f149400ea60 tx=0x7f149400ef20 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 shutdown_connections 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f148c0a2e20 0x7f148c0a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14a4107630 0x7f14a4197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14a4101f30 0x7f14a419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 --2- 172.21.15.38:0/3847291181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14a4101570 0x7f14a419da60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 >> 172.21.15.38:0/3847291181 conn(0x7f14a40fd0a0 msgr2=0x7f14a40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 shutdown_connections 2024-04-22T00:02:11.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.082+0000 7f14a9201640 1 -- 172.21.15.38:0/3847291181 wait complete. 2024-04-22T00:02:11.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:10 smithi038 ceph-mon[27972]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:11.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:10 smithi038 ceph-mon[27972]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:11.086 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:02:11.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:02:11.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- 172.21.15.38:0/4120431433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 msgr2=0x7f1844100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 --2- 172.21.15.38:0/4120431433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f1844100a90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1830009930 tx=0x7f183002bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- 172.21.15.38:0/4120431433 shutdown_connections 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 --2- 172.21.15.38:0/4120431433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1844100fd0 0x7f184410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 --2- 172.21.15.38:0/4120431433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f1844100a90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 --2- 172.21.15.38:0/4120431433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18441066b0 0x7f1844106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- 172.21.15.38:0/4120431433 >> 172.21.15.38:0/4120431433 conn(0x7f1844077740 msgr2=0x7f1844077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- 172.21.15.38:0/4120431433 shutdown_connections 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- 172.21.15.38:0/4120431433 wait complete. 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 Processor -- start 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.201+0000 7f184b1f7640 1 -- start start 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1844100fd0 0x7f184410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18441066b0 0x7f184410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1844111b30 con 0x7f1844100610 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f18441119b0 con 0x7f1844100fd0 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1844111cb0 con 0x7f18441066b0 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40182/0 (socket says 172.21.15.38:40182) 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 -- 172.21.15.38:0/2583098392 learned_addr learned my addr 172.21.15.38:0/2583098392 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1849794640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18441066b0 0x7f184410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1843fff640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1844100fd0 0x7f184410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 -- 172.21.15.38:0/2583098392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18441066b0 msgr2=0x7f184410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18441066b0 0x7f184410e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 -- 172.21.15.38:0/2583098392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1844100fd0 msgr2=0x7f184410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1844100fd0 0x7f184410e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 -- 172.21.15.38:0/2583098392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18441a6490 con 0x7f1844100610 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1848f93640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f183800e9c0 tx=0x7f183800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183800cd80 con 0x7f1844100610 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18380035f0 con 0x7f1844100610 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18441a6780 con 0x7f1844100610 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.202+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838010690 con 0x7f1844100610 2024-04-22T00:02:11.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.203+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18441a6ca0 con 0x7f1844100610 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.204+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1844068560 con 0x7f1844100610 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.205+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f18380037a0 con 0x7f1844100610 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.206+0000 7f1841ffb640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 0x7f18340a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.206+0000 7f1843fff640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 0x7f18340a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.206+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f18380cfdc0 con 0x7f1844100610 2024-04-22T00:02:11.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.206+0000 7f1843fff640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 0x7f18340a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1830005ab0 tx=0x7f1830005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.210+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f183809c450 con 0x7f1844100610 2024-04-22T00:02:11.279 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65572]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:11.279 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65572]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:11.279 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65572]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:11.279 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:02:10 smithi203 sudo[65572]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:11.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:10 smithi111 ceph-mon[31543]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:11.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:10 smithi111 ceph-mon[31543]: from='client.15681 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:11.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.395+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18441a6f60 con 0x7f1844100610 2024-04-22T00:02:11.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.395+0000 7f1841ffb640 1 -- 172.21.15.38:0/2583098392 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f183809bd50 con 0x7f1844100610 2024-04-22T00:02:11.388 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 msgr2=0x7f18340a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 0x7f18340a52e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1830005ab0 tx=0x7f1830005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 msgr2=0x7f184410df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f183800e9c0 tx=0x7f183800ee80 comp rx=0 tx=0).stop 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 shutdown_connections 2024-04-22T00:02:11.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f18340a2e20 0x7f18340a52e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18441066b0 0x7f184410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1844100fd0 0x7f184410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 --2- 172.21.15.38:0/2583098392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1844100610 0x7f184410df70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.397+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 >> 172.21.15.38:0/2583098392 conn(0x7f1844077740 msgr2=0x7f18440ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.398+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 shutdown_connections 2024-04-22T00:02:11.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.398+0000 7f184b1f7640 1 -- 172.21.15.38:0/2583098392 wait complete. 2024-04-22T00:02:11.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:02:11.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- 172.21.15.38:0/252108268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c1066b0 msgr2=0x7f0b7c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 --2- 172.21.15.38:0/252108268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c106a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009860 tx=0x7f0b6c02bb70 comp rx=0 tx=0).stop 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- 172.21.15.38:0/252108268 shutdown_connections 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 --2- 172.21.15.38:0/252108268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 --2- 172.21.15.38:0/252108268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 --2- 172.21.15.38:0/252108268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c106a90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- 172.21.15.38:0/252108268 >> 172.21.15.38:0/252108268 conn(0x7f0b7c077740 msgr2=0x7f0b7c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- 172.21.15.38:0/252108268 shutdown_connections 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- 172.21.15.38:0/252108268 wait complete. 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 Processor -- start 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.516+0000 7f0b821bf640 1 -- start start 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c1109f0 con 0x7f0b7c100610 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c110870 con 0x7f0b7c100fd0 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b821bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c110b70 con 0x7f0b7c1066b0 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40196/0 (socket says 172.21.15.38:40196) 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 -- 172.21.15.38:0/3251544865 learned_addr learned my addr 172.21.15.38:0/3251544865 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7affd640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7bfff640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 -- 172.21.15.38:0/3251544865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1066b0 msgr2=0x7f0b7c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 -- 172.21.15.38:0/3251544865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 msgr2=0x7f0b7c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 -- 172.21.15.38:0/3251544865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7c19cfc0 con 0x7f0b7c100610 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7affd640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.517+0000 7f0b7b7fe640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.518+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c041020 con 0x7f0b7c100610 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.518+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7c19d250 con 0x7f0b7c100610 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.518+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b6c02c9f0 con 0x7f0b7c100610 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.518+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c039040 con 0x7f0b7c100610 2024-04-22T00:02:11.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.518+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7c1a4b40 con 0x7f0b7c100610 2024-04-22T00:02:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.519+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b7c068560 con 0x7f0b7c100610 2024-04-22T00:02:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.520+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f0b6c044020 con 0x7f0b7c100610 2024-04-22T00:02:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.521+0000 7f0b78ff9640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 0x7f0b640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.521+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0b6c0f1460 con 0x7f0b7c100610 2024-04-22T00:02:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.521+0000 7f0b7affd640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 0x7f0b640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.522+0000 7f0b7affd640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 0x7f0b640a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b700096c0 tx=0x7f0b70009260 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.525+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0b6c04b020 con 0x7f0b7c100610 2024-04-22T00:02:11.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.710+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0b7c1a4d70 con 0x7f0b7c100610 2024-04-22T00:02:11.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.711+0000 7f0b78ff9640 1 -- 172.21.15.38:0/3251544865 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f0b6c0bdd20 con 0x7f0b7c100610 2024-04-22T00:02:11.704 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:11.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 msgr2=0x7f0b640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 0x7f0b640a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b700096c0 tx=0x7f0b70009260 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 msgr2=0x7f0b7c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c000c00 tx=0x7f0b6c004530 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 shutdown_connections 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0b640a2e20 0x7f0b640a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1066b0 0x7f0b7c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b7c100fd0 0x7f0b7c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 --2- 172.21.15.38:0/3251544865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b7c100610 0x7f0b7c198020 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 >> 172.21.15.38:0/3251544865 conn(0x7f0b7c077740 msgr2=0x7f0b7c0ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 shutdown_connections 2024-04-22T00:02:11.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.713+0000 7f0b821bf640 1 -- 172.21.15.38:0/3251544865 wait complete. 2024-04-22T00:02:11.707 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[84953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:11.707 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[84953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:11.707 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[84953]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:11.707 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[84953]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:11.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:02:11.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 -- 172.21.15.38:0/1338457440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d01066b0 msgr2=0x7f69d0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/1338457440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d01066b0 0x7f69d0106a90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009930 tx=0x7f69cc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:11.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 -- 172.21.15.38:0/1338457440 shutdown_connections 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/1338457440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d0100fd0 0x7f69d010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/1338457440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 0x7f69d0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/1338457440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d01066b0 0x7f69d0106a90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 -- 172.21.15.38:0/1338457440 >> 172.21.15.38:0/1338457440 conn(0x7f69d0077740 msgr2=0x7f69d0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 -- 172.21.15.38:0/1338457440 shutdown_connections 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.830+0000 7f69d7fe5640 1 -- 172.21.15.38:0/1338457440 wait complete. 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 Processor -- start 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 -- start start 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 0x7f69d0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d01066b0 0x7f69d019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f69d0110a80 con 0x7f69d0100fd0 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f69d0110900 con 0x7f69d01066b0 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d7fe5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69d0110c00 con 0x7f69d0100610 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d5580640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d5580640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40222/0 (socket says 172.21.15.38:40222) 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d5580640 1 -- 172.21.15.38:0/2089917296 learned_addr learned my addr 172.21.15.38:0/2089917296 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.831+0000 7f69d6582640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d01066b0 0x7f69d019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5d81640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 0x7f69d0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 -- 172.21.15.38:0/2089917296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 msgr2=0x7f69d0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 0x7f69d0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 -- 172.21.15.38:0/2089917296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d01066b0 msgr2=0x7f69d019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d01066b0 0x7f69d019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:11.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 -- 172.21.15.38:0/2089917296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d019d030 con 0x7f69d0100fd0 2024-04-22T00:02:11.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d5580640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b4a0 tx=0x7f69c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0014070 con 0x7f69d0100fd0 2024-04-22T00:02:11.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d019d320 con 0x7f69d0100fd0 2024-04-22T00:02:11.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c0010af0 con 0x7f69d0100fd0 2024-04-22T00:02:11.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0017d10 con 0x7f69d0100fd0 2024-04-22T00:02:11.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.832+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d01a4bb0 con 0x7f69d0100fd0 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.833+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d0068560 con 0x7f69d0100fd0 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.834+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f69c0017070 con 0x7f69d0100fd0 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.835+0000 7f69c6ffd640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 0x7f69bc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.835+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f69c00ced30 con 0x7f69d0100fd0 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.835+0000 7f69d5d81640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 0x7f69bc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:11.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.835+0000 7f69d5d81640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 0x7f69bc0a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f69cc02c170 tx=0x7f69cc02df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:11.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:11.839+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f69c009b4d0 con 0x7f69d0100fd0 2024-04-22T00:02:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.027+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f69d01a4ea0 con 0x7f69d0100fd0 2024-04-22T00:02:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.028+0000 7f69c6ffd640 1 -- 172.21.15.38:0/2089917296 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f69c009add0 con 0x7f69d0100fd0 2024-04-22T00:02:12.021 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 msgr2=0x7f69bc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 0x7f69bc0a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f69cc02c170 tx=0x7f69cc02df90 comp rx=0 tx=0).stop 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 msgr2=0x7f69d0198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b4a0 tx=0x7f69c000b960 comp rx=0 tx=0).stop 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 shutdown_connections 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f69bc0a2e20 0x7f69bc0a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69d01066b0 0x7f69d019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69d0100fd0 0x7f69d0198580 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 --2- 172.21.15.38:0/2089917296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d0100610 0x7f69d0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 >> 172.21.15.38:0/2089917296 conn(0x7f69d0077740 msgr2=0x7f69d00ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 shutdown_connections 2024-04-22T00:02:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.030+0000 7f69d7fe5640 1 -- 172.21.15.38:0/2089917296 wait complete. 2024-04-22T00:02:12.033 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[85019]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:12.033 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[85019]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:12.033 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[85019]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:12.033 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:02:11 smithi038 sudo[85019]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:12.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:11 smithi038 ceph-mon[27972]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:12.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:11 smithi038 ceph-mon[27972]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-22T00:02:12.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3847291181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:12.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2583098392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:11 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3251544865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:02:12.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.147+0000 7fdb804ed640 1 -- 172.21.15.38:0/3474137819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 msgr2=0x7fdb7806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.147+0000 7fdb804ed640 1 --2- 172.21.15.38:0/3474137819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb7806a240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009860 tx=0x7fdb6c02bb70 comp rx=0 tx=0).stop 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- 172.21.15.38:0/3474137819 shutdown_connections 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- 172.21.15.38:0/3474137819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb7806a780 0x7fdb7810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- 172.21.15.38:0/3474137819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb7806a240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- 172.21.15.38:0/3474137819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78107730 0x7fdb78107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- 172.21.15.38:0/3474137819 >> 172.21.15.38:0/3474137819 conn(0x7fdb780776a0 msgr2=0x7fdb78077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- 172.21.15.38:0/3474137819 shutdown_connections 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- 172.21.15.38:0/3474137819 wait complete. 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 Processor -- start 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- start start 2024-04-22T00:02:12.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb78197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 0x7fdb781984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78111b40 con 0x7fdb78107730 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdb781119c0 con 0x7fdb78069dc0 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.148+0000 7fdb804ed640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78111cc0 con 0x7fdb7806a780 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40246/0 (socket says 172.21.15.38:40246) 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7e289640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb78197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 -- 172.21.15.38:0/1048826978 learned_addr learned my addr 172.21.15.38:0/1048826978 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7da88640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 0x7fdb781984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 -- 172.21.15.38:0/1048826978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 msgr2=0x7fdb781984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 0x7fdb781984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 -- 172.21.15.38:0/1048826978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 msgr2=0x7fdb78197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb78197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 -- 172.21.15.38:0/1048826978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7819cf70 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7da88640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 0x7fdb781984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7e289640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb78197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb7ea8a640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000efc0 tx=0x7fdb7000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70003970 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb70004020 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7819d260 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000b9f0 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.149+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb781a4af0 con 0x7fdb78107730 2024-04-22T00:02:12.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.150+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb780689b0 con 0x7fdb78107730 2024-04-22T00:02:12.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.151+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fdb7001d020 con 0x7fdb78107730 2024-04-22T00:02:12.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.152+0000 7fdb6b7fe640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 0x7fdb640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.152+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fdb700cf960 con 0x7fdb78107730 2024-04-22T00:02:12.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.152+0000 7fdb7e289640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 0x7fdb640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.152+0000 7fdb7e289640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 0x7fdb640a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009f40 tx=0x7fdb74005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.156+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdb7009bff0 con 0x7fdb78107730 2024-04-22T00:02:12.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:11 smithi203 ceph-mon[31431]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:12.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:11 smithi203 ceph-mon[31431]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-22T00:02:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3847291181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2583098392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:11 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3251544865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.314 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64531]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:12.314 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:12.314 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64531]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:12.314 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64531]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:12.315 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:11 smithi111 ceph-mon[31543]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T00:02:12.315 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:11 smithi111 ceph-mon[31543]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-22T00:02:12.315 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3847291181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:12.315 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2583098392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.315 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:11 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3251544865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.340+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdb78109700 con 0x7fdb78107730 2024-04-22T00:02:12.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.340+0000 7fdb6b7fe640 1 -- 172.21.15.38:0/1048826978 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7fdb7009b8f0 con 0x7fdb78107730 2024-04-22T00:02:12.333 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 msgr2=0x7fdb640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 0x7fdb640a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdb74009f40 tx=0x7fdb74005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 msgr2=0x7fdb7819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000efc0 tx=0x7fdb7000c800 comp rx=0 tx=0).stop 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 shutdown_connections 2024-04-22T00:02:12.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb640a2e20 0x7fdb640a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb78107730 0x7fdb7819c840 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb7806a780 0x7fdb781984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 --2- 172.21.15.38:0/1048826978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb78069dc0 0x7fdb78197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 >> 172.21.15.38:0/1048826978 conn(0x7fdb780776a0 msgr2=0x7fdb780fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 shutdown_connections 2024-04-22T00:02:12.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.342+0000 7fdb804ed640 1 -- 172.21.15.38:0/1048826978 wait complete. 2024-04-22T00:02:12.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 -- 172.21.15.38:0/2458430556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 msgr2=0x7fd14c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 --2- 172.21.15.38:0/2458430556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c100aa0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009930 tx=0x7fd13c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 -- 172.21.15.38:0/2458430556 shutdown_connections 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 --2- 172.21.15.38:0/2458430556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c100fe0 0x7fd14c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 --2- 172.21.15.38:0/2458430556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c100aa0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 --2- 172.21.15.38:0/2458430556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c106630 0x7fd14c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 -- 172.21.15.38:0/2458430556 >> 172.21.15.38:0/2458430556 conn(0x7fd14c077740 msgr2=0x7fd14c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 -- 172.21.15.38:0/2458430556 shutdown_connections 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.456+0000 7fd153b77640 1 -- 172.21.15.38:0/2458430556 wait complete. 2024-04-22T00:02:12.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 Processor -- start 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 -- start start 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c100fe0 0x7fd14c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 0x7fd14c1a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c110a50 con 0x7fd14c100620 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c1108d0 con 0x7fd14c100fe0 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd153b77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c110bd0 con 0x7fd14c106630 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40272/0 (socket says 172.21.15.38:40272) 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 -- 172.21.15.38:0/3971307225 learned_addr learned my addr 172.21.15.38:0/3971307225 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd152114640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 0x7fd14c1a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151112640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c100fe0 0x7fd14c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 -- 172.21.15.38:0/3971307225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 msgr2=0x7fd14c1a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 0x7fd14c1a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 -- 172.21.15.38:0/3971307225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c100fe0 msgr2=0x7fd14c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c100fe0 0x7fd14c199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 -- 172.21.15.38:0/3971307225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c1a4490 con 0x7fd14c100620 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd152114640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 0x7fd14c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.457+0000 7fd151913640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e9f0 tx=0x7fd14800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.458+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14800cd80 con 0x7fd14c100620 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.458+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd148010910 con 0x7fd14c100620 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.458+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c1a4660 con 0x7fd14c100620 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.458+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148018990 con 0x7fd14c100620 2024-04-22T00:02:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.458+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c1a4b20 con 0x7fd14c100620 2024-04-22T00:02:12.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.459+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14c068560 con 0x7fd14c100620 2024-04-22T00:02:12.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.460+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fd148010ac0 con 0x7fd14c100620 2024-04-22T00:02:12.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.460+0000 7fd142ffd640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 0x7fd1380a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.460+0000 7fd151112640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 0x7fd1380a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.460+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd1480cfe30 con 0x7fd14c100620 2024-04-22T00:02:12.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.461+0000 7fd151112640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 0x7fd1380a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd13c036040 tx=0x7fd13c02df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.465+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd14809c4c0 con 0x7fd14c100620 2024-04-22T00:02:12.576 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64534]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:12.576 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:12.576 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64534]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:12.576 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:02:12 smithi111 sudo[64534]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:12.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.687+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd14c02d170 con 0x7fd14c100620 2024-04-22T00:02:12.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.688+0000 7fd142ffd640 1 -- 172.21.15.38:0/3971307225 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7fd14809bdc0 con 0x7fd14c100620 2024-04-22T00:02:12.682 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:12.682 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:02:03.216530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:02:12.682 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 msgr2=0x7fd1380a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 0x7fd1380a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd13c036040 tx=0x7fd13c02df90 comp rx=0 tx=0).stop 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 msgr2=0x7fd14c198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e9f0 tx=0x7fd14800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 shutdown_connections 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd1380a2e20 0x7fd1380a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd14c106630 0x7fd14c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd14c100fe0 0x7fd14c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 --2- 172.21.15.38:0/3971307225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14c100620 0x7fd14c198ed0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 >> 172.21.15.38:0/3971307225 conn(0x7fd14c077740 msgr2=0x7fd14c102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 shutdown_connections 2024-04-22T00:02:12.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.690+0000 7fd153b77640 1 -- 172.21.15.38:0/3971307225 wait complete. 2024-04-22T00:02:12.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:02:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2315033483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 msgr2=0x7fa0e410d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2315033483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e410d510 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009cd0 tx=0x7fa0dc02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2315033483 shutdown_connections 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2315033483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e410d510 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2315033483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4100690 0x7fa0e4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2315033483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4106650 0x7fa0e4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2315033483 >> 172.21.15.38:0/2315033483 conn(0x7fa0e40777a0 msgr2=0x7fa0e4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2315033483 shutdown_connections 2024-04-22T00:02:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.804+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2315033483 wait complete. 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 Processor -- start 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 -- start start 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 0x7fa0e4196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 0x7fa0e419b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e4110a60 con 0x7fa0e4101050 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41108e0 con 0x7fa0e4106650 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e8c39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e4110be0 con 0x7fa0e4100690 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 0x7fa0e4196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40286/0 (socket says 172.21.15.38:40286) 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0dbfff640 1 -- 172.21.15.38:0/2702504568 learned_addr learned my addr 172.21.15.38:0/2702504568 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e2d9d640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 0x7fa0e419b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 -- 172.21.15.38:0/2702504568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 msgr2=0x7fa0e419b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 0x7fa0e419b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 -- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 msgr2=0x7fa0e4197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0e259c640 1 -- 172.21.15.38:0/2702504568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e419bdf0 con 0x7fa0e4100690 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.805+0000 7fa0dbfff640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0e2d9d640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 0x7fa0e419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0e259c640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 0x7fa0e4196e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e9f0 tx=0x7fa0d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400ccc0 con 0x7fa0e4100690 2024-04-22T00:02:12.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e419c0e0 con 0x7fa0e4100690 2024-04-22T00:02:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d4010910 con 0x7fa0e4100690 2024-04-22T00:02:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4018960 con 0x7fa0e4100690 2024-04-22T00:02:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.806+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e41a3970 con 0x7fa0e4100690 2024-04-22T00:02:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.807+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e40689b0 con 0x7fa0e4100690 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.808+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fa0d4010ac0 con 0x7fa0e4100690 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.809+0000 7fa0db7fe640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 0x7fa0d00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.809+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa0d40d00d0 con 0x7fa0e4100690 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.809+0000 7fa0dbfff640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 0x7fa0d00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.809+0000 7fa0dbfff640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 0x7fa0d00a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4077ca0 tx=0x7fa0cc00a380 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:12.812+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa0d409c700 con 0x7fa0e4100690 2024-04-22T00:02:12.826 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65575]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:12.827 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65575]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:12.827 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65575]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:12.827 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65575]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:12 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2089917296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:12 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1048826978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:12.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:12 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3971307225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:12.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.000+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0e4108620 con 0x7fa0e4100690 2024-04-22T00:02:12.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.001+0000 7fa0db7fe640 1 -- 172.21.15.38:0/2702504568 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+25015 (secure 0 0 0) 0x7fa0d407ae20 con 0x7fa0e4100690 2024-04-22T00:02:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:12.995 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":92,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:02:04.818207+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:02:12.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 msgr2=0x7fa0d00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 0x7fa0d00a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4077ca0 tx=0x7fa0cc00a380 comp rx=0 tx=0).stop 2024-04-22T00:02:12.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 msgr2=0x7fa0e4196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:12.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 0x7fa0e4196e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e9f0 tx=0x7fa0d400eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 shutdown_connections 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa0d00a2e20 0x7fa0d00a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0e4106650 0x7fa0e419b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.003+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa0e4101050 0x7fa0e4197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.004+0000 7fa0e8c39640 1 --2- 172.21.15.38:0/2702504568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0e4100690 0x7fa0e4196e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.004+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 >> 172.21.15.38:0/2702504568 conn(0x7fa0e40777a0 msgr2=0x7fa0e4103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.004+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 shutdown_connections 2024-04-22T00:02:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.004+0000 7fa0e8c39640 1 -- 172.21.15.38:0/2702504568 wait complete. 2024-04-22T00:02:13.009 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T00:02:13.009 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T00:02:13.019 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T00:02:13.019 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:02:13.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-22T00:02:13.020 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T00:02:13.020 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:02:13.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:13.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:02:13.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.134+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/2531697309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 msgr2=0x7fbdf010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.134+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/2531697309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf010b7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009860 tx=0x7fbde002bca0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.134+0000 7f97219e7640 1 -- 172.21.15.38:0/3108068395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c100670 msgr2=0x7f971c100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.134+0000 7f97219e7640 1 --2- 172.21.15.38:0/3108068395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c100670 0x7f971c100af0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9710009890 tx=0x7f971002bb70 comp rx=0 tx=0).stop 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.134+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/2531697309 shutdown_connections 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/2531697309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf010bd00 0x7fbdf010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/2531697309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf010b7c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/2531697309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf01029c0 0x7fbdf0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/2531697309 >> 172.21.15.38:0/2531697309 conn(0x7fbdf0077740 msgr2=0x7fbdf0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 -- 172.21.15.38:0/3108068395 shutdown_connections 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 --2- 172.21.15.38:0/3108068395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c101030 0x7f971c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 --2- 172.21.15.38:0/3108068395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c100670 0x7f971c100af0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 --2- 172.21.15.38:0/3108068395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c106630 0x7f971c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 -- 172.21.15.38:0/3108068395 >> 172.21.15.38:0/3108068395 conn(0x7f971c0777c0 msgr2=0x7f971c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 -- 172.21.15.38:0/3108068395 shutdown_connections 2024-04-22T00:02:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 -- 172.21.15.38:0/3108068395 wait complete. 2024-04-22T00:02:13.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/2531697309 shutdown_connections 2024-04-22T00:02:13.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/2531697309 wait complete. 2024-04-22T00:02:13.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 Processor -- start 2024-04-22T00:02:13.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 Processor -- start 2024-04-22T00:02:13.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- start start 2024-04-22T00:02:13.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 0x7fbdf0100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7f97219e7640 1 -- start start 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf010bd00 0x7fbdf0101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0111b00 con 0x7fbdf01029c0 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0111980 con 0x7fbdf01032e0 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.135+0000 7fbdf7d8c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0111c80 con 0x7fbdf010bd00 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 0x7fbdf0100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51030/0 (socket says 172.21.15.38:51030) 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 -- 172.21.15.38:0/3292444121 learned_addr learned my addr 172.21.15.38:0/3292444121 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 0x7fbdf0100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40316/0 (socket says 172.21.15.38:40316) 2024-04-22T00:02:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf6329640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf010bd00 0x7fbdf0101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 0x7f971c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 0x7f971c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f971c110a00 con 0x7f971c101030 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f971c110880 con 0x7f971c100670 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 0x7f971c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 -- 172.21.15.38:0/3292444121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf010bd00 msgr2=0x7fbdf0101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf010bd00 0x7fbdf0101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 -- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 msgr2=0x7fbdf0100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 0x7fbdf0100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 -- 172.21.15.38:0/3292444121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf01a6520 con 0x7fbdf01032e0 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51034/0 (socket says 172.21.15.38:51034) 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 -- 172.21.15.38:0/3785700319 learned_addr learned my addr 172.21.15.38:0/3785700319 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f97219e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f971c110b80 con 0x7f971c106630 2024-04-22T00:02:13.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971b7fe640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 0x7f971c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbde0037040 tx=0x7fbde002c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 -- 172.21.15.38:0/3785700319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 msgr2=0x7f971c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 0x7f971c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 -- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 msgr2=0x7f971c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 0x7f971c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbde6ffd640 1 -- 172.21.15.38:0/3292444121 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0041020 con 0x7fbdf01032e0 2024-04-22T00:02:13.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971affd640 1 -- 172.21.15.38:0/3785700319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971c19cfa0 con 0x7f971c100670 2024-04-22T00:02:13.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971a7fc640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 0x7f971c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:13.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7f971b7fe640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 0x7f971c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:13.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf01a67b0 con 0x7fbdf01032e0 2024-04-22T00:02:13.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.136+0000 7fbde6ffd640 1 -- 172.21.15.38:0/3292444121 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde00044d0 con 0x7fbdf01032e0 2024-04-22T00:02:13.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7fbde6ffd640 1 -- 172.21.15.38:0/3292444121 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0039040 con 0x7fbdf01032e0 2024-04-22T00:02:13.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf01a6cf0 con 0x7fbdf01032e0 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f971affd640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f970c00e9c0 tx=0x7f970c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970c00cde0 con 0x7f971c100670 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f970c0035f0 con 0x7f971c100670 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970c010690 con 0x7f971c100670 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971c19d290 con 0x7f971c100670 2024-04-22T00:02:13.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.137+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f971c1a4b20 con 0x7f971c100670 2024-04-22T00:02:13.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.138+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971c04f500 con 0x7f971c100670 2024-04-22T00:02:13.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.139+0000 7fbde6ffd640 1 -- 172.21.15.38:0/3292444121 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fbde0044020 con 0x7fbdf01032e0 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.139+0000 7fbde6ffd640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 0x7fbddc0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.139+0000 7fbdfa97a740 1 Processor -- start 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbde6ffd640 1 -- 172.21.15.38:0/3292444121 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbde00f1130 con 0x7fbdf01032e0 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 0x7fbddc0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 -- start start 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec20ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56147eb84170 con 0x56147ec21720 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56147eb2c7c0 con 0x56147ec20a40 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56147eb80130 con 0x56147ec29db0 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40340/0 (socket says 172.21.15.38:40340) 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 -- 172.21.15.38:0/2665848406 learned_addr learned my addr 172.21.15.38:0/2665848406 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5327640 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec20ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 0x7fbddc0a5360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0095f0 tx=0x7fbdec005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf6329640 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 -- 172.21.15.38:0/2665848406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 msgr2=0x56147ec2c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 -- 172.21.15.38:0/2665848406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 msgr2=0x56147ec20ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.140+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec20ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7fbdf5b28640 1 -- 172.21.15.38:0/2665848406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56147eb3de60 con 0x56147ec21720 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00e640 tx=0x7fbdec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad21e87665cd8e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f970c0037a0 con 0x7f971c100670 2024-04-22T00:02:13.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7fbddbfff640 1 -- 172.21.15.38:0/2665848406 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec004020 con 0x56147ec21720 2024-04-22T00:02:13.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7fbddbfff640 1 -- 172.21.15.38:0/2665848406 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdec0041c0 con 0x56147ec21720 2024-04-22T00:02:13.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7fbddbfff640 1 -- 172.21.15.38:0/2665848406 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec01d7e0 con 0x56147ec21720 2024-04-22T00:02:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.141+0000 7f97209e5640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 0x7f97040a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7f971a7fc640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 0x7f97040a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f970c0cfbf0 con 0x7f971c100670 2024-04-22T00:02:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7f971a7fc640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 0x7f97040a5400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97100097b0 tx=0x7f971002e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 -- 172.21.15.38:0/2665848406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 msgr2=0x56147ec20470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00e640 tx=0x7fbdec014000 comp rx=0 tx=0).stop 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 -- 172.21.15.38:0/2665848406 shutdown_connections 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec20ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/2665848406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec20470 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.142+0000 7fbdfa97a740 1 -- 172.21.15.38:0/2665848406 >> 172.21.15.38:0/2665848406 conn(0x56147ebe74f0 msgr2=0x56147ec1b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.143+0000 7fbdfa97a740 1 -- 172.21.15.38:0/2665848406 shutdown_connections 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.143+0000 7fbdfa97a740 1 -- 172.21.15.38:0/2665848406 wait complete. 2024-04-22T00:02:13.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.143+0000 7fbdfa97a740 1 -- start start 2024-04-22T00:02:13.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.143+0000 7fbdfa97a740 1 Processor -- start 2024-04-22T00:02:13.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f970c09c280 con 0x7f971c100670 2024-04-22T00:02:13.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 0x56147ec27400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec27940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56147ec2cc20 con 0x56147ec29db0 2024-04-22T00:02:13.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56147eb3fd40 con 0x56147ec21720 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56147ec2cda0 con 0x56147ec20a40 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdfa97a740 10 client.0 _ll_register_callbacks cb 0x56147eb7b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf6329640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec27940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf5b28640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:41496/0 (socket says 172.21.15.38:41496) 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf6329640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec27940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40356/0 (socket says 172.21.15.38:40356) 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf5b28640 1 -- 172.21.15.38:0/3093665707 learned_addr learned my addr 172.21.15.38:0/3093665707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.144+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 0x56147ec27400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 -- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 msgr2=0x56147ec27400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 0x56147ec27400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 -- 172.21.15.38:0/3093665707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 msgr2=0x56147ec27940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec27940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 -- 172.21.15.38:0/3093665707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56147e9607e0 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 0x56147ec27400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0118d0 tx=0x7fbdec004000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2aeb5c74d890ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec01d560 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbdec01d700 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbdec0041c0 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbdec038050 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdfa97a740 10 client.25282 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56147e9c0f70 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.145+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdec0041c0 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbdec0035f0 con 0x56147ec20a40 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 10 client.25282 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 10 client.25282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 20 client.25282 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147ec39e70 0x56147ec3c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 4 client.25282 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56147eb58450 con 0x56147ec39e70 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147ec39e70 0x56147ec3c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdd97fa640 10 client.25282 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.146+0000 7fbdf5327640 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147ec39e70 0x56147ec3c260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=100f9cf921d71d1a server_cookie=84b4400a18e1f48e in_seq=0 out_seq=0 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdd97fa640 1 -- 172.21.15.38:0/3093665707 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56147eb58450 con 0x56147ec39e70 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdd97fa640 10 client.25282 handle_command_reply: tid=0 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 client.25282 shutdown 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 msgr2=0x56147ec26ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0118d0 tx=0x7fbdec004000 comp rx=0 tx=0).stop 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 shutdown_connections 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147ec39e70 0x56147ec3c260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec27940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec21720 0x56147ec27400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/3093665707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec20a40 0x56147ec26ec0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdd97fa640 0 client.25282 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 >> 172.21.15.38:0/3093665707 conn(0x56147ebe74f0 msgr2=0x56147ec1d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.147+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 shutdown_connections 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.148+0000 7fbdfa97a740 1 -- 172.21.15.38:0/3093665707 wait complete. 2024-04-22T00:02:13.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.148+0000 7fbdfa97a740 20 client.25282 trim_cache size 0 max 16384 2024-04-22T00:02:13.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.165+0000 7fbdfa97a740 1 Processor -- start 2024-04-22T00:02:13.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.165+0000 7fbdfa97a740 1 -- start start 2024-04-22T00:02:13.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec295f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec21720 0x56147ec35b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01b0c20 con 0x56147ec29db0 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01b2760 con 0x56147ec20a40 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01b12f0 con 0x56147ec21720 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40372/0 (socket says 172.21.15.38:40372) 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 -- 172.21.15.38:0/1552600085 learned_addr learned my addr 172.21.15.38:0/1552600085 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec295f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf5327640 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec21720 0x56147ec35b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 -- 172.21.15.38:0/1552600085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec21720 msgr2=0x56147ec35b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec21720 0x56147ec35b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 -- 172.21.15.38:0/1552600085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 msgr2=0x56147ec295f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec295f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 -- 172.21.15.38:0/1552600085 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56147e858170 con 0x56147ec29db0 2024-04-22T00:02:13.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.166+0000 7fbdf6329640 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbde80105a0 tx=0x7fbde8010a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf57c39bd87ac444 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbddb7fe640 1 -- 172.21.15.38:0/1552600085 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde800d780 con 0x56147ec29db0 2024-04-22T00:02:13.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbddb7fe640 1 -- 172.21.15.38:0/1552600085 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde801f070 con 0x56147ec29db0 2024-04-22T00:02:13.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbddb7fe640 1 -- 172.21.15.38:0/1552600085 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde800ecc0 con 0x56147ec29db0 2024-04-22T00:02:13.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 -- 172.21.15.38:0/1552600085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 msgr2=0x56147ec418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbde80105a0 tx=0x7fbde8010a60 comp rx=0 tx=0).stop 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 -- 172.21.15.38:0/1552600085 shutdown_connections 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec29db0 0x56147ec418c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec21720 0x56147ec35b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/1552600085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec295f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 -- 172.21.15.38:0/1552600085 >> 172.21.15.38:0/1552600085 conn(0x56147ebe74f0 msgr2=0x56147ec1d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 -- 172.21.15.38:0/1552600085 shutdown_connections 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.167+0000 7fbdfa97a740 1 -- 172.21.15.38:0/1552600085 wait complete. 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 -- start start 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 Processor -- start 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec32030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01b2090 con 0x56147ec21720 2024-04-22T00:02:13.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01b19c0 con 0x56147ec20a40 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.168+0000 7fbdfa97a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01aec00 con 0x56147ec29db0 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdfa97a740 10 client.0 _ll_register_callbacks cb 0x7fbde0005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40386/0 (socket says 172.21.15.38:40386) 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 -- 172.21.15.38:0/4102569412 learned_addr learned my addr 172.21.15.38:0/4102569412 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec32030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf6329640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 -- 172.21.15.38:0/4102569412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 msgr2=0x56147ec2afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 -- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 msgr2=0x56147ec32030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec32030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 -- 172.21.15.38:0/4102569412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56147ec2b4f0 con 0x56147ec21720 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec32030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:13.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdf5327640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbde002c2b0 tx=0x7fbde00dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39328097609d8c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00d5070 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbde00035f0 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.169+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbde801f070 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbde00c5e40 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 10 client.15753 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56147eca09d0 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde801f070 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbde003c030 con 0x56147ec21720 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 10 client.15753 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 20 client.15753 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147eca1100 0x56147eca3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 4 client.15753 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56147eb916b0 con 0x56147eca1100 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.170+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147eca1100 0x56147eca3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.171+0000 7fbdf5b28640 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147eca1100 0x56147eca3520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee3aaccce366a0c2 server_cookie=5c9afaa0930e3b21 in_seq=0 out_seq=0 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.171+0000 7fbdd97fa640 10 client.15753 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.171+0000 7fbdd97fa640 1 -- 172.21.15.38:0/4102569412 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56147eb916b0 con 0x56147eca1100 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.171+0000 7fbdd97fa640 10 client.15753 handle_command_reply: tid=0 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 client.15753 shutdown 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 msgr2=0x56147ec32570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fbde002c2b0 tx=0x7fbde00dd040 comp rx=0 tx=0).stop 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 shutdown_connections 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56147ec29db0 0x56147ec2afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56147eca1100 0x56147eca3520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56147ec21720 0x56147ec32570 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 --2- 172.21.15.38:0/4102569412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56147ec20a40 0x56147ec32030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdd97fa640 0 client.15753 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 >> 172.21.15.38:0/4102569412 conn(0x56147ebe74f0 msgr2=0x56147ec26e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 shutdown_connections 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 1 -- 172.21.15.38:0/4102569412 wait complete. 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.172+0000 7fbdfa97a740 20 client.15753 trim_cache size 0 max 16384 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "77f2d943-70ea-45c7-a5ca-d1c7eedf4353", 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:02:13.171 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:13.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 msgr2=0x7fbddc0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 0x7fbddc0a5360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0095f0 tx=0x7fbdec005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:13.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 msgr2=0x7fbdf0100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbde0037040 tx=0x7fbde002c8f0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 shutdown_connections 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbddc0a2ea0 0x7fbddc0a5360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdf010bd00 0x7fbdf0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbdf01032e0 0x7fbdf0100b50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 --2- 172.21.15.38:0/3292444121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbdf01029c0 0x7fbdf0100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 >> 172.21.15.38:0/3292444121 conn(0x7fbdf0077740 msgr2=0x7fbdf00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 shutdown_connections 2024-04-22T00:02:13.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.173+0000 7fbdf7d8c640 1 -- 172.21.15.38:0/3292444121 wait complete. 2024-04-22T00:02:13.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:02:13.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:02:13.279 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65578]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:13.279 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:13.279 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65578]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:13.279 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:02:12 smithi203 sudo[65578]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:13.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:12 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2089917296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:13.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:12 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1048826978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:13.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:12 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3971307225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 -- 172.21.15.38:0/2939111318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc0ffc50 msgr2=0x7fafdc1000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/2939111318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc0ffc50 0x7fafdc1000b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009930 tx=0x7fafd002bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 -- 172.21.15.38:0/2939111318 shutdown_connections 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/2939111318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc1005f0 0x7fafdc10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/2939111318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc0ffc50 0x7fafdc1000b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/2939111318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc107710 0x7fafdc107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 -- 172.21.15.38:0/2939111318 >> 172.21.15.38:0/2939111318 conn(0x7fafdc076be0 msgr2=0x7fafdc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 -- 172.21.15.38:0/2939111318 shutdown_connections 2024-04-22T00:02:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.300+0000 7fafe1f1e640 1 -- 172.21.15.38:0/2939111318 wait complete. 2024-04-22T00:02:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.301+0000 7fafe1f1e640 1 Processor -- start 2024-04-22T00:02:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.301+0000 7fafe1f1e640 1 -- start start 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc0ffc50 0x7fafdc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc107710 0x7fafdc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc111ba0 con 0x7fafdc1005f0 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc111a20 con 0x7fafdc0ffc50 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafe1f1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc111d20 con 0x7fafdc107710 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40402/0 (socket says 172.21.15.38:40402) 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 -- 172.21.15.38:0/3969420680 learned_addr learned my addr 172.21.15.38:0/3969420680 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdbfff640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc107710 0x7fafdc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc0ffc50 0x7fafdc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 -- 172.21.15.38:0/3969420680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc107710 msgr2=0x7fafdc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc107710 0x7fafdc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 -- 172.21.15.38:0/3969420680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc0ffc50 msgr2=0x7fafdc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc0ffc50 0x7fafdc198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 -- 172.21.15.38:0/3969420680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc19cff0 con 0x7fafdc1005f0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.302+0000 7fafdaffd640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009900 tx=0x7fafd0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.303+0000 7fafd8ff9640 1 -- 172.21.15.38:0/3969420680 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0043020 con 0x7fafdc1005f0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.303+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc19d280 con 0x7fafdc1005f0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.303+0000 7fafd8ff9640 1 -- 172.21.15.38:0/3969420680 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd00054e0 con 0x7fafdc1005f0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.304+0000 7fafd8ff9640 1 -- 172.21.15.38:0/3969420680 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0003bb0 con 0x7fafdc1005f0 2024-04-22T00:02:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.304+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1a4b70 con 0x7fafdc1005f0 2024-04-22T00:02:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.305+0000 7fafd8ff9640 1 -- 172.21.15.38:0/3969420680 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fafd0046020 con 0x7fafdc1005f0 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 Processor -- start 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafd8ff9640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 0x7fafc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 -- start start 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 0x7fafc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafd8ff9640 1 -- 172.21.15.38:0/3969420680 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fafd00f1570 con 0x7fafdc1005f0 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 0x55e6ee551fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee561c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee4b6560 con 0x55e6ee552500 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafe4be3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee45e730 con 0x55e6ee55f840 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.306+0000 7fafdaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafe4be3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee4b2520 con 0x55e6ee551be0 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 0x55e6ee551fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40418/0 (socket says 172.21.15.38:40418) 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 -- 172.21.15.38:0/2461824411 learned_addr learned my addr 172.21.15.38:0/2461824411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdbfff640 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee561c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 0x7fafc40a51c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009cf0 tx=0x7fafcc011810 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 -- 172.21.15.38:0/2461824411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 msgr2=0x55e6ee551fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 0x55e6ee551fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 -- 172.21.15.38:0/2461824411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 msgr2=0x55e6ee561c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee561c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 -- 172.21.15.38:0/2461824411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6ee2947d0 con 0x55e6ee552500 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 0x55e6ee551fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fafd00bd5a0 tx=0x7fafd0004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50651840f8ce308 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafc1ffb640 1 -- 172.21.15.38:0/2461824411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd00d6070 con 0x55e6ee552500 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafc1ffb640 1 -- 172.21.15.38:0/2461824411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd00b9040 con 0x55e6ee552500 2024-04-22T00:02:13.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.307+0000 7fafc1ffb640 1 -- 172.21.15.38:0/2461824411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0004c40 con 0x55e6ee552500 2024-04-22T00:02:13.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 -- 172.21.15.38:0/2461824411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 msgr2=0x55e6ee55f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fafd00bd5a0 tx=0x7fafd0004ee0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 -- 172.21.15.38:0/2461824411 shutdown_connections 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee561c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee552500 0x55e6ee55f300 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2461824411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee551be0 0x55e6ee551fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 -- 172.21.15.38:0/2461824411 >> 172.21.15.38:0/2461824411 conn(0x55e6ee54c660 msgr2=0x55e6ee54ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 -- 172.21.15.38:0/2461824411 shutdown_connections 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.308+0000 7fafe4be3740 1 -- 172.21.15.38:0/2461824411 wait complete. 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 -- start start 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 Processor -- start 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee55ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee55d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee562440 con 0x55e6ee551be0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee471e00 con 0x55e6ee55f840 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e6ee5625c0 con 0x55e6ee552500 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafe4be3740 10 client.0 _ll_register_callbacks cb 0x55e6ee4aceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40426/0 (socket says 172.21.15.38:40426) 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafdb7fe640 1 -- 172.21.15.38:0/4138572665 learned_addr learned my addr 172.21.15.38:0/4138572665 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafdaffd640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee55ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.309+0000 7fafdbfff640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee55d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 -- 172.21.15.38:0/4138572665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 msgr2=0x55e6ee55ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee55ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 -- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 msgr2=0x55e6ee55d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee55d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 -- 172.21.15.38:0/4138572665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6ee1ca3f0 con 0x55e6ee551be0 2024-04-22T00:02:13.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdaffd640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee55ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdbfff640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee55d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fafcc01eb40 tx=0x7fafcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d55158e44a0e8e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafbaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc01dd90 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fafcc005020 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafd00b9040 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fafcc005220 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafe4be3740 10 client.15765 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.310+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6ee18a0f0 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd00b9040 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fafcc025070 con 0x55e6ee551be0 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 10 client.15765 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 20 client.15765 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee56fa70 0x55e6ee571ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 4 client.15765 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.311+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6ee4c3aa0 con 0x55e6ee56fa70 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.312+0000 7fafdaffd640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee56fa70 0x55e6ee571ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.312+0000 7fafdaffd640 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee56fa70 0x55e6ee571ea0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f3095c7a0b26e86 server_cookie=304216c3c4c19575 in_seq=0 out_seq=0 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.312+0000 7fafbaffd640 10 client.15765 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafbaffd640 1 -- 172.21.15.38:0/4138572665 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6ee4c3aa0 con 0x55e6ee56fa70 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafbaffd640 10 client.15765 handle_command_reply: tid=0 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 client.15765 shutdown 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 msgr2=0x55e6ee55c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fafcc01eb40 tx=0x7fafcc004730 comp rx=0 tx=0).stop 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 shutdown_connections 2024-04-22T00:02:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee55d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee55ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee56fa70 0x55e6ee571ea0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 --2- 172.21.15.38:0/4138572665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee55c900 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 >> 172.21.15.38:0/4138572665 conn(0x55e6ee54c660 msgr2=0x55e6ee54e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafbaffd640 0 client.15765 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 shutdown_connections 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 1 -- 172.21.15.38:0/4138572665 wait complete. 2024-04-22T00:02:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.313+0000 7fafe4be3740 20 client.15765 trim_cache size 0 max 16384 2024-04-22T00:02:13.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:12 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2089917296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:02:13.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:12 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1048826978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:02:13.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:12 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3971307225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 Processor -- start 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 -- start start 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee552500 0x55e6ee55c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee55f840 0x55e6ee55cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ac5b0 con 0x55e6ee551be0 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ae0f0 con 0x55e6ee552500 2024-04-22T00:02:13.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.331+0000 7fafe4be3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1acc80 con 0x55e6ee55f840 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40436/0 (socket says 172.21.15.38:40436) 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 -- 172.21.15.38:0/1354638015 learned_addr learned my addr 172.21.15.38:0/1354638015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdaffd640 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee552500 0x55e6ee55c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdbfff640 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee55f840 0x55e6ee55cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 -- 172.21.15.38:0/1354638015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee55f840 msgr2=0x55e6ee55cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee55f840 0x55e6ee55cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 -- 172.21.15.38:0/1354638015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee552500 msgr2=0x55e6ee55c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee552500 0x55e6ee55c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 -- 172.21.15.38:0/1354638015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6ee551020 con 0x55e6ee551be0 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0097c0 tx=0x7fafcc026980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9ff0a27c601f28 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafc17fa640 1 -- 172.21.15.38:0/1354638015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc009910 con 0x55e6ee551be0 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafc17fa640 1 -- 172.21.15.38:0/1354638015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafcc0053c0 con 0x55e6ee551be0 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.332+0000 7fafc17fa640 1 -- 172.21.15.38:0/1354638015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0047a0 con 0x55e6ee551be0 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- 172.21.15.38:0/1354638015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 msgr2=0x55e6ee56b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0097c0 tx=0x7fafcc026980 comp rx=0 tx=0).stop 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- 172.21.15.38:0/1354638015 shutdown_connections 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee55f840 0x55e6ee55cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee552500 0x55e6ee55c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 --2- 172.21.15.38:0/1354638015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee56b300 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- 172.21.15.38:0/1354638015 >> 172.21.15.38:0/1354638015 conn(0x55e6ee54c660 msgr2=0x55e6ee55c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- 172.21.15.38:0/1354638015 shutdown_connections 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- 172.21.15.38:0/1354638015 wait complete. 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.333+0000 7fafe4be3740 1 -- start start 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 Processor -- start 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee567d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee559130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ada20 con 0x55e6ee551be0 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ad350 con 0x55e6ee55f840 2024-04-22T00:02:13.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1aa590 con 0x55e6ee552500 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafe4be3740 10 client.0 _ll_register_callbacks cb 0x7fafcc009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40446/0 (socket says 172.21.15.38:40446) 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 -- 172.21.15.38:0/2155404953 learned_addr learned my addr 172.21.15.38:0/2155404953 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdbfff640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee559130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee567d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 -- 172.21.15.38:0/2155404953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 msgr2=0x55e6ee567d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee567d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 -- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 msgr2=0x55e6ee559130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee559130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.334+0000 7fafdb7fe640 1 -- 172.21.15.38:0/2155404953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6ee559960 con 0x55e6ee551be0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafdb7fe640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00a010 tx=0x7fafcc01e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eab3116779c1bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafbaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc042060 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fafcc009040 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafcc0053c0 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fafcc009910 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafe4be3740 10 client.15771 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e6ee5d5ba0 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.335+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafcc0053c0 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fafcc045020 con 0x55e6ee551be0 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 10 client.15771 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 20 client.15771 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee5d61b0 0x55e6ee5d85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 4 client.15771 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e6ee4bddf0 con 0x55e6ee5d61b0 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee5d61b0 0x55e6ee5d85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafbaffd640 10 client.15771 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafdaffd640 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee5d61b0 0x55e6ee5d85a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21be7a4e39a66dd server_cookie=1a9b79b7403a727d in_seq=0 out_seq=0 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafbaffd640 1 -- 172.21.15.38:0/2155404953 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e6ee4bddf0 con 0x55e6ee5d61b0 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafbaffd640 10 client.15771 handle_command_reply: tid=0 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.336+0000 7fafe4be3740 1 client.15771 shutdown 2024-04-22T00:02:13.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 msgr2=0x55e6ee567810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00a010 tx=0x7fafcc01e860 comp rx=0 tx=0).stop 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 shutdown_connections 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e6ee55f840 0x55e6ee559130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6ee552500 0x55e6ee567d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e6ee5d61b0 0x55e6ee5d85a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 --2- 172.21.15.38:0/2155404953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e6ee551be0 0x55e6ee567810 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafbaffd640 0 client.15771 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 >> 172.21.15.38:0/2155404953 conn(0x55e6ee54c660 msgr2=0x55e6ee55d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 shutdown_connections 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 1 -- 172.21.15.38:0/2155404953 wait complete. 2024-04-22T00:02:13.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.337+0000 7fafe4be3740 20 client.15771 trim_cache size 0 max 16384 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "77f2d943-70ea-45c7-a5ca-d1c7eedf4353": { 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "77f2d943-70ea-45c7-a5ca-d1c7eedf4353", 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:02:13.341 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:13.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 msgr2=0x7fafc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 0x7fafc40a51c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009cf0 tx=0x7fafcc011810 comp rx=0 tx=0).stop 2024-04-22T00:02:13.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 msgr2=0x7fafdc198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009900 tx=0x7fafd0004360 comp rx=0 tx=0).stop 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 shutdown_connections 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafc40a2d00 0x7fafc40a51c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafdc107710 0x7fafdc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fafdc1005f0 0x7fafdc198540 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 --2- 172.21.15.38:0/3969420680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafdc0ffc50 0x7fafdc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 >> 172.21.15.38:0/3969420680 conn(0x7fafdc076be0 msgr2=0x7fafdc0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 shutdown_connections 2024-04-22T00:02:13.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.338+0000 7fafe1f1e640 1 -- 172.21.15.38:0/3969420680 wait complete. 2024-04-22T00:02:13.351 INFO:tasks.cephfs.filesystem:scrub status for tag:77f2d943-70ea-45c7-a5ca-d1c7eedf4353 - {'path': '/', 'tag': '77f2d943-70ea-45c7-a5ca-d1c7eedf4353', 'options': 'recursive,force'} 2024-04-22T00:02:13.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.362+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f971c068df0 con 0x7f97040a2f40 2024-04-22T00:02:13.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.427+0000 7f97209e5640 1 -- 172.21.15.38:0/3785700319 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f971c068df0 con 0x7f97040a2f40 2024-04-22T00:02:13.421 INFO:teuthology.orchestra.run.smithi038.stdout:Schedule set for path / 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 msgr2=0x7f97040a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 0x7f97040a5400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97100097b0 tx=0x7f971002e040 comp rx=0 tx=0).stop 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 msgr2=0x7f971c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f970c00e9c0 tx=0x7f970c00ee80 comp rx=0 tx=0).stop 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 shutdown_connections 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97040a2f40 0x7f97040a5400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971c106630 0x7f971c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f971c101030 0x7f971c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 --2- 172.21.15.38:0/3785700319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f971c100670 0x7f971c198000 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 >> 172.21.15.38:0/3785700319 conn(0x7f971c0777c0 msgr2=0x7f971c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 shutdown_connections 2024-04-22T00:02:13.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.429+0000 7f97219e7640 1 -- 172.21.15.38:0/3785700319 wait complete. 2024-04-22T00:02:13.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-22T00:02:13.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 -- 172.21.15.38:0/2887296598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 msgr2=0x7f3a20107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2887296598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a20107b20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009930 tx=0x7f3a1002bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 -- 172.21.15.38:0/2887296598 shutdown_connections 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2887296598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a201020f0 0x7f3a2010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2887296598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a20101730 0x7f3a20101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2887296598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a20107b20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 -- 172.21.15.38:0/2887296598 >> 172.21.15.38:0/2887296598 conn(0x7f3a200fd200 msgr2=0x7f3a200ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 -- 172.21.15.38:0/2887296598 shutdown_connections 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.585+0000 7f3a26a37640 1 -- 172.21.15.38:0/2887296598 wait complete. 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 Processor -- start 2024-04-22T00:02:13.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 -- start start 2024-04-22T00:02:13.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a20101730 0x7f3a20198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a201020f0 0x7f3a20198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20111b50 con 0x7f3a20107740 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201119d0 con 0x7f3a201020f0 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a26a37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20111cd0 con 0x7f3a20101730 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a24fd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a24fd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40464/0 (socket says 172.21.15.38:40464) 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a24fd4640 1 -- 172.21.15.38:0/2490159234 learned_addr learned my addr 172.21.15.38:0/2490159234 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.586+0000 7f3a1f7fe640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a201020f0 0x7f3a20198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a1ffff640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a20101730 0x7f3a20198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 -- 172.21.15.38:0/2490159234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a20101730 msgr2=0x7f3a20198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a20101730 0x7f3a20198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 -- 172.21.15.38:0/2490159234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a201020f0 msgr2=0x7f3a20198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a201020f0 0x7f3a20198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 -- 172.21.15.38:0/2490159234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2019cfd0 con 0x7f3a20107740 2024-04-22T00:02:13.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a24fd4640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800eb10 tx=0x7f3a1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004b20 con 0x7f3a20107740 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2019d2c0 con 0x7f3a20107740 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a201a4b50 con 0x7f3a20107740 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a18004cc0 con 0x7f3a20107740 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.587+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18010760 con 0x7f3a20107740 2024-04-22T00:02:13.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.588+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a00005290 con 0x7f3a20107740 2024-04-22T00:02:13.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.590+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f3a18005000 con 0x7f3a20107740 2024-04-22T00:02:13.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.590+0000 7f3a1d7fa640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 0x7f3a0c0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.591+0000 7f3a1ffff640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 0x7f3a0c0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.591+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3a180d0620 con 0x7f3a20107740 2024-04-22T00:02:13.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.591+0000 7f3a1ffff640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 0x7f3a0c0a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a10006010 tx=0x7f3a10000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.601+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3a1809cc50 con 0x7f3a20107740 2024-04-22T00:02:13.707 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:13.708 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:13.708 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85288]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:13.708 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85288]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:13.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.810+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f3a00000c60 con 0x7f3a0c0a2ec0 2024-04-22T00:02:13.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.872+0000 7f3a1d7fa640 1 -- 172.21.15.38:0/2490159234 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f3a00000c60 con 0x7f3a0c0a2ec0 2024-04-22T00:02:13.866 INFO:teuthology.orchestra.run.smithi038.stdout:Retention added to path / 2024-04-22T00:02:13.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 msgr2=0x7f3a0c0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 0x7f3a0c0a5380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a10006010 tx=0x7f3a10000bc0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 msgr2=0x7f3a2019c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800eb10 tx=0x7f3a1800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 shutdown_connections 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a0c0a2ec0 0x7f3a0c0a5380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a20107740 0x7f3a2019c8a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a201020f0 0x7f3a20198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 --2- 172.21.15.38:0/2490159234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a20101730 0x7f3a20198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 >> 172.21.15.38:0/2490159234 conn(0x7f3a200fd200 msgr2=0x7f3a200fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 shutdown_connections 2024-04-22T00:02:13.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.874+0000 7f3a26a37640 1 -- 172.21.15.38:0/2490159234 wait complete. 2024-04-22T00:02:13.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T00:02:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 -- 172.21.15.38:0/826562615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc15810bdc0 msgr2=0x7fc15810e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/826562615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc15810bdc0 0x7fc15810e1f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc150009ca0 tx=0x7fc15002c180 comp rx=0 tx=0).stop 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 -- 172.21.15.38:0/826562615 shutdown_connections 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/826562615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc15810bdc0 0x7fc15810e1f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/826562615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc158103310 0x7fc15810b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/826562615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1581029f0 0x7fc158102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 -- 172.21.15.38:0/826562615 >> 172.21.15.38:0/826562615 conn(0x7fc1580769c0 msgr2=0x7fc158076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 -- 172.21.15.38:0/826562615 shutdown_connections 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.988+0000 7fc15e3f5640 1 -- 172.21.15.38:0/826562615 wait complete. 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 Processor -- start 2024-04-22T00:02:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 -- start start 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1581029f0 0x7fc158197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc15810bdc0 0x7fc15819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc158111b50 con 0x7fc158103310 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1581119d0 con 0x7fc15810bdc0 2024-04-22T00:02:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.989+0000 7fc15e3f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc158111cd0 con 0x7fc1581029f0 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50884/0 (socket says 172.21.15.38:50884) 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 -- 172.21.15.38:0/3719886099 learned_addr learned my addr 172.21.15.38:0/3719886099 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc15c992640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc15810bdc0 0x7fc15819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc157fff640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1581029f0 0x7fc158197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 -- 172.21.15.38:0/3719886099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1581029f0 msgr2=0x7fc158197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1581029f0 0x7fc158197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 -- 172.21.15.38:0/3719886099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc15810bdc0 msgr2=0x7fc15819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc15810bdc0 0x7fc15819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 -- 172.21.15.38:0/3719886099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15819cec0 con 0x7fc158103310 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1577fe640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c890 tx=0x7fc14800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc148004220 con 0x7fc158103310 2024-04-22T00:02:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1480043c0 con 0x7fc158103310 2024-04-22T00:02:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15819d1b0 con 0x7fc158103310 2024-04-22T00:02:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1480049e0 con 0x7fc158103310 2024-04-22T00:02:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.990+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1581a4a40 con 0x7fc158103310 2024-04-22T00:02:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.991+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15804f500 con 0x7fc158103310 2024-04-22T00:02:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.992+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fc1480079b0 con 0x7fc158103310 2024-04-22T00:02:13.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.993+0000 7fc1557fa640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 0x7fc1440a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:13.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.993+0000 7fc157fff640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 0x7fc1440a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:13.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.993+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc1480cfa00 con 0x7fc158103310 2024-04-22T00:02:13.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.994+0000 7fc157fff640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 0x7fc1440a5400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc140009aa0 tx=0x7fc140005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:13.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:13.998+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc148014030 con 0x7fc158103310 2024-04-22T00:02:14.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.204+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fc1580fdb40 con 0x7fc1440a2f40 2024-04-22T00:02:14.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.212+0000 7fc1557fa640 1 -- 172.21.15.38:0/3719886099 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fc1580fdb40 con 0x7fc1440a2f40 2024-04-22T00:02:14.206 INFO:teuthology.orchestra.run.smithi038.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T00:02:13", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-22T00:02:14.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.213+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 msgr2=0x7fc1440a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.213+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 0x7fc1440a5400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc140009aa0 tx=0x7fc140005f90 comp rx=0 tx=0).stop 2024-04-22T00:02:14.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.213+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 msgr2=0x7fc158198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.213+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c890 tx=0x7fc14800cd50 comp rx=0 tx=0).stop 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 shutdown_connections 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1440a2f40 0x7fc1440a5400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc15810bdc0 0x7fc15819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc158103310 0x7fc158198410 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 --2- 172.21.15.38:0/3719886099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1581029f0 0x7fc158197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 >> 172.21.15.38:0/3719886099 conn(0x7fc1580769c0 msgr2=0x7fc1580fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 shutdown_connections 2024-04-22T00:02:14.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.214+0000 7fc15e3f5640 1 -- 172.21.15.38:0/3719886099 wait complete. 2024-04-22T00:02:14.208 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85376]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:14.208 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:14.209 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85376]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:14.209 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:02:13 smithi038 sudo[85376]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 169 KiB/s wr, 40 op/s 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2702504568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:13 smithi038 ceph-mon[27972]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:14.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 169 KiB/s wr, 40 op/s 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2702504568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:02:14.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:13 smithi203 ceph-mon[31431]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:14.287 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 169 KiB/s wr, 40 op/s 2024-04-22T00:02:14.287 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2702504568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:02:14.288 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:02:14.288 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.288 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:02:14.288 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:14.288 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:13 smithi111 ceph-mon[31543]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:14.288 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64548]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:14.288 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64548]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:14.288 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64548]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:14.288 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64548]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 -- 172.21.15.38:0/3836966365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f681029f0 msgr2=0x7f1f68102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/3836966365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f681029f0 0x7f1f68102dd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009900 tx=0x7f1f5802bcd0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 -- 172.21.15.38:0/3836966365 shutdown_connections 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/3836966365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f6810bdc0 0x7f1f6810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/3836966365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 0x7f1f6810b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/3836966365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f681029f0 0x7f1f68102dd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 -- 172.21.15.38:0/3836966365 >> 172.21.15.38:0/3836966365 conn(0x7f1f680769c0 msgr2=0x7f1f68076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 -- 172.21.15.38:0/3836966365 shutdown_connections 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.329+0000 7f1f6d9de640 1 -- 172.21.15.38:0/3836966365 wait complete. 2024-04-22T00:02:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 Processor -- start 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 -- start start 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f681029f0 0x7f1f68100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 0x7f1f68100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68111b50 con 0x7f1f6810bdc0 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681119d0 con 0x7f1f68103310 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f6d9de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68111cd0 con 0x7f1f681029f0 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50906/0 (socket says 172.21.15.38:50906) 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 -- 172.21.15.38:0/403927090 learned_addr learned my addr 172.21.15.38:0/403927090 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f66ffd640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f681029f0 0x7f1f68100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f5ffff640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 0x7f1f68100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 -- 172.21.15.38:0/403927090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f681029f0 msgr2=0x7f1f68100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f681029f0 0x7f1f68100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 -- 172.21.15.38:0/403927090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 msgr2=0x7f1f68100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 0x7f1f68100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.330+0000 7f1f677fe640 1 -- 172.21.15.38:0/403927090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f681a6490 con 0x7f1f6810bdc0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f677fe640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000eb40 tx=0x7f1f6000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60004b20 con 0x7f1f6810bdc0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f60004cc0 con 0x7f1f6810bdc0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f681a6780 con 0x7f1f6810bdc0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60010850 con 0x7f1f6810bdc0 2024-04-22T00:02:14.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.331+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681a6cd0 con 0x7f1f6810bdc0 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.332+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6804f500 con 0x7f1f6810bdc0 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.333+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f1f60005020 con 0x7f1f6810bdc0 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.333+0000 7f1f6c9dc640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 0x7f1f540a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.334+0000 7f1f66ffd640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 0x7f1f540a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.334+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f1f600d0140 con 0x7f1f6810bdc0 2024-04-22T00:02:14.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.334+0000 7f1f66ffd640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 0x7f1f540a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1f5802efd0 tx=0x7f1f5802cf90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:14.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.338+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1f6009c7d0 con 0x7f1f6810bdc0 2024-04-22T00:02:14.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.544+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1f680fdb20 con 0x7f1f540a2ec0 2024-04-22T00:02:14.543 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:14.544 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:14.544 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64551]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:14.544 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:02:14 smithi111 sudo[64551]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:14.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.550+0000 7f1f6c9dc640 1 -- 172.21.15.38:0/403927090 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1f680fdb20 con 0x7f1f540a2ec0 2024-04-22T00:02:14.544 INFO:teuthology.orchestra.run.smithi038.stdout:/ 1m 6m3h 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 msgr2=0x7f1f540a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 0x7f1f540a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1f5802efd0 tx=0x7f1f5802cf90 comp rx=0 tx=0).stop 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 msgr2=0x7f1f681010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000eb40 tx=0x7f1f6000c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 shutdown_connections 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1f540a2ec0 0x7f1f540a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6810bdc0 0x7f1f681010c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f68103310 0x7f1f68100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 --2- 172.21.15.38:0/403927090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f681029f0 0x7f1f68100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:14.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 >> 172.21.15.38:0/403927090 conn(0x7f1f680769c0 msgr2=0x7f1f680fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:14.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 shutdown_connections 2024-04-22T00:02:14.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:14.552+0000 7f1f6d9de640 1 -- 172.21.15.38:0/403927090 wait complete. 2024-04-22T00:02:14.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-22T00:02:14.583 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-22T00:02:14.634 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T00:02:14.709 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:02:14.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T00:02:14.712 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T00:02:14.725 INFO:tasks.workunit:Pulling workunits from ref 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T00:02:14.725 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T00:02:14.725 INFO:tasks.workunit:timeout=3h 2024-04-22T00:02:14.726 INFO:tasks.workunit:cleanup=True 2024-04-22T00:02:14.726 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:14.741 INFO:teuthology.orchestra.run.smithi038.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:14.741 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-22 00:00:05.364561891 +0000 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-22 00:00:05.364561891 +0000 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-22 00:02:02.471175307 +0000 2024-04-22T00:02:14.742 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-22 00:00:05.364561891 +0000 2024-04-22T00:02:14.742 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T00:02:14.742 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T00:02:14.790 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65581]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:14.790 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65581]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:14.790 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65581]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:14.790 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65581]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:14.815 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T00:02:14.873 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T00:02:15.279 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:15.279 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:15.279 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65584]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:15.279 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:02:14 smithi203 sudo[65584]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:15.695 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85554]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:15.695 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:15.695 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85554]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:15.695 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85554]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:15.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85557]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 sudo[85557]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 ceph-mon[27972]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 ceph-mon[27972]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:15.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:15 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 ceph-mon[31431]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 ceph-mon[31431]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 sudo[65588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 sudo[65588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:15.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 sudo[65588]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:15.966 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:15 smithi203 sudo[65588]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:16.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:15 smithi111 ceph-mon[31543]: from='client.15777 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:16.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:15 smithi111 ceph-mon[31543]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:16.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:15 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:16.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 sudo[65591]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:16.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 sudo[65591]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:16.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 sudo[65591]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:16.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 sudo[65591]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:16.704 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T00:02:16.704 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:16.704 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64557]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:16.704 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64557]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:17.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:17.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-22T00:02:17.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.030 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:16 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64560]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T00:02:17.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64560]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T00:02:17.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64560]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 sudo[64560]: pam_unix(sudo:session): session closed for user root 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:16 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:17.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:17.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T00:02:17.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T00:02:17.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:16 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T00:02:18.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:17 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:18.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:17 smithi203 ceph-mon[31431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:18.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:17 smithi203 ceph-mon[31431]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 192 KiB/s wr, 65 op/s 2024-04-22T00:02:18.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:17 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:18.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:17 smithi111 ceph-mon[31543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:18.076 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:17 smithi111 ceph-mon[31543]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 192 KiB/s wr, 65 op/s 2024-04-22T00:02:18.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:17 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T00:02:18.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:17 smithi038 ceph-mon[27972]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T00:02:18.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:17 smithi038 ceph-mon[27972]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 192 KiB/s wr, 65 op/s 2024-04-22T00:02:20.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:19 smithi038 ceph-mon[27972]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:20.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:19 smithi203 ceph-mon[31431]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:20.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:19 smithi111 ceph-mon[31543]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:22.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:21 smithi038 ceph-mon[27972]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:22.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:21 smithi203 ceph-mon[31431]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:22.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:21 smithi111 ceph-mon[31543]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 176 KiB/s wr, 60 op/s 2024-04-22T00:02:24.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:23 smithi038 ceph-mon[27972]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 197 KiB/s wr, 66 op/s 2024-04-22T00:02:24.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:23 smithi203 ceph-mon[31431]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 197 KiB/s wr, 66 op/s 2024-04-22T00:02:24.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:23 smithi111 ceph-mon[31543]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 197 KiB/s wr, 66 op/s 2024-04-22T00:02:26.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:25 smithi038 ceph-mon[27972]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 99 KiB/s wr, 43 op/s 2024-04-22T00:02:26.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:25 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:02:26.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:25 smithi203 ceph-mon[31431]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 99 KiB/s wr, 43 op/s 2024-04-22T00:02:26.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:25 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:02:26.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:25 smithi111 ceph-mon[31543]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 99 KiB/s wr, 43 op/s 2024-04-22T00:02:26.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:25 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:02:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 -- 172.21.15.38:0/3517792638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230106630 msgr2=0x7f4230106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 --2- 172.21.15.38:0/3517792638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230106630 0x7f4230106a10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f422c009930 tx=0x7f422c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 -- 172.21.15.38:0/3517792638 shutdown_connections 2024-04-22T00:02:27.854 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 --2- 172.21.15.38:0/3517792638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230101030 0x7f423010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 --2- 172.21.15.38:0/3517792638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230100670 0x7f4230100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 --2- 172.21.15.38:0/3517792638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230106630 0x7f4230106a10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 -- 172.21.15.38:0/3517792638 >> 172.21.15.38:0/3517792638 conn(0x7f42300777c0 msgr2=0x7f4230077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 -- 172.21.15.38:0/3517792638 shutdown_connections 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.861+0000 7f4237f19640 1 -- 172.21.15.38:0/3517792638 wait complete. 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 Processor -- start 2024-04-22T00:02:27.855 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 -- start start 2024-04-22T00:02:27.857 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 0x7f42301973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230106630 0x7f423019b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4230110a00 con 0x7f4230100670 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4230110880 con 0x7f4230101030 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4237f19640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4230110b80 con 0x7f4230106630 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44142/0 (socket says 172.21.15.38:44142) 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 -- 172.21.15.38:0/3879979825 learned_addr learned my addr 172.21.15.38:0/3879979825 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f42364b6640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230106630 0x7f423019b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f42354b4640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 0x7f42301973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 -- 172.21.15.38:0/3879979825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230106630 msgr2=0x7f423019b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.858 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230106630 0x7f423019b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 -- 172.21.15.38:0/3879979825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 msgr2=0x7f42301973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.862+0000 7f4235cb5640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 0x7f42301973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4235cb5640 1 -- 172.21.15.38:0/3879979825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423019be00 con 0x7f4230100670 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f42354b4640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 0x7f42301973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4235cb5640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f422c009230 tx=0x7f422c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4226ffd640 1 -- 172.21.15.38:0/3879979825 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c004450 con 0x7f4230100670 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4226ffd640 1 -- 172.21.15.38:0/3879979825 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f422c0054e0 con 0x7f4230100670 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423019c090 con 0x7f4230100670 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4226ffd640 1 -- 172.21.15.38:0/3879979825 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c03b6a0 con 0x7f4230100670 2024-04-22T00:02:27.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.863+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4230068df0 con 0x7f4230100670 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f4226ffd640 1 -- 172.21.15.38:0/3879979825 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f422c003880 con 0x7f4230100670 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 Processor -- start 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 -- start start 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c346ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3470440 0x55c1c3478cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1c33d3b70 con 0x55c1c3479210 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c1c337c470 con 0x55c1c346fb20 2024-04-22T00:02:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f423abd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c1c33cfb30 con 0x55c1c3470440 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f4226ffd640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 0x7f421c0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.866+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44154/0 (socket says 172.21.15.38:44154) 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 -- 172.21.15.38:0/3811740155 learned_addr learned my addr 172.21.15.38:0/3811740155 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42364b6640 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c346ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f4235cb5640 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3470440 0x55c1c3478cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42364b6640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 0x7f421c0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f4226ffd640 1 -- 172.21.15.38:0/3879979825 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f422c0f3450 con 0x7f4230100670 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 -- 172.21.15.38:0/3811740155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3470440 msgr2=0x55c1c3478cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3470440 0x55c1c3478cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 -- 172.21.15.38:0/3811740155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 msgr2=0x55c1c346ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c346ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 -- 172.21.15.38:0/3811740155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1c30e8210 con 0x55c1c3479210 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42354b4640 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f422000bcb0 tx=0x7f422000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb903d29e701efe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f42364b6640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 0x7f421c0a5400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4228007fd0 tx=0x7f4228007520 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f421bfff640 1 -- 172.21.15.38:0/3811740155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220011440 con 0x55c1c3479210 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f421bfff640 1 -- 172.21.15.38:0/3811740155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4220011ac0 con 0x55c1c3479210 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.867+0000 7f421bfff640 1 -- 172.21.15.38:0/3811740155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422001bad0 con 0x55c1c3479210 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 -- 172.21.15.38:0/3811740155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 msgr2=0x55c1c347b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f422000bcb0 tx=0x7f422000f730 comp rx=0 tx=0).stop 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 -- 172.21.15.38:0/3811740155 shutdown_connections 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3479210 0x55c1c347b600 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3470440 0x55c1c3478cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 --2- 172.21.15.38:0/3811740155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c346ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 -- 172.21.15.38:0/3811740155 >> 172.21.15.38:0/3811740155 conn(0x55c1c346a440 msgr2=0x55c1c346a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 -- 172.21.15.38:0/3811740155 shutdown_connections 2024-04-22T00:02:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.868+0000 7f423abd6740 1 -- 172.21.15.38:0/3811740155 wait complete. 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 -- start start 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 Processor -- start 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c3476c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1c347be40 con 0x55c1c3470440 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c1c338fa10 con 0x55c1c346fb20 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c1c347bfc0 con 0x55c1c3479210 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f423abd6740 10 client.0 _ll_register_callbacks cb 0x55c1c33cab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44156/0 (socket says 172.21.15.38:44156) 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f42354b4640 1 -- 172.21.15.38:0/2037118026 learned_addr learned my addr 172.21.15.38:0/2037118026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f42364b6640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c3476c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.869+0000 7f4235cb5640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 -- 172.21.15.38:0/2037118026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 msgr2=0x55c1c3476c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c3476c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 -- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 msgr2=0x55c1c34761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 -- 172.21.15.38:0/2037118026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1c31b4620 con 0x55c1c3470440 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42354b4640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4220000c00 tx=0x7f422000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317b6ae1f55f4162 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42197fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220029060 con 0x55c1c3470440 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4220011650 con 0x55c1c3470440 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4220011ac0 con 0x55c1c3470440 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4220030050 con 0x55c1c3470440 2024-04-22T00:02:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f423abd6740 10 client.15804 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1c3488040 con 0x55c1c3470440 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.870+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4220011ac0 con 0x55c1c3470440 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f422001bcb0 con 0x55c1c3470440 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 10 client.15804 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 10 client.15804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 20 client.15804 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34893f0 0x55c1c348b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 4 client.15804 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1c33e10b0 con 0x55c1c34893f0 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f4235cb5640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34893f0 0x55c1c348b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f4235cb5640 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34893f0 0x55c1c348b7e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c17b37a9fca7d4 server_cookie=1a3496777c330193 in_seq=0 out_seq=0 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.871+0000 7f42197fa640 10 client.15804 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f42197fa640 1 -- 172.21.15.38:0/2037118026 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1c33e10b0 con 0x55c1c34893f0 2024-04-22T00:02:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f42197fa640 10 client.15804 handle_command_reply: tid=0 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 client.15804 shutdown 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 msgr2=0x55c1c3476720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4220000c00 tx=0x7f422000bfd0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 shutdown_connections 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c3476c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34893f0 0x55c1c348b7e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3476720 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 --2- 172.21.15.38:0/2037118026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f42197fa640 0 client.15804 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.872+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 >> 172.21.15.38:0/2037118026 conn(0x55c1c346a440 msgr2=0x55c1c346c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.873+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 shutdown_connections 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.873+0000 7f423abd6740 1 -- 172.21.15.38:0/2037118026 wait complete. 2024-04-22T00:02:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.873+0000 7f423abd6740 20 client.15804 trim_cache size 0 max 16384 2024-04-22T00:02:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 Processor -- start 2024-04-22T00:02:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 -- start start 2024-04-22T00:02:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 0x55c1c3478910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c3479210 0x55c1c34815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f42301ac7e0 con 0x55c1c3470440 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f42301ae320 con 0x55c1c3479210 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f423abd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42301aceb0 con 0x55c1c346fb20 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44172/0 (socket says 172.21.15.38:44172) 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 -- 172.21.15.38:0/2353218638 learned_addr learned my addr 172.21.15.38:0/2353218638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 -- 172.21.15.38:0/2353218638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 msgr2=0x55c1c3478910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f4235cb5640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 0x55c1c3478910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 0x55c1c3478910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42364b6640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c3479210 0x55c1c34815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 -- 172.21.15.38:0/2353218638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c3479210 msgr2=0x55c1c34815f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c3479210 0x55c1c34815f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f42354b4640 1 -- 172.21.15.38:0/2353218638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1c3474730 con 0x55c1c3470440 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.888+0000 7f4235cb5640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 0x55c1c3478910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f42354b4640 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f42200059f0 tx=0x7f422000ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdc7eafaa932ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f421b7fe640 1 -- 172.21.15.38:0/2353218638 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220029040 con 0x55c1c3470440 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f421b7fe640 1 -- 172.21.15.38:0/2353218638 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4220003b90 con 0x55c1c3470440 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f421b7fe640 1 -- 172.21.15.38:0/2353218638 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220005320 con 0x55c1c3470440 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f423abd6740 1 -- 172.21.15.38:0/2353218638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 msgr2=0x55c1c3484d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f423abd6740 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f42200059f0 tx=0x7f422000ff60 comp rx=0 tx=0).stop 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.889+0000 7f423abd6740 1 -- 172.21.15.38:0/2353218638 shutdown_connections 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c3479210 0x55c1c34815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c3484d00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- 172.21.15.38:0/2353218638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c346fb20 0x55c1c3478910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- 172.21.15.38:0/2353218638 >> 172.21.15.38:0/2353218638 conn(0x55c1c346a440 msgr2=0x55c1c3471590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- 172.21.15.38:0/2353218638 shutdown_connections 2024-04-22T00:02:27.885 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- 172.21.15.38:0/2353218638 wait complete. 2024-04-22T00:02:27.885 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- start start 2024-04-22T00:02:27.885 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 Processor -- start 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34e75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c34e8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f42301adc50 con 0x55c1c3470440 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f42301ad580 con 0x55c1c346fb20 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.890+0000 7f423abd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42301aa7c0 con 0x55c1c3479210 2024-04-22T00:02:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f423abd6740 10 client.0 _ll_register_callbacks cb 0x7f4220015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44176/0 (socket says 172.21.15.38:44176) 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 -- 172.21.15.38:0/1920991341 learned_addr learned my addr 172.21.15.38:0/1920991341 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42364b6640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c34e8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f4235cb5640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34e75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 -- 172.21.15.38:0/1920991341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 msgr2=0x55c1c34e8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c34e8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 -- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 msgr2=0x55c1c34e75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34e75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 -- 172.21.15.38:0/1920991341 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1c34f27e0 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42354b4640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4220000c00 tx=0x7f4220003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82e58e8b4e85c97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42197fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422000b7c0 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4220011860 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4220003b90 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.891+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4220045050 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 10 client.15810 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1c34f3a60 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4220003b90 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f422001b9c0 con 0x55c1c3470440 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 10 client.15810 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 20 client.15810 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34f40f0 0x55c1c34f6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 4 client.15810 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1c33db400 con 0x55c1c34f40f0 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.892+0000 7f4235cb5640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34f40f0 0x55c1c34f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.893+0000 7f42197fa640 10 client.15810 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.893+0000 7f4235cb5640 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34f40f0 0x55c1c34f6510 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68e2bd07d8472e7f server_cookie=af41371f08118ceb in_seq=0 out_seq=0 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.905+0000 7f42197fa640 1 -- 172.21.15.38:0/1920991341 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1c33db400 con 0x55c1c34f40f0 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.905+0000 7f42197fa640 10 client.15810 handle_command_reply: tid=0 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.905+0000 7f423abd6740 1 client.15810 shutdown 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 msgr2=0x55c1c34e7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4220000c00 tx=0x7f4220003ad0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 shutdown_connections 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1c3479210 0x55c1c34e8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55c1c34f40f0 0x55c1c34f6510 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1c3470440 0x55c1c34e7b20 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 --2- 172.21.15.38:0/1920991341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1c346fb20 0x55c1c34e75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f42197fa640 0 client.15810 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 >> 172.21.15.38:0/1920991341 conn(0x55c1c346a440 msgr2=0x55c1c3476180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 shutdown_connections 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 1 -- 172.21.15.38:0/1920991341 wait complete. 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.906+0000 7f423abd6740 20 client.15810 trim_cache size 0 max 16384 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:02:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 msgr2=0x7f421c0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 0x7f421c0a5400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4228007fd0 tx=0x7f4228007520 comp rx=0 tx=0).stop 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 msgr2=0x7f4230196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f422c009230 tx=0x7f422c02cc70 comp rx=0 tx=0).stop 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 shutdown_connections 2024-04-22T00:02:27.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f421c0a2f40 0x7f421c0a5400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4230106630 0x7f423019b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4230101030 0x7f42301973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 --2- 172.21.15.38:0/3879979825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4230100670 0x7f4230196e60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 >> 172.21.15.38:0/3879979825 conn(0x7f42300777c0 msgr2=0x7f4230103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 shutdown_connections 2024-04-22T00:02:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:27.907+0000 7f4237f19640 1 -- 172.21.15.38:0/3879979825 wait complete. 2024-04-22T00:02:28.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:27 smithi038 ceph-mon[27972]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 99 KiB/s wr, 47 op/s 2024-04-22T00:02:28.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:27 smithi203 ceph-mon[31431]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 99 KiB/s wr, 47 op/s 2024-04-22T00:02:28.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:27 smithi111 ceph-mon[31543]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 99 KiB/s wr, 47 op/s 2024-04-22T00:02:30.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:29 smithi038 ceph-mon[27972]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-22T00:02:30.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:29 smithi203 ceph-mon[31431]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-22T00:02:30.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:29 smithi111 ceph-mon[31543]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-22T00:02:32.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:31 smithi038 ceph-mon[27972]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-22T00:02:32.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:31 smithi203 ceph-mon[31431]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-22T00:02:32.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:31 smithi111 ceph-mon[31543]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-22T00:02:34.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:33 smithi038 ceph-mon[27972]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-22T00:02:34.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:33 smithi203 ceph-mon[31431]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-22T00:02:34.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:33 smithi111 ceph-mon[31543]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-22T00:02:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:35 smithi038 ceph-mon[27972]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:36.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:35 smithi203 ceph-mon[31431]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:36.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:35 smithi111 ceph-mon[31543]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:38.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:37 smithi038 ceph-mon[27972]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-22T00:02:38.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:37 smithi203 ceph-mon[31431]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-22T00:02:38.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:37 smithi111 ceph-mon[31543]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-22T00:02:40.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:39 smithi038 ceph-mon[27972]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:40.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:39 smithi203 ceph-mon[31431]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:40.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:39 smithi111 ceph-mon[31543]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 -- 172.21.15.38:0/558168522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 msgr2=0x7f08b010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 --2- 172.21.15.38:0/558168522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b010d4f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009cd0 tx=0x7f08a802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 -- 172.21.15.38:0/558168522 shutdown_connections 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 --2- 172.21.15.38:0/558168522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b010d4f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 --2- 172.21.15.38:0/558168522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0100670 0x7f08b0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 --2- 172.21.15.38:0/558168522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0106630 0x7f08b0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.008+0000 7f08b72b5640 1 -- 172.21.15.38:0/558168522 >> 172.21.15.38:0/558168522 conn(0x7f08b00777c0 msgr2=0x7f08b0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- 172.21.15.38:0/558168522 shutdown_connections 2024-04-22T00:02:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- 172.21.15.38:0/558168522 wait complete. 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 Processor -- start 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- start start 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0100670 0x7f08b0198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0106630 0x7f08b01a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0110a00 con 0x7f08b0101030 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0110880 con 0x7f08b0100670 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.009+0000 7f08b72b5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08b0110b80 con 0x7f08b0106630 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54270/0 (socket says 172.21.15.38:54270) 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 -- 172.21.15.38:0/2748285524 learned_addr learned my addr 172.21.15.38:0/2748285524 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b5051640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0100670 0x7f08b0198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b5852640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0106630 0x7f08b01a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 -- 172.21.15.38:0/2748285524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0106630 msgr2=0x7f08b01a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0106630 0x7f08b01a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 -- 172.21.15.38:0/2748285524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0100670 msgr2=0x7f08b0198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0100670 0x7f08b0198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 -- 172.21.15.38:0/2748285524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b01a42c0 con 0x7f08b0101030 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b4850640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c890 tx=0x7f08a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08a67fc640 1 -- 172.21.15.38:0/2748285524 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0004220 con 0x7f08b0101030 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08a67fc640 1 -- 172.21.15.38:0/2748285524 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a00043c0 con 0x7f08b0101030 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b01a4490 con 0x7f08b0101030 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.010+0000 7f08a67fc640 1 -- 172.21.15.38:0/2748285524 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0004a00 con 0x7f08b0101030 2024-04-22T00:02:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.011+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b01a4980 con 0x7f08b0101030 2024-04-22T00:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.013+0000 7f08a67fc640 1 -- 172.21.15.38:0/2748285524 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f08a00079b0 con 0x7f08b0101030 2024-04-22T00:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.013+0000 7f08b9f78740 1 Processor -- start 2024-04-22T00:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.013+0000 7f08b9f78740 1 -- start start 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6298cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62998600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55db628f0b70 con 0x55db6298cb20 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55db62899470 con 0x55db6298d440 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b9f78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55db628ecb30 con 0x55db62996210 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08a67fc640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 0x7f089c0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5852640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6298cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33240/0 (socket says 172.21.15.38:33240) 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 -- 172.21.15.38:0/805668693 learned_addr learned my addr 172.21.15.38:0/805668693 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b4850640 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62998600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5852640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 0x7f089c0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08a67fc640 1 -- 172.21.15.38:0/2748285524 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08a00cf680 con 0x7f08b0101030 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 -- 172.21.15.38:0/805668693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 msgr2=0x55db62998600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62998600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 -- 172.21.15.38:0/805668693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 msgr2=0x55db6298cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6298cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 -- 172.21.15.38:0/805668693 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db62605210 con 0x55db6298d440 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b4850640 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62998600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.014+0000 7f08b5051640 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00ac90 tx=0x7f08ac00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c4ba39602cbe52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b5852640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 0x7f089c0a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08a8006010 tx=0x7f08a802c0f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f089b7fe640 1 -- 172.21.15.38:0/805668693 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac01d070 con 0x55db6298d440 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f089b7fe640 1 -- 172.21.15.38:0/805668693 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac018b20 con 0x55db6298d440 2024-04-22T00:02:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f089b7fe640 1 -- 172.21.15.38:0/805668693 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac010ba0 con 0x55db6298d440 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 -- 172.21.15.38:0/805668693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 msgr2=0x55db62995cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00ac90 tx=0x7f08ac00f790 comp rx=0 tx=0).stop 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 -- 172.21.15.38:0/805668693 shutdown_connections 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62998600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db62995cd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 --2- 172.21.15.38:0/805668693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6298cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.015+0000 7f08b9f78740 1 -- 172.21.15.38:0/805668693 >> 172.21.15.38:0/805668693 conn(0x55db62987440 msgr2=0x55db62987820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- 172.21.15.38:0/805668693 shutdown_connections 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- 172.21.15.38:0/805668693 wait complete. 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- start start 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 Processor -- start 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298cb20 0x55db62993250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62993cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55db62998e40 con 0x55db6298d440 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55db628aca10 con 0x55db6298cb20 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.016+0000 7f08b9f78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55db62998fc0 con 0x55db62996210 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b9f78740 10 client.0 _ll_register_callbacks cb 0x55db628e7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54284/0 (socket says 172.21.15.38:54284) 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 -- 172.21.15.38:0/3915043904 learned_addr learned my addr 172.21.15.38:0/3915043904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b5051640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298cb20 0x55db62993250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 -- 172.21.15.38:0/3915043904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 msgr2=0x55db62993cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62993cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 -- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298cb20 msgr2=0x55db62993250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298cb20 0x55db62993250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 -- 172.21.15.38:0/3915043904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db626d1620 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b4850640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f08a009b5a0 tx=0x7f08a006a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920ccd7944daf2fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f0898ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a001d070 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08a0007ca0 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08ac018b20 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08a009b150 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b9f78740 10 client.15822 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.017+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db629a5120 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac018b20 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08a00cf2c0 con 0x55db6298d440 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 10 client.15822 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 20 client.15822 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db629a65e0 0x55db629a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 4 client.15822 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db628fe0b0 con 0x55db629a65e0 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.018+0000 7f08b5051640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db629a65e0 0x55db629a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.019+0000 7f08b5051640 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db629a65e0 0x55db629a89f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36491e391b9aa275 server_cookie=adc4a5e65bbfa790 in_seq=0 out_seq=0 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.019+0000 7f0898ff9640 10 client.15822 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.019+0000 7f0898ff9640 1 -- 172.21.15.38:0/3915043904 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db628fe0b0 con 0x55db629a65e0 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.019+0000 7f0898ff9640 10 client.15822 handle_command_reply: tid=0 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.019+0000 7f08b9f78740 1 client.15822 shutdown 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 msgr2=0x55db62993790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f08a009b5a0 tx=0x7f08a006a540 comp rx=0 tx=0).stop 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 shutdown_connections 2024-04-22T00:02:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db62993cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db629a65e0 0x55db629a89f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298d440 0x55db62993790 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 --2- 172.21.15.38:0/3915043904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298cb20 0x55db62993250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f0898ff9640 0 client.15822 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 >> 172.21.15.38:0/3915043904 conn(0x55db62987440 msgr2=0x55db62989190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 shutdown_connections 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 1 -- 172.21.15.38:0/3915043904 wait complete. 2024-04-22T00:02:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.020+0000 7f08b9f78740 20 client.15822 trim_cache size 0 max 16384 2024-04-22T00:02:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 Processor -- start 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 -- start start 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db629a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db629ae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01aea00 con 0x55db6298cb20 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01b0540 con 0x55db6298d440 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.035+0000 7f08b9f78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01af0d0 con 0x55db62996210 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54298/0 (socket says 172.21.15.38:54298) 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 -- 172.21.15.38:0/334021545 learned_addr learned my addr 172.21.15.38:0/334021545 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b4850640 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db629a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 -- 172.21.15.38:0/334021545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 msgr2=0x55db629ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db629ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 -- 172.21.15.38:0/334021545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 msgr2=0x55db629a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db629a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 -- 172.21.15.38:0/334021545 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6298c3c0 con 0x55db6298cb20 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f08b5051640 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0107d0 tx=0x7f08ac010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861be81f0bdb4b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f089affd640 1 -- 172.21.15.38:0/334021545 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac02f070 con 0x55db6298cb20 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f089affd640 1 -- 172.21.15.38:0/334021545 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac005d60 con 0x55db6298cb20 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.036+0000 7f089affd640 1 -- 172.21.15.38:0/334021545 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac01d040 con 0x55db6298cb20 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- 172.21.15.38:0/334021545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 msgr2=0x55db62995980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f08ac0107d0 tx=0x7f08ac010760 comp rx=0 tx=0).stop 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- 172.21.15.38:0/334021545 shutdown_connections 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db62996210 0x55db629ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db6298d440 0x55db629a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 --2- 172.21.15.38:0/334021545 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db62995980 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- 172.21.15.38:0/334021545 >> 172.21.15.38:0/334021545 conn(0x55db62987440 msgr2=0x55db62991b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- 172.21.15.38:0/334021545 shutdown_connections 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- 172.21.15.38:0/334021545 wait complete. 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 -- start start 2024-04-22T00:02:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.037+0000 7f08b9f78740 1 Processor -- start 2024-04-22T00:02:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 0x55db6298fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db62996210 0x55db629900c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01afe70 con 0x55db6298cb20 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01af7a0 con 0x55db62996210 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08b01ac9e0 con 0x55db6298d440 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 10 client.0 _ll_register_callbacks cb 0x7f08a00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54300/0 (socket says 172.21.15.38:54300) 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 -- 172.21.15.38:0/2951077337 learned_addr learned my addr 172.21.15.38:0/2951077337 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5852640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db62996210 0x55db629900c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b4850640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 0x55db6298fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 -- 172.21.15.38:0/2951077337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 msgr2=0x55db6298fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 0x55db6298fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 -- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db62996210 msgr2=0x55db629900c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db62996210 0x55db629900c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 -- 172.21.15.38:0/2951077337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db629908f0 con 0x55db6298cb20 2024-04-22T00:02:41.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b4850640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 0x55db6298fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b5051640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00afd0 tx=0x7f08ac00a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda7053c89f92e7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f0898ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac02f070 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08ac010800 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.038+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08ac005d60 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08ac03c050 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 10 client.15828 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db62a0cb10 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ac005d60 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08ac02a040 con 0x55db6298cb20 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 10 client.15828 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 20 client.15828 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db62a0cf90 0x55db62a0f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 4 client.15828 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db628f8400 con 0x55db62a0cf90 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.039+0000 7f08b4850640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db62a0cf90 0x55db62a0f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.040+0000 7f0898ff9640 10 client.15828 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.040+0000 7f08b4850640 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db62a0cf90 0x55db62a0f380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a0589775b084d3 server_cookie=791888000eb11475 in_seq=0 out_seq=0 2024-04-22T00:02:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f0898ff9640 1 -- 172.21.15.38:0/2951077337 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db628f8400 con 0x55db62a0cf90 2024-04-22T00:02:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f0898ff9640 10 client.15828 handle_command_reply: tid=0 2024-04-22T00:02:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 client.15828 shutdown 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 msgr2=0x55db6299e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00afd0 tx=0x7f08ac00a6b0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 shutdown_connections 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db62996210 0x55db629900c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db6298d440 0x55db6298fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db62a0cf90 0x55db62a0f380 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f08b9f78740 1 --2- 172.21.15.38:0/2951077337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db6298cb20 0x55db6299e590 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.043+0000 7f0898ff9640 0 client.15828 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 >> 172.21.15.38:0/2951077337 conn(0x55db62987440 msgr2=0x55db62993210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 shutdown_connections 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b9f78740 1 -- 172.21.15.38:0/2951077337 wait complete. 2024-04-22T00:02:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b9f78740 20 client.15828 trim_cache size 0 max 16384 2024-04-22T00:02:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:02:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:02:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:02:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 msgr2=0x7f089c0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 0x7f089c0a5380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08a8006010 tx=0x7f08a802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 msgr2=0x7f08b0199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f08a000c890 tx=0x7f08a000cc90 comp rx=0 tx=0).stop 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 shutdown_connections 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f089c0a2ec0 0x7f089c0a5380 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08b0106630 0x7f08b01a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08b0101030 0x7f08b0199240 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 --2- 172.21.15.38:0/2748285524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08b0100670 0x7f08b0198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 >> 172.21.15.38:0/2748285524 conn(0x7f08b00777c0 msgr2=0x7f08b0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 shutdown_connections 2024-04-22T00:02:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:41.044+0000 7f08b72b5640 1 -- 172.21.15.38:0/2748285524 wait complete. 2024-04-22T00:02:42.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:41 smithi038 ceph-mon[27972]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:42.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:41 smithi203 ceph-mon[31431]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:42.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:41 smithi111 ceph-mon[31543]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:02:43.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:02:43.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 -- 172.21.15.38:0/2918831717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0106630 msgr2=0x7feec0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 --2- 172.21.15.38:0/2918831717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0106630 0x7feec0106a10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009930 tx=0x7feeb002bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 -- 172.21.15.38:0/2918831717 shutdown_connections 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 --2- 172.21.15.38:0/2918831717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 0x7feec010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 --2- 172.21.15.38:0/2918831717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0100620 0x7feec0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 --2- 172.21.15.38:0/2918831717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0106630 0x7feec0106a10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 -- 172.21.15.38:0/2918831717 >> 172.21.15.38:0/2918831717 conn(0x7feec00776a0 msgr2=0x7feec0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 -- 172.21.15.38:0/2918831717 shutdown_connections 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.479+0000 7feec6e9d640 1 -- 172.21.15.38:0/2918831717 wait complete. 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 Processor -- start 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 -- start start 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 0x7feec0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0106630 0x7feec019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feec0110a90 con 0x7feec0100620 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feec0110910 con 0x7feec0106630 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec6e9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feec0110c10 con 0x7feec0100fe0 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54318/0 (socket says 172.21.15.38:54318) 2024-04-22T00:02:43.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 -- 172.21.15.38:0/860741137 learned_addr learned my addr 172.21.15.38:0/860741137 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feebffff640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 0x7feec0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec543a640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0106630 0x7feec019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 -- 172.21.15.38:0/860741137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 msgr2=0x7feec0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 0x7feec0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 -- 172.21.15.38:0/860741137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0106630 msgr2=0x7feec019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.480+0000 7feec4c39640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0106630 0x7feec019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feec4c39640 1 -- 172.21.15.38:0/860741137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec019d030 con 0x7feec0100620 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feec4c39640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009900 tx=0x7feeb0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feebdffb640 1 -- 172.21.15.38:0/860741137 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0043020 con 0x7feec0100620 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec019d2c0 con 0x7feec0100620 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feebdffb640 1 -- 172.21.15.38:0/860741137 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb00054e0 con 0x7feec0100620 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feebdffb640 1 -- 172.21.15.38:0/860741137 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0003b80 con 0x7feec0100620 2024-04-22T00:02:43.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.481+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec01a4bb0 con 0x7feec0100620 2024-04-22T00:02:43.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.483+0000 7feebdffb640 1 -- 172.21.15.38:0/860741137 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7feeb0046020 con 0x7feec0100620 2024-04-22T00:02:43.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 Processor -- start 2024-04-22T00:02:43.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 -- start start 2024-04-22T00:02:43.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb21720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb2a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a9ba81560 con 0x556a9bb2a840 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556a9ba29730 con 0x556a9bb21340 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec9b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556a9ba7d520 con 0x556a9bb21c60 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feebffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feebdffb640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 0x7feea80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feebffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54326/0 (socket says 172.21.15.38:54326) 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feebffff640 1 -- 172.21.15.38:0/3101079954 learned_addr learned my addr 172.21.15.38:0/3101079954 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec4c39640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb2a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec543a640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb21720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.484+0000 7feec543a640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 0x7feea80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 -- 172.21.15.38:0/3101079954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 msgr2=0x556a9bb2a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb2a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 -- 172.21.15.38:0/3101079954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 msgr2=0x556a9bb21720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb21720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebdffb640 1 -- 172.21.15.38:0/860741137 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7feeb00f1a60 con 0x7feec0100620 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 -- 172.21.15.38:0/3101079954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9b85f7d0 con 0x556a9bb2a840 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feec543a640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb21720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feebffff640 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009c90 tx=0x7feeb4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188969d568e13309 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feec543a640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 0x7feea80a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feeb8013540 tx=0x7feeb8005c20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feea6ffd640 1 -- 172.21.15.38:0/3101079954 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb401e070 con 0x556a9bb2a840 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feea6ffd640 1 -- 172.21.15.38:0/3101079954 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb4019b10 con 0x556a9bb2a840 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.485+0000 7feea6ffd640 1 -- 172.21.15.38:0/3101079954 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400ea70 con 0x556a9bb2a840 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- 172.21.15.38:0/3101079954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 msgr2=0x556a9bb2cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7feeb4009c90 tx=0x7feeb4011810 comp rx=0 tx=0).stop 2024-04-22T00:02:43.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- 172.21.15.38:0/3101079954 shutdown_connections 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb2cc30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb2a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 --2- 172.21.15.38:0/3101079954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb21720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- 172.21.15.38:0/3101079954 >> 172.21.15.38:0/3101079954 conn(0x556a9bb1ba70 msgr2=0x556a9bb1be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- 172.21.15.38:0/3101079954 shutdown_connections 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- 172.21.15.38:0/3101079954 wait complete. 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 -- start start 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.486+0000 7feec9b67740 1 Processor -- start 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb278b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb27df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556a9bb2d630 con 0x556a9bb2a840 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556a9ba3ce00 con 0x556a9bb21340 2024-04-22T00:02:43.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556a9bb2d7b0 con 0x556a9bb21c60 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec9b67740 10 client.0 _ll_register_callbacks cb 0x556a9ba77eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54336/0 (socket says 172.21.15.38:54336) 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 -- 172.21.15.38:0/767652699 learned_addr learned my addr 172.21.15.38:0/767652699 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec4c39640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb278b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 -- 172.21.15.38:0/767652699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 msgr2=0x556a9bb27df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb27df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 -- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 msgr2=0x556a9bb278b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb278b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 -- 172.21.15.38:0/767652699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9b7953f0 con 0x556a9bb2a840 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec4c39640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb278b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7feec543a640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7feeb800feb0 tx=0x7feeb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885fa12c5b207225 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.487+0000 7fee9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8024020 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7feeb80203f0 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb4019b10 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feeb801c040 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 10 client.15840 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a9b7550f0 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb4019b10 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feeb801f030 con 0x556a9bb2a840 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 10 client.15840 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 20 client.15840 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bb3acf0 0x556a9bb3d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 4 client.15840 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:43.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.488+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a9ba8eaa0 con 0x556a9bb3acf0 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.489+0000 7feec4c39640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bb3acf0 0x556a9bb3d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.489+0000 7feec4c39640 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bb3acf0 0x556a9bb3d190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce45be145070d380 server_cookie=44e30d929995311b in_seq=0 out_seq=0 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.489+0000 7fee9ffff640 10 client.15840 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7fee9ffff640 1 -- 172.21.15.38:0/767652699 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a9ba8eaa0 con 0x556a9bb3acf0 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7fee9ffff640 10 client.15840 handle_command_reply: tid=0 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 client.15840 shutdown 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 msgr2=0x556a9bb28330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7feeb800feb0 tx=0x7feeb8004a40 comp rx=0 tx=0).stop 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 shutdown_connections 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bb3acf0 0x556a9bb3d190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb2a840 0x556a9bb28330 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb27df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7fee9ffff640 0 client.15840 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 --2- 172.21.15.38:0/767652699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb21340 0x556a9bb278b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 >> 172.21.15.38:0/767652699 conn(0x556a9bb1ba70 msgr2=0x556a9bb1d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 shutdown_connections 2024-04-22T00:02:43.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 1 -- 172.21.15.38:0/767652699 wait complete. 2024-04-22T00:02:43.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.490+0000 7feec9b67740 20 client.15840 trim_cache size 0 max 16384 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 Processor -- start 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 -- start start 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb364f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ac950 con 0x556a9bb21340 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ae490 con 0x556a9bb2a840 2024-04-22T00:02:43.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec9b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ad020 con 0x556a9bb21c60 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54340/0 (socket says 172.21.15.38:54340) 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 -- 172.21.15.38:0/442760752 learned_addr learned my addr 172.21.15.38:0/442760752 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec543a640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feebffff640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 -- 172.21.15.38:0/442760752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 msgr2=0x556a9bb364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 -- 172.21.15.38:0/442760752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 msgr2=0x556a9bb296e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb296e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec4c39640 1 -- 172.21.15.38:0/442760752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a9bb26860 con 0x556a9bb21340 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feec543a640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb296e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.508+0000 7feebffff640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:43.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec4c39640 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7feeb00bda90 tx=0x7feeb0004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62188504e766760 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feea67fc640 1 -- 172.21.15.38:0/442760752 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb00be200 con 0x556a9bb21340 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feea67fc640 1 -- 172.21.15.38:0/442760752 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb00bdb00 con 0x556a9bb21340 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feea67fc640 1 -- 172.21.15.38:0/442760752 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb00c62e0 con 0x556a9bb21340 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 -- 172.21.15.38:0/442760752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 msgr2=0x556a9bb12440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7feeb00bda90 tx=0x7feeb0004ee0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 -- 172.21.15.38:0/442760752 shutdown_connections 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb296e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21c60 0x556a9bb364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 --2- 172.21.15.38:0/442760752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21340 0x556a9bb12440 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.509+0000 7feec9b67740 1 -- 172.21.15.38:0/442760752 >> 172.21.15.38:0/442760752 conn(0x556a9bb1ba70 msgr2=0x556a9bb28fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- 172.21.15.38:0/442760752 shutdown_connections 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- 172.21.15.38:0/442760752 wait complete. 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- start start 2024-04-22T00:02:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 Processor -- start 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 0x556a9bb2e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb2edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feec01addc0 con 0x556a9bb21c60 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feec01ad6f0 con 0x556a9bb2a840 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feec01aa930 con 0x556a9bb21340 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feec9b67740 10 client.0 _ll_register_callbacks cb 0x7feeb800d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feebffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feebffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54354/0 (socket says 172.21.15.38:54354) 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.510+0000 7feebffff640 1 -- 172.21.15.38:0/3180361345 learned_addr learned my addr 172.21.15.38:0/3180361345 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec4c39640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 0x556a9bb2e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec543a640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb2edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 -- 172.21.15.38:0/3180361345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 msgr2=0x556a9bb2e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 0x556a9bb2e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 -- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 msgr2=0x556a9bb2edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb2edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 -- 172.21.15.38:0/3180361345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a9bba02d0 con 0x556a9bb21c60 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec4c39640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 0x556a9bb2e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feebffff640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feeb401c040 tx=0x7feeb40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57de4ab5267dd9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7fee9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4027020 con 0x556a9bb21c60 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb00bdb00 con 0x556a9bb21c60 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7feeb40037f0 con 0x556a9bb21c60 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feeb4032050 con 0x556a9bb21c60 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec9b67740 10 client.15846 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:43.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.511+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a9bba1460 con 0x556a9bb21c60 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb00bdb00 con 0x556a9bb21c60 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7feeb4019a40 con 0x556a9bb21c60 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 10 client.15846 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 20 client.15846 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bba19e0 0x556a9bba3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 4 client.15846 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a9ba88df0 con 0x556a9bba19e0 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec4c39640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bba19e0 0x556a9bba3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7fee9ffff640 10 client.15846 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.512+0000 7feec4c39640 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bba19e0 0x556a9bba3dd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197aaf762540b78b server_cookie=7a21edca2a7f218f in_seq=0 out_seq=0 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7fee9ffff640 1 -- 172.21.15.38:0/3180361345 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556a9ba88df0 con 0x556a9bba19e0 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7fee9ffff640 10 client.15846 handle_command_reply: tid=0 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 client.15846 shutdown 2024-04-22T00:02:43.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 msgr2=0x556a9bb2e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7feeb401c040 tx=0x7feeb40056a0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 shutdown_connections 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556a9bb2a840 0x556a9bb2edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556a9bba19e0 0x556a9bba3dd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556a9bb21c60 0x556a9bb2e880 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 --2- 172.21.15.38:0/3180361345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556a9bb21340 0x556a9bb2e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7fee9ffff640 0 client.15846 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 >> 172.21.15.38:0/3180361345 conn(0x556a9bb1ba70 msgr2=0x556a9bb166e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 shutdown_connections 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 1 -- 172.21.15.38:0/3180361345 wait complete. 2024-04-22T00:02:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.513+0000 7feec9b67740 20 client.15846 trim_cache size 0 max 16384 2024-04-22T00:02:43.513 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:43.513 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:02:43.513 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:02:43.513 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 msgr2=0x7feea80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 0x7feea80a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feeb8013540 tx=0x7feeb8005c20 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 msgr2=0x7feec0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009900 tx=0x7feeb0004360 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 shutdown_connections 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feea80a2e20 0x7feea80a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feec0106630 0x7feec019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feec0100fe0 0x7feec0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 --2- 172.21.15.38:0/860741137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feec0100620 0x7feec0198040 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 >> 172.21.15.38:0/860741137 conn(0x7feec00776a0 msgr2=0x7feec0069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 shutdown_connections 2024-04-22T00:02:43.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.514+0000 7feec6e9d640 1 -- 172.21.15.38:0/860741137 wait complete. 2024-04-22T00:02:43.517 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:02:43.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:02:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 -- 172.21.15.38:0/3286095728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48103340 msgr2=0x7ffa4810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/3286095728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48103340 0x7ffa4810b820 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009900 tx=0x7ffa3c02bcd0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 -- 172.21.15.38:0/3286095728 shutdown_connections 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/3286095728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 0x7ffa4810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/3286095728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48103340 0x7ffa4810b820 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/3286095728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102a20 0x7ffa48102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 -- 172.21.15.38:0/3286095728 >> 172.21.15.38:0/3286095728 conn(0x7ffa480777a0 msgr2=0x7ffa48077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 -- 172.21.15.38:0/3286095728 shutdown_connections 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.627+0000 7ffa4dcec640 1 -- 172.21.15.38:0/3286095728 wait complete. 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 Processor -- start 2024-04-22T00:02:43.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 -- start start 2024-04-22T00:02:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 0x7ffa48198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 0x7ffa4819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48111b30 con 0x7ffa48102a20 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481119b0 con 0x7ffa4810bd60 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa4dcec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48111cb0 con 0x7ffa48103340 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa477fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.628+0000 7ffa477fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54376/0 (socket says 172.21.15.38:54376) 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 -- 172.21.15.38:0/4052264054 learned_addr learned my addr 172.21.15.38:0/4052264054 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa47fff640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 0x7ffa4819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa46ffd640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 0x7ffa48198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 -- 172.21.15.38:0/4052264054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 msgr2=0x7ffa48198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 0x7ffa48198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 -- 172.21.15.38:0/4052264054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 msgr2=0x7ffa4819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 0x7ffa4819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 -- 172.21.15.38:0/4052264054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4819cff0 con 0x7ffa48102a20 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa46ffd640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 0x7ffa48198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:43.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa477fe640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800d770 tx=0x7ffa3800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38017070 con 0x7ffa48102a20 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4819d2e0 con 0x7ffa48102a20 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa38005500 con 0x7ffa48102a20 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38004e00 con 0x7ffa48102a20 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.629+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a4b70 con 0x7ffa48102a20 2024-04-22T00:02:43.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.630+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa480689b0 con 0x7ffa48102a20 2024-04-22T00:02:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.631+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7ffa38004fa0 con 0x7ffa48102a20 2024-04-22T00:02:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.632+0000 7ffa44ff9640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 0x7ffa300a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.632+0000 7ffa46ffd640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 0x7ffa300a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.632+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ffa380cf520 con 0x7ffa48102a20 2024-04-22T00:02:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.633+0000 7ffa46ffd640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 0x7ffa300a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa4810da80 tx=0x7ffa3c005a20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.637+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ffa3809bb50 con 0x7ffa48102a20 2024-04-22T00:02:43.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.860+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa48106ee0 con 0x7ffa48102a20 2024-04-22T00:02:43.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.861+0000 7ffa44ff9640 1 -- 172.21.15.38:0/4052264054 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7ffa3809b450 con 0x7ffa48102a20 2024-04-22T00:02:43.854 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:43.854 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:02:03.216530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:02:43.854 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 msgr2=0x7ffa300a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 0x7ffa300a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa4810da80 tx=0x7ffa3c005a20 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 msgr2=0x7ffa48198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800d770 tx=0x7ffa3800dc30 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 shutdown_connections 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa300a2e20 0x7ffa300a52e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4810bd60 0x7ffa4819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48103340 0x7ffa48198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 --2- 172.21.15.38:0/4052264054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa48102a20 0x7ffa48198050 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 >> 172.21.15.38:0/4052264054 conn(0x7ffa480777a0 msgr2=0x7ffa480fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 shutdown_connections 2024-04-22T00:02:43.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.863+0000 7ffa4dcec640 1 -- 172.21.15.38:0/4052264054 wait complete. 2024-04-22T00:02:43.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 -- 172.21.15.38:0/1478811948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 msgr2=0x7fcab410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 --2- 172.21.15.38:0/1478811948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab410d460 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009ca0 tx=0x7fcaac02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 -- 172.21.15.38:0/1478811948 shutdown_connections 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 --2- 172.21.15.38:0/1478811948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab410d460 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 --2- 172.21.15.38:0/1478811948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 --2- 172.21.15.38:0/1478811948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab4106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 -- 172.21.15.38:0/1478811948 >> 172.21.15.38:0/1478811948 conn(0x7fcab4076440 msgr2=0x7fcab4076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 -- 172.21.15.38:0/1478811948 shutdown_connections 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.979+0000 7fcabad5c640 1 -- 172.21.15.38:0/1478811948 wait complete. 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 Processor -- start 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 -- start start 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab41a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4110a00 con 0x7fcab4101330 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4110880 con 0x7fcab4106690 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcabad5c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4110b80 con 0x7fcab4100990 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44742/0 (socket says 172.21.15.38:44742) 2024-04-22T00:02:43.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 -- 172.21.15.38:0/2129154931 learned_addr learned my addr 172.21.15.38:0/2129154931 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcab92f9640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab41a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 -- 172.21.15.38:0/2129154931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 msgr2=0x7fcab4198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 -- 172.21.15.38:0/2129154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 msgr2=0x7fcab41a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab41a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 -- 172.21.15.38:0/2129154931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab41a42e0 con 0x7fcab4101330 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcab92f9640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab41a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400b4a0 tx=0x7fcaa400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.980+0000 7fcaa9ffb640 1 -- 172.21.15.38:0/2129154931 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4014070 con 0x7fcab4101330 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.981+0000 7fcaa9ffb640 1 -- 172.21.15.38:0/2129154931 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa4010af0 con 0x7fcab4101330 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.981+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab41a44b0 con 0x7fcab4101330 2024-04-22T00:02:43.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.981+0000 7fcaa9ffb640 1 -- 172.21.15.38:0/2129154931 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4017d10 con 0x7fcab4101330 2024-04-22T00:02:43.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.981+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab41a4980 con 0x7fcab4101330 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcaa9ffb640 1 -- 172.21.15.38:0/2129154931 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fcaa4017070 con 0x7fcab4101330 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 Processor -- start 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 -- start start 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 0x56503fd72400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcaa9ffb640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 0x7fca9c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56503fcd6530 con 0x56503fd72940 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56503fc7e830 con 0x56503fd72020 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.983+0000 7fcabda17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56503fcd24f0 con 0x56503fd7b520 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 0x7fca9c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44744/0 (socket says 172.21.15.38:44744) 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 -- 172.21.15.38:0/667433680 learned_addr learned my addr 172.21.15.38:0/667433680 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaa9ffb640 1 -- 172.21.15.38:0/2129154931 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcaa40cedb0 con 0x7fcab4101330 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcab92f9640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 0x56503fd72400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcab92f9640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 -- 172.21.15.38:0/667433680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 msgr2=0x56503fd7d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 -- 172.21.15.38:0/667433680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 msgr2=0x56503fd72400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 0x56503fd72400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 -- 172.21.15.38:0/667433680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56503fca22c0 con 0x56503fd72940 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcab92f9640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 0x56503fd72400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 0x7fca9c0a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcab000baa0 tx=0x7fcab0005d90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fcaabfff640 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40972b0 tx=0x7fcaa4069d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45607abe6cea603 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fca9affd640 1 -- 172.21.15.38:0/667433680 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa401e040 con 0x56503fd72940 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fca9affd640 1 -- 172.21.15.38:0/667433680 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa409d070 con 0x56503fd72940 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.984+0000 7fca9affd640 1 -- 172.21.15.38:0/667433680 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40a3ac0 con 0x56503fd72940 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- 172.21.15.38:0/667433680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 msgr2=0x56503fd7afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40972b0 tx=0x7fcaa4069d80 comp rx=0 tx=0).stop 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- 172.21.15.38:0/667433680 shutdown_connections 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72940 0x56503fd7afe0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 --2- 172.21.15.38:0/667433680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72020 0x56503fd72400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- 172.21.15.38:0/667433680 >> 172.21.15.38:0/667433680 conn(0x56503fd6c750 msgr2=0x56503fd6cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- 172.21.15.38:0/667433680 shutdown_connections 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- 172.21.15.38:0/667433680 wait complete. 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.985+0000 7fcabda17740 1 -- start start 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 Processor -- start 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd78270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd78880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56503fd7e310 con 0x56503fd72020 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56503fc91f10 con 0x56503fd7b520 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56503fd7e490 con 0x56503fd72940 2024-04-22T00:02:43.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcabda17740 10 client.0 _ll_register_callbacks cb 0x56503f9e9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44750/0 (socket says 172.21.15.38:44750) 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 -- 172.21.15.38:0/567286761 learned_addr learned my addr 172.21.15.38:0/567286761 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab92f9640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd78880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 -- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 msgr2=0x56503fd78270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcaabfff640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd78270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd78270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 -- 172.21.15.38:0/567286761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 msgr2=0x56503fd78880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd78880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab8af8640 1 -- 172.21.15.38:0/567286761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56503fc63c80 con 0x56503fd72020 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.986+0000 7fcab92f9640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd78880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcab8af8640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fcab000f9d0 tx=0x7fcab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a317f1896587a115 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:43.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0020070 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcab000ecc0 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaa409d070 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcab002b050 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 10 client.15864 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56503f9ea3f0 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaa409d070 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcab001b4d0 con 0x56503fd72020 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 10 client.15864 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 10 client.15864 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 20 client.15864 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fd8b540 0x56503fd8d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 4 client.15864 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.987+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56503fce3a70 con 0x56503fd8b540 2024-04-22T00:02:43.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.988+0000 7fcaabfff640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fd8b540 0x56503fd8d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.988+0000 7fca93fff640 10 client.15864 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.988+0000 7fcaabfff640 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fd8b540 0x56503fd8d950 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1309c52380b6893 server_cookie=7581d588e7ce9561 in_seq=0 out_seq=0 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fca93fff640 1 -- 172.21.15.38:0/567286761 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56503fce3a70 con 0x56503fd8b540 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fca93fff640 10 client.15864 handle_command_reply: tid=0 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 client.15864 shutdown 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 msgr2=0x56503fd7aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fcab000f9d0 tx=0x7fcab00046c0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 shutdown_connections 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fd8b540 0x56503fd8d950 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd78880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd78270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fca93fff640 0 client.15864 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:43.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 --2- 172.21.15.38:0/567286761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd7aee0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:43.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 >> 172.21.15.38:0/567286761 conn(0x56503fd6c750 msgr2=0x56503fd6e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:43.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 shutdown_connections 2024-04-22T00:02:43.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 1 -- 172.21.15.38:0/567286761 wait complete. 2024-04-22T00:02:43.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:43.989+0000 7fcabda17740 20 client.15864 trim_cache size 0 max 16384 2024-04-22T00:02:44.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.002+0000 7fcabda17740 1 Processor -- start 2024-04-22T00:02:44.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 -- start start 2024-04-22T00:02:44.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd7a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd7aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41aea30 con 0x56503fd72020 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41b0570 con 0x56503fd7b520 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcabda17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41af100 con 0x56503fd72940 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44754/0 (socket says 172.21.15.38:44754) 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 -- 172.21.15.38:0/2815718647 learned_addr learned my addr 172.21.15.38:0/2815718647 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 -- 172.21.15.38:0/2815718647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 msgr2=0x56503fd7a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd7a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 -- 172.21.15.38:0/2815718647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 msgr2=0x56503fd7aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd7aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 -- 172.21.15.38:0/2815718647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56503fab47d0 con 0x56503fd72020 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005fb0 tx=0x7fcab000f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9188abcb019eec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fca9a7fc640 1 -- 172.21.15.38:0/2815718647 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0033070 con 0x56503fd72020 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fca9a7fc640 1 -- 172.21.15.38:0/2815718647 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab001bce0 con 0x56503fd72020 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.003+0000 7fca9a7fc640 1 -- 172.21.15.38:0/2815718647 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0020050 con 0x56503fd72020 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- 172.21.15.38:0/2815718647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 msgr2=0x56503fd871d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcab0005fb0 tx=0x7fcab000f6b0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- 172.21.15.38:0/2815718647 shutdown_connections 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd7b520 0x56503fd7aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd72940 0x56503fd7a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 --2- 172.21.15.38:0/2815718647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd871d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- 172.21.15.38:0/2815718647 >> 172.21.15.38:0/2815718647 conn(0x56503fd6c750 msgr2=0x56503fd73930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- 172.21.15.38:0/2815718647 shutdown_connections 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- 172.21.15.38:0/2815718647 wait complete. 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 -- start start 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.004+0000 7fcabda17740 1 Processor -- start 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 0x56503fd7ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41afea0 con 0x56503fd72020 2024-04-22T00:02:44.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41af7d0 con 0x56503fd72940 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41aca10 con 0x56503fd7b520 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcabda17740 10 client.0 _ll_register_callbacks cb 0x7fcab0013070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44762/0 (socket says 172.21.15.38:44762) 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 -- 172.21.15.38:0/2502547502 learned_addr learned my addr 172.21.15.38:0/2502547502 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 0x56503fd7ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 -- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 msgr2=0x56503fd7f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab92f9640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 -- 172.21.15.38:0/2502547502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 msgr2=0x56503fd7ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 0x56503fd7ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 -- 172.21.15.38:0/2502547502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56503fd7fd80 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 0x56503fd7ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab92f9640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fcab8af8640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab001b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e65164443a4eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fca93fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab003b070 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.005+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcab0005030 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcab001bce0 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcab0047050 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 10 client.15870 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56503fdf1bc0 con 0x56503fd72020 2024-04-22T00:02:44.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab001bce0 con 0x56503fd72020 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcab0004cc0 con 0x56503fd72020 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 10 client.15870 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 10 client.15870 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 20 client.15870 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fdf21d0 0x56503fdf45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 4 client.15870 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56503fcdddc0 con 0x56503fdf21d0 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.006+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fdf21d0 0x56503fdf45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fca93fff640 10 client.15870 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcaabfff640 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fdf21d0 0x56503fdf45c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f0db24b9c32794f6 server_cookie=660a3bb2bb834a02 in_seq=0 out_seq=0 2024-04-22T00:02:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fca93fff640 1 -- 172.21.15.38:0/2502547502 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56503fcdddc0 con 0x56503fdf21d0 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fca93fff640 10 client.15870 handle_command_reply: tid=0 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 client.15870 shutdown 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 msgr2=0x56503fd83bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcab0000c00 tx=0x7fcab001b430 comp rx=0 tx=0).stop 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 shutdown_connections 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x56503fdf21d0 0x56503fdf45c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56503fd7b520 0x56503fd7f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56503fd72940 0x56503fd7ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 --2- 172.21.15.38:0/2502547502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56503fd72020 0x56503fd83bd0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fca93fff640 0 client.15870 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 >> 172.21.15.38:0/2502547502 conn(0x56503fd6c750 msgr2=0x56503fd6dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 shutdown_connections 2024-04-22T00:02:44.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 1 -- 172.21.15.38:0/2502547502 wait complete. 2024-04-22T00:02:44.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.007+0000 7fcabda17740 20 client.15870 trim_cache size 0 max 16384 2024-04-22T00:02:44.009 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 msgr2=0x7fca9c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 0x7fca9c0a52e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcab000baa0 tx=0x7fcab0005d90 comp rx=0 tx=0).stop 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 msgr2=0x7fcab4199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400b4a0 tx=0x7fcaa400b960 comp rx=0 tx=0).stop 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 shutdown_connections 2024-04-22T00:02:44.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fca9c0a2e20 0x7fca9c0a52e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcab4106690 0x7fcab41a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcab4101330 0x7fcab4199260 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 --2- 172.21.15.38:0/2129154931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcab4100990 0x7fcab4198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 >> 172.21.15.38:0/2129154931 conn(0x7fcab4076440 msgr2=0x7fcab40ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 shutdown_connections 2024-04-22T00:02:44.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.008+0000 7fcabad5c640 1 -- 172.21.15.38:0/2129154931 wait complete. 2024-04-22T00:02:44.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 -- 172.21.15.38:0/511144531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a041020d0 msgr2=0x7f4a0410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/511144531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a041020d0 0x7f4a0410e5e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009cd0 tx=0x7f49fc02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 -- 172.21.15.38:0/511144531 shutdown_connections 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/511144531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a041020d0 0x7f4a0410e5e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/511144531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a04101710 0x7f4a04101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/511144531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04107720 0x7f4a04107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 -- 172.21.15.38:0/511144531 >> 172.21.15.38:0/511144531 conn(0x7f4a040fd180 msgr2=0x7f4a040ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 -- 172.21.15.38:0/511144531 shutdown_connections 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.123+0000 7f4a0ad77640 1 -- 172.21.15.38:0/511144531 wait complete. 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 Processor -- start 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 -- start start 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 0x7f4a04198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a041020d0 0x7f4a041985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a04111b30 con 0x7f4a04107720 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041119b0 con 0x7f4a04101710 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a0ad77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a04111cb0 con 0x7f4a041020d0 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a09314640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a09314640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44788/0 (socket says 172.21.15.38:44788) 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a09314640 1 -- 172.21.15.38:0/2267324891 learned_addr learned my addr 172.21.15.38:0/2267324891 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a041020d0 0x7f4a041985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a09314640 1 -- 172.21.15.38:0/2267324891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a041020d0 msgr2=0x7f4a041985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 0x7f4a04198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.124+0000 7f4a09314640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a041020d0 0x7f4a041985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a09314640 1 -- 172.21.15.38:0/2267324891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 msgr2=0x7f4a04198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a09314640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 0x7f4a04198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a09314640 1 -- 172.21.15.38:0/2267324891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0419d060 con 0x7f4a04107720 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 0x7f4a04198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a09314640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f49fc005e30 tx=0x7f49fc032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a01ffb640 1 -- 172.21.15.38:0/2267324891 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc041020 con 0x7f4a04107720 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a01ffb640 1 -- 172.21.15.38:0/2267324891 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc034cd0 con 0x7f4a04107720 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0419d2f0 con 0x7f4a04107720 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a01ffb640 1 -- 172.21.15.38:0/2267324891 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc039040 con 0x7f4a04107720 2024-04-22T00:02:44.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.125+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a041a4be0 con 0x7f4a04107720 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.127+0000 7f4a01ffb640 1 -- 172.21.15.38:0/2267324891 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f49fc044020 con 0x7f4a04107720 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.127+0000 7f4a0da3a740 1 Processor -- start 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 -- start start 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c0bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c193f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560665b70530 con 0x560665c19930 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560665b18830 con 0x560665c0c3b0 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560665b6c4f0 con 0x560665c0ba00 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a01ffb640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 0x7f49f40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44798/0 (socket says 172.21.15.38:44798) 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 -- 172.21.15.38:0/1343915787 learned_addr learned my addr 172.21.15.38:0/1343915787 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a08b13640 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c193f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c0bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a09314640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 0x7f49f40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a01ffb640 1 -- 172.21.15.38:0/2267324891 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f49fc0f17b0 con 0x7f4a04107720 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 -- 172.21.15.38:0/1343915787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 msgr2=0x560665c0bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c0bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 -- 172.21.15.38:0/1343915787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 msgr2=0x560665c193f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c193f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 -- 172.21.15.38:0/1343915787 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560665b3c2c0 con 0x560665c19930 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.128+0000 7f4a03fff640 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f49f0013580 tx=0x7f49f000c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c59ee809753d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a09314640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 0x7f49f40a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0eebb0 tx=0x7f49fc0b5040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f49eeffd640 1 -- 172.21.15.38:0/1343915787 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0005020 con 0x560665c19930 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f49eeffd640 1 -- 172.21.15.38:0/1343915787 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f00047a0 con 0x560665c19930 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f49eeffd640 1 -- 172.21.15.38:0/1343915787 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0011760 con 0x560665c19930 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 -- 172.21.15.38:0/1343915787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 msgr2=0x560665c1bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f49f0013580 tx=0x7f49f000c4d0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 -- 172.21.15.38:0/1343915787 shutdown_connections 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c19930 0x560665c1bd20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c193f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/1343915787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c0bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 -- 172.21.15.38:0/1343915787 >> 172.21.15.38:0/1343915787 conn(0x560665c06750 msgr2=0x560665c06b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 -- 172.21.15.38:0/1343915787 shutdown_connections 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.129+0000 7f4a0da3a740 1 -- 172.21.15.38:0/1343915787 wait complete. 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 -- start start 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 Processor -- start 2024-04-22T00:02:44.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c165b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c16af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560665c1c6f0 con 0x560665c0ba00 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560665b2bf10 con 0x560665c0c3b0 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560665c1c870 con 0x560665c19930 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a0da3a740 10 client.0 _ll_register_callbacks cb 0x560665883be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44800/0 (socket says 172.21.15.38:44800) 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.130+0000 7f4a03fff640 1 -- 172.21.15.38:0/3614372669 learned_addr learned my addr 172.21.15.38:0/3614372669 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a08b13640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c165b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c16af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 -- 172.21.15.38:0/3614372669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 msgr2=0x560665c16af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c16af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 -- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 msgr2=0x560665c165b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c165b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 -- 172.21.15.38:0/3614372669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560665afdc80 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a03fff640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c840 tx=0x7f49f0005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7ce3d9fab5281c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f49e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0011500 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f00047a0 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f49f00116a0 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49f0035050 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a0da3a740 10 client.15882 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5606658843f0 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.131+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f00047a0 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f49f002a020 con 0x560665c0ba00 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 10 client.15882 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 10 client.15882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 20 client.15882 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c29880 0x560665c2bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 4 client.15882 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560665b7da70 con 0x560665c29880 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.132+0000 7f4a08b13640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c29880 0x560665c2bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f49e3fff640 10 client.15882 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a08b13640 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c29880 0x560665c2bc70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=99d70bf6034ac074 server_cookie=f087b023765edd9f in_seq=0 out_seq=0 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f49e3fff640 1 -- 172.21.15.38:0/3614372669 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560665b7da70 con 0x560665c29880 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f49e3fff640 10 client.15882 handle_command_reply: tid=0 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 client.15882 shutdown 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 msgr2=0x560665c19220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c840 tx=0x7f49f0005c70 comp rx=0 tx=0).stop 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 shutdown_connections 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c16af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0c3b0 0x560665c165b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c29880 0x560665c2bc70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/3614372669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0ba00 0x560665c19220 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f49e3fff640 0 client.15882 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.133+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 >> 172.21.15.38:0/3614372669 conn(0x560665c06750 msgr2=0x560665c084a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.134+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 shutdown_connections 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.134+0000 7f4a0da3a740 1 -- 172.21.15.38:0/3614372669 wait complete. 2024-04-22T00:02:44.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.134+0000 7f4a0da3a740 20 client.15882 trim_cache size 0 max 16384 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 Processor -- start 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 -- start start 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c25650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 0x560665c18ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ac870 con 0x560665c0c3b0 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ae3b0 con 0x560665c19930 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041acf40 con 0x560665c0ba00 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a08b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a08b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44814/0 (socket says 172.21.15.38:44814) 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.147+0000 7f4a08b13640 1 -- 172.21.15.38:0/2202036862 learned_addr learned my addr 172.21.15.38:0/2202036862 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 -- 172.21.15.38:0/2202036862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 msgr2=0x560665c25650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c25650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 -- 172.21.15.38:0/2202036862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 msgr2=0x560665c18ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 0x560665c18ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 0x560665c18ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 -- 172.21.15.38:0/2202036862 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56066594e7d0 con 0x560665c0c3b0 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 0x560665c18ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009cf0 tx=0x7f49f8014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e268e9d021fcc080 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f49ee7fc640 1 -- 172.21.15.38:0/2202036862 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8024070 con 0x560665c0c3b0 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f49ee7fc640 1 -- 172.21.15.38:0/2202036862 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f801e430 con 0x560665c0c3b0 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f49ee7fc640 1 -- 172.21.15.38:0/2202036862 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8003a50 con 0x560665c0c3b0 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2202036862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 msgr2=0x560665c155e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009cf0 tx=0x7f49f8014840 comp rx=0 tx=0).stop 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.148+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2202036862 shutdown_connections 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c19930 0x560665c18ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c155e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2202036862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c0ba00 0x560665c25650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2202036862 >> 172.21.15.38:0/2202036862 conn(0x560665c06750 msgr2=0x560665c11b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2202036862 shutdown_connections 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2202036862 wait complete. 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- start start 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 Processor -- start 2024-04-22T00:02:44.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0ba00 0x560665c21b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c12bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041adce0 con 0x560665c0c3b0 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ad610 con 0x560665c0ba00 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041aa850 con 0x560665c19930 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a0da3a740 10 client.0 _ll_register_callbacks cb 0x7f49f8009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a08b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a08b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44824/0 (socket says 172.21.15.38:44824) 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.149+0000 7f4a08b13640 1 -- 172.21.15.38:0/2283243242 learned_addr learned my addr 172.21.15.38:0/2283243242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c12bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 -- 172.21.15.38:0/2283243242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 msgr2=0x560665c12bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c12bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 -- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0ba00 msgr2=0x560665c21b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0ba00 0x560665c21b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 -- 172.21.15.38:0/2283243242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560665c133e0 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c12bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a08b13640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49f8022040 tx=0x7f49f80142e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffaa90d2af6dae61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f49e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f802d060 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f49f801e8e0 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f801e430 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49f8024040 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a0da3a740 10 client.15888 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560665c8f6b0 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.150+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f801e430 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f49f800f040 con 0x560665c0c3b0 2024-04-22T00:02:44.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 10 client.15888 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 10 client.15888 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 20 client.15888 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c8fda0 0x560665c921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 4 client.15888 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560665b77dc0 con 0x560665c8fda0 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c8fda0 0x560665c921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f49e3fff640 10 client.15888 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f4a03fff640 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c8fda0 0x560665c921c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb3c0e2ebaa1a85e server_cookie=906dc9a86defe3e6 in_seq=0 out_seq=0 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f49e3fff640 1 -- 172.21.15.38:0/2283243242 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560665b77dc0 con 0x560665c8fda0 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.151+0000 7f49e3fff640 10 client.15888 handle_command_reply: tid=0 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 client.15888 shutdown 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 msgr2=0x560665c220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49f8022040 tx=0x7f49f80142e0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 shutdown_connections 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560665c19930 0x560665c12bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x560665c8fda0 0x560665c921c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560665c0c3b0 0x560665c220a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 --2- 172.21.15.38:0/2283243242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560665c0ba00 0x560665c21b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f49e3fff640 0 client.15888 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 >> 172.21.15.38:0/2283243242 conn(0x560665c06750 msgr2=0x560665c08390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 shutdown_connections 2024-04-22T00:02:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 1 -- 172.21.15.38:0/2283243242 wait complete. 2024-04-22T00:02:44.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0da3a740 20 client.15888 trim_cache size 0 max 16384 2024-04-22T00:02:44.152 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 msgr2=0x7f49f40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 0x7f49f40a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0eebb0 tx=0x7f49fc0b5040 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 msgr2=0x7f4a0419c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f49fc005e30 tx=0x7f49fc032ea0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 shutdown_connections 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f49f40a2e20 0x7f49f40a52e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a04107720 0x7f4a0419c930 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a041020d0 0x7f4a041985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.152+0000 7f4a0ad77640 1 --2- 172.21.15.38:0/2267324891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a04101710 0x7f4a04198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.153+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 >> 172.21.15.38:0/2267324891 conn(0x7f4a040fd180 msgr2=0x7f4a040fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.153+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 shutdown_connections 2024-04-22T00:02:44.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.153+0000 7f4a0ad77640 1 -- 172.21.15.38:0/2267324891 wait complete. 2024-04-22T00:02:44.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:43 smithi038 ceph-mon[27972]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-04-22T00:02:44.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:43 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4052264054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:44.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 -- 172.21.15.38:0/4227433975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc107730 msgr2=0x7fe0cc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/4227433975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc107730 0x7fe0cc107b10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009930 tx=0x7fe0c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 -- 172.21.15.38:0/4227433975 shutdown_connections 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/4227433975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc06a780 0x7fe0cc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/4227433975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 0x7fe0cc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/4227433975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc107730 0x7fe0cc107b10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 -- 172.21.15.38:0/4227433975 >> 172.21.15.38:0/4227433975 conn(0x7fe0cc0776a0 msgr2=0x7fe0cc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 -- 172.21.15.38:0/4227433975 shutdown_connections 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.268+0000 7fe0d44c9640 1 -- 172.21.15.38:0/4227433975 wait complete. 2024-04-22T00:02:44.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 Processor -- start 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 -- start start 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 0x7fe0cc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 0x7fe0cc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc111b00 con 0x7fe0cc06a780 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc111980 con 0x7fe0cc069dc0 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d44c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc111c80 con 0x7fe0cc107730 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44846/0 (socket says 172.21.15.38:44846) 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1017315802 learned_addr learned my addr 172.21.15.38:0/1017315802 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d2265640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 0x7fe0cc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 0x7fe0cc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1017315802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 msgr2=0x7fe0cc19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 0x7fe0cc19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1017315802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 msgr2=0x7fe0cc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 0x7fe0cc198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1017315802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc19d050 con 0x7fe0cc06a780 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 0x7fe0cc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.269+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00d950 tx=0x7fe0bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.270+0000 7fe0c37fe640 1 -- 172.21.15.38:0/1017315802 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc017070 con 0x7fe0cc06a780 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.270+0000 7fe0c37fe640 1 -- 172.21.15.38:0/1017315802 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0bc009c30 con 0x7fe0cc06a780 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.270+0000 7fe0c37fe640 1 -- 172.21.15.38:0/1017315802 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc00f6b0 con 0x7fe0cc06a780 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.270+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc19d340 con 0x7fe0cc06a780 2024-04-22T00:02:44.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.270+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc1a4bd0 con 0x7fe0cc06a780 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.272+0000 7fe0c37fe640 1 -- 172.21.15.38:0/1017315802 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe0bc0109d0 con 0x7fe0cc06a780 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.272+0000 7fe0d718f740 1 Processor -- start 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 -- start start 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4904400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555ef4868530 con 0x555ef490d520 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555ef4810830 con 0x555ef4904020 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d718f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555ef48644f0 con 0x555ef4904940 2024-04-22T00:02:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44850/0 (socket says 172.21.15.38:44850) 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 -- 172.21.15.38:0/14893202 learned_addr learned my addr 172.21.15.38:0/14893202 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0c37fe640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 0x7fe0b80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d2265640 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4904400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 -- 172.21.15.38:0/14893202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 msgr2=0x555ef490cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 -- 172.21.15.38:0/14893202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 msgr2=0x555ef4904400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4904400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 0x7fe0b80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.273+0000 7fe0d1a64640 1 -- 172.21.15.38:0/14893202 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef48342c0 con 0x555ef490d520 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0c37fe640 1 -- 172.21.15.38:0/1017315802 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0bc0cf5d0 con 0x7fe0cc06a780 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0cc970 tx=0x7fe0bc066190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa26eb2719a0621 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 0x7fe0b80a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4005fd0 tx=0x7fe0c4009240 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0c0bfe640 1 -- 172.21.15.38:0/14893202 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc09b0a0 con 0x555ef490d520 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0c0bfe640 1 -- 172.21.15.38:0/14893202 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0bc09bc60 con 0x555ef490d520 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.274+0000 7fe0c0bfe640 1 -- 172.21.15.38:0/14893202 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc09aec0 con 0x555ef490d520 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- 172.21.15.38:0/14893202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 msgr2=0x555ef490f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0cc970 tx=0x7fe0bc066190 comp rx=0 tx=0).stop 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- 172.21.15.38:0/14893202 shutdown_connections 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490f910 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- 172.21.15.38:0/14893202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4904400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- 172.21.15.38:0/14893202 >> 172.21.15.38:0/14893202 conn(0x555ef48fe750 msgr2=0x555ef48feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- 172.21.15.38:0/14893202 shutdown_connections 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- 172.21.15.38:0/14893202 wait complete. 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- start start 2024-04-22T00:02:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 Processor -- start 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 0x555ef490a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef490a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555ef49102b0 con 0x555ef4904020 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555ef4823f10 con 0x555ef4904940 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555ef4910430 con 0x555ef490d520 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d718f740 10 client.0 _ll_register_callbacks cb 0x555ef457bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.275+0000 7fe0d2265640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44862/0 (socket says 172.21.15.38:44862) 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 -- 172.21.15.38:0/2495577404 learned_addr learned my addr 172.21.15.38:0/2495577404 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 0x555ef490a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 -- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 msgr2=0x555ef490a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef490a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 -- 172.21.15.38:0/2495577404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 msgr2=0x555ef490a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 0x555ef490a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 -- 172.21.15.38:0/2495577404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef47f5c80 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 0x555ef490a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d2265640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0c802cfd0 tx=0x7fe0c8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f179b1911464aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8004380 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0c80046c0 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0bc09bc60 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0c80048c0 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d718f740 10 client.15900 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ef457c3f0 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.276+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0bc09bc60 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe0c80035f0 con 0x555ef4904020 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 10 client.15900 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 20 client.15900 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef491d530 0x555ef491f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 4 client.15900 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ef4875a70 con 0x555ef491d530 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef491d530 0x555ef491f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0b5ffb640 10 client.15900 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.277+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef491d530 0x555ef491f9c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8b0e1bee6a8c8d9 server_cookie=307d05857df79e20 in_seq=0 out_seq=0 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/2495577404 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ef4875a70 con 0x555ef491d530 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0b5ffb640 10 client.15900 handle_command_reply: tid=0 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 client.15900 shutdown 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 msgr2=0x555ef490ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0c802cfd0 tx=0x7fe0c8005dd0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 shutdown_connections 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef491d530 0x555ef491f9c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef490a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904940 0x555ef490a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0b5ffb640 0 client.15900 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2495577404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904020 0x555ef490ceb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 >> 172.21.15.38:0/2495577404 conn(0x555ef48fe750 msgr2=0x555ef49004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 shutdown_connections 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.278+0000 7fe0d718f740 1 -- 172.21.15.38:0/2495577404 wait complete. 2024-04-22T00:02:44.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.279+0000 7fe0d718f740 20 client.15900 trim_cache size 0 max 16384 2024-04-22T00:02:44.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:43 smithi203 ceph-mon[31431]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-04-22T00:02:44.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:43 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4052264054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 Processor -- start 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 -- start start 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4919170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1aeb10 con 0x555ef490d520 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1b0650 con 0x555ef4904020 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d718f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1af1e0 con 0x555ef4904940 2024-04-22T00:02:44.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2a66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2a66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44878/0 (socket says 172.21.15.38:44878) 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2a66640 1 -- 172.21.15.38:0/2314954676 learned_addr learned my addr 172.21.15.38:0/2314954676 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2265640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4919170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2a66640 1 -- 172.21.15.38:0/2314954676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 msgr2=0x555ef490c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.292+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d2a66640 1 -- 172.21.15.38:0/2314954676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 msgr2=0x555ef4919170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4919170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d2a66640 1 -- 172.21.15.38:0/2314954676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef46467d0 con 0x555ef490d520 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d2265640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4919170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4011010 tx=0x7fe0c4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b791c236a6871263 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0b7fff640 1 -- 172.21.15.38:0/2314954676 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4022070 con 0x555ef490d520 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0b7fff640 1 -- 172.21.15.38:0/2314954676 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c4027b40 con 0x555ef490d520 2024-04-22T00:02:44.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0b7fff640 1 -- 172.21.15.38:0/2314954676 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c401ec90 con 0x555ef490d520 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 -- 172.21.15.38:0/2314954676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 msgr2=0x555ef490ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4011010 tx=0x7fe0c4009680 comp rx=0 tx=0).stop 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 -- 172.21.15.38:0/2314954676 shutdown_connections 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef490d520 0x555ef490ceb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef4904940 0x555ef490c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 --2- 172.21.15.38:0/2314954676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4919170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.293+0000 7fe0d718f740 1 -- 172.21.15.38:0/2314954676 >> 172.21.15.38:0/2314954676 conn(0x555ef48fe750 msgr2=0x555ef4905810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- 172.21.15.38:0/2314954676 shutdown_connections 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- 172.21.15.38:0/2314954676 wait complete. 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- start start 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 Processor -- start 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4911230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef4911cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1aff80 con 0x555ef4904940 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1af8b0 con 0x555ef4904020 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc1acaf0 con 0x555ef490d520 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d718f740 10 client.0 _ll_register_callbacks cb 0x7fe0c8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d1a64640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44882/0 (socket says 172.21.15.38:44882) 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1104279504 learned_addr learned my addr 172.21.15.38:0/1104279504 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.294+0000 7fe0d2a66640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef4911cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d2265640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4911230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 msgr2=0x555ef4911cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef4911cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1104279504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 msgr2=0x555ef4911230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4911230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 -- 172.21.15.38:0/1104279504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef4982d20 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d1a64640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0979c0 tx=0x7fe0bc010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4c9eb0195da1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc020040 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0bc099860 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0c4027b40 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0bc09b0a0 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 10 client.15906 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ef4983f50 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c4027b40 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe0bc097bd0 con 0x555ef4904940 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 10 client.15906 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 20 client.15906 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef4984230 0x555ef4986620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 4 client.15906 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.295+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555ef486fdc0 con 0x555ef4984230 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d2265640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef4984230 0x555ef4986620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d2265640 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef4984230 0x555ef4986620 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ebb6f1b2c0b4cc1 server_cookie=2178e55e8a524e73 in_seq=0 out_seq=0 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0b5ffb640 10 client.15906 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0b5ffb640 1 -- 172.21.15.38:0/1104279504 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555ef486fdc0 con 0x555ef4984230 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0b5ffb640 10 client.15906 handle_command_reply: tid=0 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 client.15906 shutdown 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 msgr2=0x555ef4911770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0979c0 tx=0x7fe0bc010d70 comp rx=0 tx=0).stop 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 shutdown_connections 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x555ef4984230 0x555ef4986620 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555ef490d520 0x555ef4911cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef4904940 0x555ef4911770 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 --2- 172.21.15.38:0/1104279504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ef4904020 0x555ef4911230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0b5ffb640 0 client.15906 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 >> 172.21.15.38:0/1104279504 conn(0x555ef48fe750 msgr2=0x555ef4904ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 shutdown_connections 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.296+0000 7fe0d718f740 1 -- 172.21.15.38:0/1104279504 wait complete. 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d718f740 20 client.15906 trim_cache size 0 max 16384 2024-04-22T00:02:44.296 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 msgr2=0x7fe0b80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 0x7fe0b80a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4005fd0 tx=0x7fe0c4009240 comp rx=0 tx=0).stop 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 msgr2=0x7fe0cc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00d950 tx=0x7fe0bc00de10 comp rx=0 tx=0).stop 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 shutdown_connections 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe0b80a2ea0 0x7fe0b80a5360 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0cc107730 0x7fe0cc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0cc06a780 0x7fe0cc1985a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 --2- 172.21.15.38:0/1017315802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc069dc0 0x7fe0cc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 >> 172.21.15.38:0/1017315802 conn(0x7fe0cc0776a0 msgr2=0x7fe0cc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 shutdown_connections 2024-04-22T00:02:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.297+0000 7fe0d44c9640 1 -- 172.21.15.38:0/1017315802 wait complete. 2024-04-22T00:02:44.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:02:44.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:43 smithi111 ceph-mon[31543]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-04-22T00:02:44.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:43 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4052264054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 -- 172.21.15.38:0/2021937675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c106690 msgr2=0x7f378c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 --2- 172.21.15.38:0/2021937675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c106690 0x7f378c106a70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3788009900 tx=0x7f378802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 -- 172.21.15.38:0/2021937675 shutdown_connections 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 --2- 172.21.15.38:0/2021937675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 0x7f378c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 --2- 172.21.15.38:0/2021937675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c100990 0x7f378c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 --2- 172.21.15.38:0/2021937675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c106690 0x7f378c106a70 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.411+0000 7f379330c640 1 -- 172.21.15.38:0/2021937675 >> 172.21.15.38:0/2021937675 conn(0x7f378c076440 msgr2=0x7f378c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- 172.21.15.38:0/2021937675 shutdown_connections 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- 172.21.15.38:0/2021937675 wait complete. 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 Processor -- start 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- start start 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 0x7f378c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 0x7f378c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f378c110a20 con 0x7f378c100990 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1108a0 con 0x7f378c106690 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f379330c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f378c110ba0 con 0x7f378c101330 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f37910a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.412+0000 7f37910a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44892/0 (socket says 172.21.15.38:44892) 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 -- 172.21.15.38:0/2383353415 learned_addr learned my addr 172.21.15.38:0/2383353415 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37918a9640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 0x7f378c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37908a7640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 0x7f378c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 -- 172.21.15.38:0/2383353415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 msgr2=0x7f378c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 0x7f378c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 -- 172.21.15.38:0/2383353415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 msgr2=0x7f378c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 0x7f378c19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 -- 172.21.15.38:0/2383353415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c19cfc0 con 0x7f378c100990 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37918a9640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 0x7f378c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37910a8640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3788006010 tx=0x7f378802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37827fc640 1 -- 172.21.15.38:0/2383353415 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788041070 con 0x7f378c100990 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378c19d250 con 0x7f378c100990 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37827fc640 1 -- 172.21.15.38:0/2383353415 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3788004450 con 0x7f378c100990 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f37827fc640 1 -- 172.21.15.38:0/2383353415 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788034440 con 0x7f378c100990 2024-04-22T00:02:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.413+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378c101ff0 con 0x7f378c100990 2024-04-22T00:02:44.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.415+0000 7f37827fc640 1 -- 172.21.15.38:0/2383353415 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f3788044070 con 0x7f378c100990 2024-04-22T00:02:44.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 Processor -- start 2024-04-22T00:02:44.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 -- start start 2024-04-22T00:02:44.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e200400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e208fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37827fc640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 0x7f37780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564a3e164530 con 0x564a3e209520 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564a3e10c830 con 0x564a3e200940 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f3795fd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564a3e1604f0 con 0x564a3e200020 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37910a8640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 0x7f37780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44906/0 (socket says 172.21.15.38:44906) 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37918a9640 1 -- 172.21.15.38:0/1669243968 learned_addr learned my addr 172.21.15.38:0/1669243968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37827fc640 1 -- 172.21.15.38:0/2383353415 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f37880f16e0 con 0x7f378c100990 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37918a9640 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e200400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.416+0000 7f37908a7640 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e208fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 -- 172.21.15.38:0/1669243968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 msgr2=0x564a3e200400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e200400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 -- 172.21.15.38:0/1669243968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 msgr2=0x564a3e208fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e208fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 -- 172.21.15.38:0/1669243968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a3e1302c0 con 0x564a3e209520 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37910a8640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 0x7f37780a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37880eea80 tx=0x7f37880b2bb0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37918a9640 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3784012870 tx=0x7f37840097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d5391c49df86a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37777fe640 1 -- 172.21.15.38:0/1669243968 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784004d20 con 0x564a3e209520 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37777fe640 1 -- 172.21.15.38:0/1669243968 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3784010d50 con 0x564a3e209520 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.417+0000 7f37777fe640 1 -- 172.21.15.38:0/1669243968 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37840057d0 con 0x564a3e209520 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- 172.21.15.38:0/1669243968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 msgr2=0x564a3e20b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3784012870 tx=0x7f37840097d0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- 172.21.15.38:0/1669243968 shutdown_connections 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e20b910 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e208fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1669243968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e200400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- 172.21.15.38:0/1669243968 >> 172.21.15.38:0/1669243968 conn(0x564a3e1fa750 msgr2=0x564a3e1fab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- 172.21.15.38:0/1669243968 shutdown_connections 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- 172.21.15.38:0/1669243968 wait complete. 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 -- start start 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.418+0000 7f3795fd2740 1 Processor -- start 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e208f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e209520 0x564a3e2068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564a3e20c310 con 0x564a3e200940 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564a3e11ff10 con 0x564a3e209520 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564a3e20c490 con 0x564a3e200020 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 10 client.0 _ll_register_callbacks cb 0x564a3de77be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44916/0 (socket says 172.21.15.38:44916) 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 -- 172.21.15.38:0/1613930797 learned_addr learned my addr 172.21.15.38:0/1613930797 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37918a9640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e209520 0x564a3e2068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37918a9640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e208f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 -- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 msgr2=0x564a3e208f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e208f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 -- 172.21.15.38:0/1613930797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e209520 msgr2=0x564a3e2068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e209520 0x564a3e2068d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 -- 172.21.15.38:0/1613930797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a3e0f1c80 con 0x564a3e200940 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f37908a7640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f377c00bf90 tx=0x7f377c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15f7fec8414b92d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3774ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c018020 con 0x564a3e200940 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f377c014870 con 0x564a3e200940 2024-04-22T00:02:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.419+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3784010d50 con 0x564a3e200940 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f377c023050 con 0x564a3e200940 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 10 client.15918 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a3de783f0 con 0x564a3e200940 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3784010d50 con 0x564a3e200940 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f377c013b20 con 0x564a3e200940 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 10 client.15918 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 20 client.15918 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e21bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 4 client.15918 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a3e171a70 con 0x564a3e219760 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.420+0000 7f37918a9640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e21bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.421+0000 7f3774ff9640 10 client.15918 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:44.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.421+0000 7f37918a9640 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e21bb80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=697e6a452eedbb66 server_cookie=deab657e927ec89b in_seq=0 out_seq=0 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.421+0000 7f3774ff9640 1 -- 172.21.15.38:0/1613930797 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a3e171a70 con 0x564a3e219760 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.421+0000 7f3774ff9640 10 client.15918 handle_command_reply: tid=0 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.421+0000 7f3795fd2740 1 client.15918 shutdown 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 msgr2=0x564a3e2062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f377c00bf90 tx=0x7f377c00f670 comp rx=0 tx=0).stop 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 shutdown_connections 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e21bb80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e209520 0x564a3e2068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3774ff9640 0 client.15918 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e200940 0x564a3e2062c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1613930797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e208f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 >> 172.21.15.38:0/1613930797 conn(0x564a3e1fa750 msgr2=0x564a3e1fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 shutdown_connections 2024-04-22T00:02:44.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 1 -- 172.21.15.38:0/1613930797 wait complete. 2024-04-22T00:02:44.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.422+0000 7f3795fd2740 20 client.15918 trim_cache size 0 max 16384 2024-04-22T00:02:44.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 Processor -- start 2024-04-22T00:02:44.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 -- start start 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e215200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e2089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1b0cc0 con 0x564a3e209520 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1b2800 con 0x564a3e200940 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.435+0000 7f3795fd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1b1390 con 0x564a3e200020 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44918/0 (socket says 172.21.15.38:44918) 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 -- 172.21.15.38:0/1242178667 learned_addr learned my addr 172.21.15.38:0/1242178667 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37908a7640 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e2089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 -- 172.21.15.38:0/1242178667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 msgr2=0x564a3e215200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e215200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 -- 172.21.15.38:0/1242178667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 msgr2=0x564a3e2089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e2089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 -- 172.21.15.38:0/1242178667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a3e206c40 con 0x564a3e209520 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37908a7640 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e2089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f37918a9640 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3784005400 tx=0x7f3784009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbd2a38114cf231 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f3776ffd640 1 -- 172.21.15.38:0/1242178667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784034070 con 0x564a3e209520 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f3776ffd640 1 -- 172.21.15.38:0/1242178667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37840124d0 con 0x564a3e209520 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.436+0000 7f3776ffd640 1 -- 172.21.15.38:0/1242178667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784026040 con 0x564a3e209520 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- 172.21.15.38:0/1242178667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 msgr2=0x564a3e208f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3784005400 tx=0x7f3784009b60 comp rx=0 tx=0).stop 2024-04-22T00:02:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- 172.21.15.38:0/1242178667 shutdown_connections 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e208f30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200940 0x564a3e2089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1242178667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200020 0x564a3e215200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- 172.21.15.38:0/1242178667 >> 172.21.15.38:0/1242178667 conn(0x564a3e1fa750 msgr2=0x564a3e1f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- 172.21.15.38:0/1242178667 shutdown_connections 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- 172.21.15.38:0/1242178667 wait complete. 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 -- start start 2024-04-22T00:02:44.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.437+0000 7f3795fd2740 1 Processor -- start 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200020 0x564a3e202e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 0x564a3e2033c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1b2130 con 0x564a3e209520 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1b1a60 con 0x564a3e200020 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1aeca0 con 0x564a3e200940 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 10 client.0 _ll_register_callbacks cb 0x7f377c00b2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44922/0 (socket says 172.21.15.38:44922) 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 -- 172.21.15.38:0/1220312001 learned_addr learned my addr 172.21.15.38:0/1220312001 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200020 0x564a3e202e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37908a7640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 0x564a3e2033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 -- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 msgr2=0x564a3e2033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 0x564a3e2033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 -- 172.21.15.38:0/1220312001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200020 msgr2=0x564a3e202e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200020 0x564a3e202e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 -- 172.21.15.38:0/1220312001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a3df427d0 con 0x564a3e209520 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37908a7640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 0x564a3e2033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f37918a9640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3784015010 tx=0x7f3784005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be54a8cdd6dd05f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3774ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784034040 con 0x564a3e209520 2024-04-22T00:02:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f378402e620 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.438+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37840124d0 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3784041050 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 10 client.15924 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a3e27fc60 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37840124d0 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f378403a060 con 0x564a3e209520 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 10 client.15924 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 20 client.15924 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e282020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 4 client.15924 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a3e16bdc0 con 0x564a3e219760 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.439+0000 7f37908a7640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e282020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3774ff9640 10 client.15924 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f37908a7640 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e282020 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4593c824bc50b68 server_cookie=e119cebc4643af60 in_seq=0 out_seq=0 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3774ff9640 1 -- 172.21.15.38:0/1220312001 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a3e16bdc0 con 0x564a3e219760 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3774ff9640 10 client.15924 handle_command_reply: tid=0 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 client.15924 shutdown 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 msgr2=0x564a3e203900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3784015010 tx=0x7f3784005cf0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 shutdown_connections 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x564a3e219760 0x564a3e282020 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a3e209520 0x564a3e203900 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564a3e200940 0x564a3e2033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3774ff9640 0 client.15924 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 --2- 172.21.15.38:0/1220312001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a3e200020 0x564a3e202e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 >> 172.21.15.38:0/1220312001 conn(0x564a3e1fa750 msgr2=0x564a3e1fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 shutdown_connections 2024-04-22T00:02:44.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 1 -- 172.21.15.38:0/1220312001 wait complete. 2024-04-22T00:02:44.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.440+0000 7f3795fd2740 20 client.15924 trim_cache size 0 max 16384 2024-04-22T00:02:44.440 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 msgr2=0x7f37780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 0x7f37780a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37880eea80 tx=0x7f37880b2bb0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 msgr2=0x7f378c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3788006010 tx=0x7f378802cc70 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 shutdown_connections 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f37780a2e20 0x7f37780a52e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f378c106690 0x7f378c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f378c101330 0x7f378c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 --2- 172.21.15.38:0/2383353415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f378c100990 0x7f378c197fd0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 >> 172.21.15.38:0/2383353415 conn(0x7f378c076440 msgr2=0x7f378c069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 shutdown_connections 2024-04-22T00:02:44.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.441+0000 7f379330c640 1 -- 172.21.15.38:0/2383353415 wait complete. 2024-04-22T00:02:44.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.555+0000 7fe43944d640 1 -- 172.21.15.38:0/2392341777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4341066b0 msgr2=0x7fe434106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.555+0000 7fe43944d640 1 --2- 172.21.15.38:0/2392341777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4341066b0 0x7fe434106a90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe424009930 tx=0x7fe42402bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 -- 172.21.15.38:0/2392341777 shutdown_connections 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- 172.21.15.38:0/2392341777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe434100fd0 0x7fe43410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- 172.21.15.38:0/2392341777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 0x7fe434100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- 172.21.15.38:0/2392341777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4341066b0 0x7fe434106a90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 -- 172.21.15.38:0/2392341777 >> 172.21.15.38:0/2392341777 conn(0x7fe434077740 msgr2=0x7fe434077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 -- 172.21.15.38:0/2392341777 shutdown_connections 2024-04-22T00:02:44.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 -- 172.21.15.38:0/2392341777 wait complete. 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 Processor -- start 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 -- start start 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 0x7fe43410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.556+0000 7fe43944d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 0x7fe43410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe43944d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe434111b30 con 0x7fe434100fd0 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe43944d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341119b0 con 0x7fe4341066b0 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe43944d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe434111cb0 con 0x7fe434100610 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44940/0 (socket says 172.21.15.38:44940) 2024-04-22T00:02:44.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 -- 172.21.15.38:0/3473705520 learned_addr learned my addr 172.21.15.38:0/3473705520 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 0x7fe43410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe432ffd640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 0x7fe43410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 -- 172.21.15.38:0/3473705520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 msgr2=0x7fe43410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 0x7fe43410df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 -- 172.21.15.38:0/3473705520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 msgr2=0x7fe43410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 0x7fe43410e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 -- 172.21.15.38:0/3473705520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4341a6410 con 0x7fe434100fd0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 0x7fe43410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c920 tx=0x7fe42800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe41bfff640 1 -- 172.21.15.38:0/3473705520 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428007c70 con 0x7fe434100fd0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe41bfff640 1 -- 172.21.15.38:0/3473705520 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe428007e10 con 0x7fe434100fd0 2024-04-22T00:02:44.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4341a6700 con 0x7fe434100fd0 2024-04-22T00:02:44.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe41bfff640 1 -- 172.21.15.38:0/3473705520 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428018890 con 0x7fe434100fd0 2024-04-22T00:02:44.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.557+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4341a6c50 con 0x7fe434100fd0 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.560+0000 7fe41bfff640 1 -- 172.21.15.38:0/3473705520 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe428018a70 con 0x7fe434100fd0 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.560+0000 7fe43c031740 1 Processor -- start 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.560+0000 7fe43c031740 1 -- start start 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a90e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a919910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56288a872530 con 0x56288a90e940 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56288a81a830 con 0x56288a917520 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe43c031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56288a86e4f0 con 0x56288a90e020 2024-04-22T00:02:44.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44946/0 (socket says 172.21.15.38:44946) 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 -- 172.21.15.38:0/1023093751 learned_addr learned my addr 172.21.15.38:0/1023093751 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe41bfff640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 0x7fe41c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe4337fe640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a90e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 -- 172.21.15.38:0/1023093751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 msgr2=0x56288a90e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe4327fc640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a919910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a90e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 -- 172.21.15.38:0/1023093751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 msgr2=0x56288a919910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a919910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 -- 172.21.15.38:0/1023093751 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56288a83e2c0 con 0x56288a90e940 2024-04-22T00:02:44.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe4337fe640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a90e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe4327fc640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a919910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe41bfff640 1 -- 172.21.15.38:0/3473705520 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe4280cfc40 con 0x7fe434100fd0 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 0x7fe41c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.561+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe424035000 tx=0x7fe424005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45518c09c43ca4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe4197fa640 1 -- 172.21.15.38:0/1023093751 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42402ca80 con 0x56288a90e940 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe4197fa640 1 -- 172.21.15.38:0/1023093751 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42402cc20 con 0x56288a90e940 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe4197fa640 1 -- 172.21.15.38:0/1023093751 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240048c0 con 0x56288a90e940 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 -- 172.21.15.38:0/1023093751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 msgr2=0x56288a916fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe424035000 tx=0x7fe424005e70 comp rx=0 tx=0).stop 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 0x7fe41c0a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0092f0 tx=0x7fe42c004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 -- 172.21.15.38:0/1023093751 shutdown_connections 2024-04-22T00:02:44.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a919910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a916fe0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 --2- 172.21.15.38:0/1023093751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a90e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 -- 172.21.15.38:0/1023093751 >> 172.21.15.38:0/1023093751 conn(0x56288a908750 msgr2=0x56288a908b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 -- 172.21.15.38:0/1023093751 shutdown_connections 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.562+0000 7fe43c031740 1 -- 172.21.15.38:0/1023093751 wait complete. 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 -- start start 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 Processor -- start 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a914110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a914650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56288a91a2b0 con 0x56288a90e020 2024-04-22T00:02:44.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56288a82df10 con 0x56288a917520 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56288a91a430 con 0x56288a90e940 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe43c031740 10 client.0 _ll_register_callbacks cb 0x56288a585be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44954/0 (socket says 172.21.15.38:44954) 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 -- 172.21.15.38:0/3970416929 learned_addr learned my addr 172.21.15.38:0/3970416929 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a914110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a914650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 -- 172.21.15.38:0/3970416929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 msgr2=0x56288a914110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a914110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 -- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 msgr2=0x56288a914650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a914650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 -- 172.21.15.38:0/3970416929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56288a7ffc80 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a914110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.563+0000 7fe432ffd640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe4240057c0 tx=0x7fe42402cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705ff07513c91c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42404a070 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe4240046c0 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42402cc20 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe424056050 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 10 client.15936 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56288a5863f0 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42402cc20 con 0x56288a90e020 2024-04-22T00:02:44.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe424003af0 con 0x56288a90e020 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 10 client.15936 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 20 client.15936 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a927200 0x56288a9295f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 4 client.15936 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.564+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56288a87fa70 con 0x56288a927200 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.565+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a927200 0x56288a9295f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.565+0000 7fe416ff5640 10 client.15936 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.565+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a927200 0x56288a9295f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71c36353de782ec server_cookie=4c5c028ba7e773bf in_seq=0 out_seq=0 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe416ff5640 1 -- 172.21.15.38:0/3970416929 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56288a87fa70 con 0x56288a927200 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe416ff5640 10 client.15936 handle_command_reply: tid=0 2024-04-22T00:02:44.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 client.15936 shutdown 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 msgr2=0x56288a916d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe4240057c0 tx=0x7fe42402cbb0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 shutdown_connections 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a914650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a914110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a927200 0x56288a9295f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 --2- 172.21.15.38:0/3970416929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e020 0x56288a916d80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 >> 172.21.15.38:0/3970416929 conn(0x56288a908750 msgr2=0x56288a90a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe416ff5640 0 client.15936 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 shutdown_connections 2024-04-22T00:02:44.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 1 -- 172.21.15.38:0/3970416929 wait complete. 2024-04-22T00:02:44.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.566+0000 7fe43c031740 20 client.15936 trim_cache size 0 max 16384 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 Processor -- start 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 -- start start 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a923170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a916840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341b0c10 con 0x56288a90e940 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341b2750 con 0x56288a917520 2024-04-22T00:02:44.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe43c031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341b12e0 con 0x56288a90e020 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe4327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.579+0000 7fe4327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44964/0 (socket says 172.21.15.38:44964) 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 -- 172.21.15.38:0/3904974608 learned_addr learned my addr 172.21.15.38:0/3904974608 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 -- 172.21.15.38:0/3904974608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 msgr2=0x56288a923170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a916840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a923170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 -- 172.21.15.38:0/3904974608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 msgr2=0x56288a916840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a916840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 -- 172.21.15.38:0/3904974608 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56288a6507d0 con 0x56288a90e940 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4337fe640 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a916840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:44.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe4327fc640 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe4280985a0 tx=0x7fe428020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba240552bd25cf1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe418ff9640 1 -- 172.21.15.38:0/3904974608 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280a4840 con 0x56288a90e940 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe418ff9640 1 -- 172.21.15.38:0/3904974608 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42809bbd0 con 0x56288a90e940 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.580+0000 7fe418ff9640 1 -- 172.21.15.38:0/3904974608 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280a43b0 con 0x56288a90e940 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- 172.21.15.38:0/3904974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 msgr2=0x56288a90f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe4280985a0 tx=0x7fe428020000 comp rx=0 tx=0).stop 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- 172.21.15.38:0/3904974608 shutdown_connections 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a917520 0x56288a916840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a90e940 0x56288a90f570 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- 172.21.15.38:0/3904974608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e020 0x56288a923170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- 172.21.15.38:0/3904974608 >> 172.21.15.38:0/3904974608 conn(0x56288a908750 msgr2=0x56288a90ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- 172.21.15.38:0/3904974608 shutdown_connections 2024-04-22T00:02:44.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- 172.21.15.38:0/3904974608 wait complete. 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- start start 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 Processor -- start 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 0x56288a91f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a91ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341b2080 con 0x56288a917520 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341b19b0 con 0x56288a90e020 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341aebf0 con 0x56288a90e940 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe43c031740 10 client.0 _ll_register_callbacks cb 0x7fe424005dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe4337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe4337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44974/0 (socket says 172.21.15.38:44974) 2024-04-22T00:02:44.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.581+0000 7fe4337fe640 1 -- 172.21.15.38:0/1137318147 learned_addr learned my addr 172.21.15.38:0/1137318147 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4327fc640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a91ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 0x56288a91f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 -- 172.21.15.38:0/1137318147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 msgr2=0x56288a91ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a91ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 -- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 msgr2=0x56288a91f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 0x56288a91f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 -- 172.21.15.38:0/1137318147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56288a91ba40 con 0x56288a917520 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 0x56288a91f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4327fc640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a91ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe4337fe640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe42c01f900 tx=0x7fe42c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64f1236e930d537 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe416ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c011dd0 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe42c028440 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42809bbd0 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe42c031050 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe43c031740 10 client.15942 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56288a98d3a0 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.582+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42809bbd0 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe42c026020 con 0x56288a917520 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 10 client.15942 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 20 client.15942 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:44.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a98db50 0x56288a98ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 4 client.15942 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56288a9149e0 con 0x56288a98db50 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a98db50 0x56288a98ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe416ff5640 10 client.15942 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.583+0000 7fe432ffd640 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a98db50 0x56288a98ff70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4138d48cef329f9c server_cookie=48507c6616ed7842 in_seq=0 out_seq=0 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe416ff5640 1 -- 172.21.15.38:0/1137318147 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56288a9149e0 con 0x56288a98db50 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe416ff5640 10 client.15942 handle_command_reply: tid=0 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 client.15942 shutdown 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 msgr2=0x56288a91b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe42c01f900 tx=0x7fe42c004ab0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 shutdown_connections 2024-04-22T00:02:44.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56288a98db50 0x56288a98ff70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56288a917520 0x56288a91b240 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56288a90e940 0x56288a91ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe416ff5640 0 client.15942 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 --2- 172.21.15.38:0/1137318147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56288a90e020 0x56288a91f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 >> 172.21.15.38:0/1137318147 conn(0x56288a908750 msgr2=0x56288a90f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 shutdown_connections 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 1 -- 172.21.15.38:0/1137318147 wait complete. 2024-04-22T00:02:44.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43c031740 20 client.15942 trim_cache size 0 max 16384 2024-04-22T00:02:44.590 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 msgr2=0x7fe41c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 0x7fe41c0a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0092f0 tx=0x7fe42c004a40 comp rx=0 tx=0).stop 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 msgr2=0x7fe43410e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe42800c920 tx=0x7fe42800cde0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 shutdown_connections 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe41c0a2e20 0x7fe41c0a52e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4341066b0 0x7fe43410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe434100fd0 0x7fe43410e4b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 --2- 172.21.15.38:0/3473705520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100610 0x7fe43410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:44.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.584+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 >> 172.21.15.38:0/3473705520 conn(0x7fe434077740 msgr2=0x7fe4340ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:44.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.585+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 shutdown_connections 2024-04-22T00:02:44.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:44.585+0000 7fe43944d640 1 -- 172.21.15.38:0/3473705520 wait complete. 2024-04-22T00:02:45.593 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:02:45.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:45.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3275084486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 msgr2=0x7f9b04100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3275084486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b04100a90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009930 tx=0x7f9af802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3275084486 shutdown_connections 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3275084486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 0x7f9b0410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3275084486 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b04100a90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3275084486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 0x7f9b04106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3275084486 >> 172.21.15.38:0/3275084486 conn(0x7f9b04077740 msgr2=0x7f9b04077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3275084486 shutdown_connections 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.717+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3275084486 wait complete. 2024-04-22T00:02:45.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 Processor -- start 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 -- start start 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 0x7f9b0410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 0x7f9b0410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b04111b80 con 0x7f9b04100610 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b04111a00 con 0x7f9b04100fd0 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b08aa4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b04111d00 con 0x7f9b041066b0 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44990/0 (socket says 172.21.15.38:44990) 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 -- 172.21.15.38:0/3659165187 learned_addr learned my addr 172.21.15.38:0/3659165187 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 0x7f9b0410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 0x7f9b0410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 -- 172.21.15.38:0/3659165187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 msgr2=0x7f9b0410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 0x7f9b0410e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 -- 172.21.15.38:0/3659165187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 msgr2=0x7f9b0410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 0x7f9b0410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.718+0000 7f9b0259c640 1 -- 172.21.15.38:0/3659165187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b041a6400 con 0x7f9b04100610 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c970 tx=0x7f9af400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9af37fe640 1 -- 172.21.15.38:0/3659165187 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af4007c10 con 0x7f9b04100610 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b041a66f0 con 0x7f9b04100610 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9af37fe640 1 -- 172.21.15.38:0/3659165187 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af4007db0 con 0x7f9b04100610 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9af37fe640 1 -- 172.21.15.38:0/3659165187 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af400f8a0 con 0x7f9b04100610 2024-04-22T00:02:45.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.719+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b041a6c40 con 0x7f9b04100610 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.721+0000 7f9af37fe640 1 -- 172.21.15.38:0/3659165187 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f9af400fa60 con 0x7f9b04100610 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.721+0000 7f9b0b76a740 1 Processor -- start 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 -- start start 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dc0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9af37fe640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 0x7f9aec0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0dcc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5569b0d24170 con 0x5569b0dc0a40 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5569b0ccc7c0 con 0x5569b0dc9db0 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569b0d20130 con 0x5569b0dc1720 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 0x7f9aec0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45006/0 (socket says 172.21.15.38:45006) 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 -- 172.21.15.38:0/3349523274 learned_addr learned my addr 172.21.15.38:0/3349523274 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9af37fe640 1 -- 172.21.15.38:0/3659165187 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9af40cfd30 con 0x7f9b04100610 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dc0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0dcc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 -- 172.21.15.38:0/3349523274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 msgr2=0x5569b0dc0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dc0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 -- 172.21.15.38:0/3349523274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 msgr2=0x5569b0dcc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0dcc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.722+0000 7f9b01d9b640 1 -- 172.21.15.38:0/3349523274 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569b0cdde60 con 0x5569b0dc0a40 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.723+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 0x7f9aec0a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9afc005fd0 tx=0x7f9afc0149d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.723+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9af8036040 tx=0x7f9af8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832c83a7323ed55e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.723+0000 7f9af0ff9640 1 -- 172.21.15.38:0/3349523274 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8043020 con 0x5569b0dc0a40 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.723+0000 7f9af0ff9640 1 -- 172.21.15.38:0/3349523274 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af802c880 con 0x5569b0dc0a40 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.723+0000 7f9af0ff9640 1 -- 172.21.15.38:0/3349523274 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8005460 con 0x5569b0dc0a40 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3349523274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 msgr2=0x5569b0dc0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9af8036040 tx=0x7f9af8005dd0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3349523274 shutdown_connections 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0dcc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc0ec0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3349523274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dc0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3349523274 >> 172.21.15.38:0/3349523274 conn(0x5569b0d874f0 msgr2=0x5569b0dbb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3349523274 shutdown_connections 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3349523274 wait complete. 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- start start 2024-04-22T00:02:45.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 Processor -- start 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc1720 0x5569b0dc7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 0x5569b0dc77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5569b0dccbc0 con 0x5569b0dc0a40 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5569b0cdfd40 con 0x5569b0dc1720 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569b0dccd40 con 0x5569b0dc9db0 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0b76a740 10 client.0 _ll_register_callbacks cb 0x5569b0d1b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45012/0 (socket says 172.21.15.38:45012) 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.724+0000 7f9b0259c640 1 -- 172.21.15.38:0/3351894288 learned_addr learned my addr 172.21.15.38:0/3351894288 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 0x5569b0dc77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 -- 172.21.15.38:0/3351894288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 msgr2=0x5569b0dc77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 0x5569b0dc77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 -- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc1720 msgr2=0x5569b0dc7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc1720 0x5569b0dc7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 -- 172.21.15.38:0/3351894288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569b0b007e0 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 0x5569b0dc77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9af40981b0 tx=0x7f9af409b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb11008f46376ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af409bd50 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9af4018af0 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9af802c880 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9af40a0100 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0b76a740 10 client.15954 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5569b0b60f70 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.725+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af802c880 con 0x5569b0dc0a40 2024-04-22T00:02:45.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9af409b1a0 con 0x5569b0dc0a40 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 10 client.15954 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 20 client.15954 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0dd9e80 0x5569b0ddc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 4 client.15954 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569b0cf8450 con 0x5569b0dd9e80 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0dd9e80 0x5569b0ddc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0dd9e80 0x5569b0ddc2f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b314708f39a7dec server_cookie=44cb762c804a3b87 in_seq=0 out_seq=0 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.726+0000 7f9ae9ffb640 10 client.15954 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3351894288 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569b0cf8450 con 0x5569b0dd9e80 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9ae9ffb640 10 client.15954 handle_command_reply: tid=0 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 client.15954 shutdown 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 msgr2=0x5569b0dc6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9af40981b0 tx=0x7f9af409b9c0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 shutdown_connections 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc9db0 0x5569b0dc77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc1720 0x5569b0dc7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0dd9e80 0x5569b0ddc2f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3351894288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc0a40 0x5569b0dc6d50 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9ae9ffb640 0 client.15954 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 >> 172.21.15.38:0/3351894288 conn(0x5569b0d874f0 msgr2=0x5569b0dbd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 shutdown_connections 2024-04-22T00:02:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3351894288 wait complete. 2024-04-22T00:02:45.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.727+0000 7f9b0b76a740 20 client.15954 trim_cache size 0 max 16384 2024-04-22T00:02:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 Processor -- start 2024-04-22T00:02:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 -- start start 2024-04-22T00:02:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 0x5569b0dc9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0de1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041b0be0 con 0x5569b0dc1720 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041b2720 con 0x5569b0dc9db0 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041b12b0 con 0x5569b0dc0a40 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45022/0 (socket says 172.21.15.38:45022) 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 -- 172.21.15.38:0/340793288 learned_addr learned my addr 172.21.15.38:0/340793288 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0de1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0259c640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 0x5569b0dc9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 -- 172.21.15.38:0/340793288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 msgr2=0x5569b0dc9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 0x5569b0dc9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 -- 172.21.15.38:0/340793288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 msgr2=0x5569b0de1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0de1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 -- 172.21.15.38:0/340793288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569b09f8170 con 0x5569b0dc1720 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0de1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b0259c640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 0x5569b0dc9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9af802c740 tx=0x7f9af802c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc562830879ee31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.746+0000 7f9aebfff640 1 -- 172.21.15.38:0/340793288 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af804d070 con 0x5569b0dc1720 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9aebfff640 1 -- 172.21.15.38:0/340793288 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af802c880 con 0x5569b0dc1720 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9aebfff640 1 -- 172.21.15.38:0/340793288 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8043040 con 0x5569b0dc1720 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- 172.21.15.38:0/340793288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 msgr2=0x5569b0dd5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9af802c740 tx=0x7f9af802c6d0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- 172.21.15.38:0/340793288 shutdown_connections 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc9db0 0x5569b0de1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc1720 0x5569b0dd5af0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/340793288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc0a40 0x5569b0dc9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- 172.21.15.38:0/340793288 >> 172.21.15.38:0/340793288 conn(0x5569b0d874f0 msgr2=0x5569b0dba4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- 172.21.15.38:0/340793288 shutdown_connections 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- 172.21.15.38:0/340793288 wait complete. 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.747+0000 7f9b0b76a740 1 -- start start 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 Processor -- start 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc0a40 0x5569b0dd2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dd2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041b2050 con 0x5569b0dc9db0 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041b1980 con 0x5569b0dc0a40 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b041aebc0 con 0x5569b0dc1720 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0b76a740 10 client.0 _ll_register_callbacks cb 0x7f9af400a5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:45.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45026/0 (socket says 172.21.15.38:45026) 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 -- 172.21.15.38:0/3061322626 learned_addr learned my addr 172.21.15.38:0/3061322626 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b01d9b640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dd2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc0a40 0x5569b0dd2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 -- 172.21.15.38:0/3061322626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 msgr2=0x5569b0dd2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dd2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 -- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc0a40 msgr2=0x5569b0dd2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc0a40 0x5569b0dd2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 -- 172.21.15.38:0/3061322626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569b0dcf8c0 con 0x5569b0dc9db0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9b02d9d640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9afc012460 tx=0x7f9afc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f736421c3d3006 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.748+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc01e770 con 0x5569b0dc9db0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9afc01e910 con 0x5569b0dc9db0 2024-04-22T00:02:45.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9af802c880 con 0x5569b0dc9db0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9afc036050 con 0x5569b0dc9db0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 10 client.15960 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5569b0e40fd0 con 0x5569b0dc9db0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af802c880 con 0x5569b0dc9db0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9afc01e040 con 0x5569b0dc9db0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 10 client.15960 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 20 client.15960 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0e417a0 0x5569b0e43bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 4 client.15960 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5569b0d316b0 con 0x5569b0e417a0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.749+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0e417a0 0x5569b0e43bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.750+0000 7f9ae9ffb640 10 client.15960 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.750+0000 7f9b0259c640 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0e417a0 0x5569b0e43bc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8605a0111e8b4cee server_cookie=63173bd388563386 in_seq=0 out_seq=0 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9ae9ffb640 1 -- 172.21.15.38:0/3061322626 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5569b0d316b0 con 0x5569b0e417a0 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9ae9ffb640 10 client.15960 handle_command_reply: tid=0 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 client.15960 shutdown 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 msgr2=0x5569b0dcf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9afc012460 tx=0x7f9afc004ab0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 shutdown_connections 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569b0e417a0 0x5569b0e43bc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569b0dc9db0 0x5569b0dcf0c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569b0dc1720 0x5569b0dd2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9ae9ffb640 0 client.15960 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 --2- 172.21.15.38:0/3061322626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569b0dc0a40 0x5569b0dd2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 >> 172.21.15.38:0/3061322626 conn(0x5569b0d874f0 msgr2=0x5569b0dc6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 shutdown_connections 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 1 -- 172.21.15.38:0/3061322626 wait complete. 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.751+0000 7f9b0b76a740 20 client.15960 trim_cache size 0 max 16384 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "93bf34e8-1e81-40ec-b4b1-aed126082cf7", 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:02:45.751 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 msgr2=0x7f9aec0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 0x7f9aec0a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9afc005fd0 tx=0x7f9afc0149d0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 msgr2=0x7f9b0410df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c970 tx=0x7f9af400ce30 comp rx=0 tx=0).stop 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 shutdown_connections 2024-04-22T00:02:45.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9aec0a2e20 0x7f9aec0a52e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b041066b0 0x7f9b0410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b04100fd0 0x7f9b0410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 --2- 172.21.15.38:0/3659165187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b04100610 0x7f9b0410df70 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 >> 172.21.15.38:0/3659165187 conn(0x7f9b04077740 msgr2=0x7f9b040feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 shutdown_connections 2024-04-22T00:02:45.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.752+0000 7f9b08aa4640 1 -- 172.21.15.38:0/3659165187 wait complete. 2024-04-22T00:02:45.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:02:45.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 -- 172.21.15.38:0/940010491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c107720 msgr2=0x7fe88c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 --2- 172.21.15.38:0/940010491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c107720 0x7fe88c107b00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009cd0 tx=0x7fe87c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 -- 172.21.15.38:0/940010491 shutdown_connections 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 --2- 172.21.15.38:0/940010491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 0x7fe88c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 --2- 172.21.15.38:0/940010491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c069d40 0x7fe88c06a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 --2- 172.21.15.38:0/940010491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c107720 0x7fe88c107b00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 -- 172.21.15.38:0/940010491 >> 172.21.15.38:0/940010491 conn(0x7fe88c076440 msgr2=0x7fe88c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 -- 172.21.15.38:0/940010491 shutdown_connections 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.868+0000 7fe8909a1640 1 -- 172.21.15.38:0/940010491 wait complete. 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 Processor -- start 2024-04-22T00:02:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 -- start start 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 0x7fe88c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c107720 0x7fe88c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c111af0 con 0x7fe88c069d40 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c111970 con 0x7fe88c06a700 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe8909a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c111c70 con 0x7fe88c107720 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45052/0 (socket says 172.21.15.38:45052) 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 -- 172.21.15.38:0/778920581 learned_addr learned my addr 172.21.15.38:0/778920581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c107720 0x7fe88c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 -- 172.21.15.38:0/778920581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c107720 msgr2=0x7fe88c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe889d9b640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 0x7fe88c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c107720 0x7fe88c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 -- 172.21.15.38:0/778920581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 msgr2=0x7fe88c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 0x7fe88c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.869+0000 7fe88a59c640 1 -- 172.21.15.38:0/778920581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c19d020 con 0x7fe88c069d40 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe88a59c640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe87c004af0 tx=0x7fe87c032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe8837fe640 1 -- 172.21.15.38:0/778920581 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c041020 con 0x7fe88c069d40 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c19d2b0 con 0x7fe88c069d40 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe8837fe640 1 -- 172.21.15.38:0/778920581 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c034cd0 con 0x7fe88c069d40 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c1a4ba0 con 0x7fe88c069d40 2024-04-22T00:02:45.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.870+0000 7fe8837fe640 1 -- 172.21.15.38:0/778920581 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c039040 con 0x7fe88c069d40 2024-04-22T00:02:45.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.872+0000 7fe8837fe640 1 -- 172.21.15.38:0/778920581 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fe87c044020 con 0x7fe88c069d40 2024-04-22T00:02:45.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.872+0000 7fe89358d740 1 Processor -- start 2024-04-22T00:02:45.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.872+0000 7fe89358d740 1 -- start start 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e418f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e419a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e5e40f3560 con 0x55e5e418f850 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e5e409b730 con 0x55e5e4198430 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe89358d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5e40ef520 con 0x55e5e418ef30 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe8837fe640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 0x7fe8780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45058/0 (socket says 172.21.15.38:45058) 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 -- 172.21.15.38:0/841814536 learned_addr learned my addr 172.21.15.38:0/841814536 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 0x7fe8780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe8837fe640 1 -- 172.21.15.38:0/778920581 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe87c0f1780 con 0x7fe88c069d40 2024-04-22T00:02:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e418f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe889d9b640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 -- 172.21.15.38:0/841814536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 msgr2=0x55e5e418f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e418f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 -- 172.21.15.38:0/841814536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 msgr2=0x55e5e419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 -- 172.21.15.38:0/841814536 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e3ed17d0 con 0x55e5e418f850 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe889d9b640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 0x7fe8780a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8840127b0 tx=0x7fe8840099d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.873+0000 7fe88a59c640 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0bd7b0 tx=0x7fe87c090890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aebcc9811dca1db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe880ff9640 1 -- 172.21.15.38:0/841814536 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0bd360 con 0x55e5e418f850 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe880ff9640 1 -- 172.21.15.38:0/841814536 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c0bdf20 con 0x55e5e418f850 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe880ff9640 1 -- 172.21.15.38:0/841814536 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0bd180 con 0x55e5e418f850 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 -- 172.21.15.38:0/841814536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 msgr2=0x55e5e4197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0bd7b0 tx=0x7fe87c090890 comp rx=0 tx=0).stop 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 -- 172.21.15.38:0/841814536 shutdown_connections 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4197ef0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 --2- 172.21.15.38:0/841814536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e418f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 -- 172.21.15.38:0/841814536 >> 172.21.15.38:0/841814536 conn(0x55e5e4189660 msgr2=0x55e5e4189a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 -- 172.21.15.38:0/841814536 shutdown_connections 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.874+0000 7fe89358d740 1 -- 172.21.15.38:0/841814536 wait complete. 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 -- start start 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 Processor -- start 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418ef30 0x55e5e41954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e4198430 0x55e5e4195f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e5e40f3560 con 0x55e5e418f850 2024-04-22T00:02:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e5e40aee00 con 0x55e5e418ef30 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5e419afe0 con 0x55e5e4198430 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe89358d740 10 client.0 _ll_register_callbacks cb 0x55e5e40e9eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe889d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe889d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45068/0 (socket says 172.21.15.38:45068) 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe889d9b640 1 -- 172.21.15.38:0/3793784375 learned_addr learned my addr 172.21.15.38:0/3793784375 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e4198430 0x55e5e4195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418ef30 0x55e5e41954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe889d9b640 1 -- 172.21.15.38:0/3793784375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e4198430 msgr2=0x55e5e4195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.875+0000 7fe889d9b640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e4198430 0x55e5e4195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe889d9b640 1 -- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418ef30 msgr2=0x55e5e41954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe889d9b640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418ef30 0x55e5e41954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe889d9b640 1 -- 172.21.15.38:0/3793784375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e3e073f0 con 0x55e5e418f850 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe889d9b640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe87400f6c0 tx=0x7fe87400fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d89c8440b515e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874016070 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87c0bdf20 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe87400bb00 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe874025050 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 10 client.15972 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e5e3dc70f0 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c0bdf20 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe87401a400 con 0x55e5e418f850 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 10 client.15972 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 20 client.15972 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.876+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e41a84a0 0x55e5e41aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.877+0000 7fe89358d740 4 client.15972 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.877+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5e4100aa0 con 0x55e5e41a84a0 2024-04-22T00:02:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.877+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e41a84a0 0x55e5e41aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.877+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e41a84a0 0x55e5e41aa8b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=529403e91ccab7a7 server_cookie=6e5331677655f7d4 in_seq=0 out_seq=0 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.877+0000 7fe871ffb640 10 client.15972 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe871ffb640 1 -- 172.21.15.38:0/3793784375 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5e4100aa0 con 0x55e5e41a84a0 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe871ffb640 10 client.15972 handle_command_reply: tid=0 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 client.15972 shutdown 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 msgr2=0x55e5e4195a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe87400f6c0 tx=0x7fe87400fb80 comp rx=0 tx=0).stop 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 shutdown_connections 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e4198430 0x55e5e4195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e41a84a0 0x55e5e41aa8b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418f850 0x55e5e4195a30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 --2- 172.21.15.38:0/3793784375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418ef30 0x55e5e41954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe871ffb640 0 client.15972 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 >> 172.21.15.38:0/3793784375 conn(0x55e5e4189660 msgr2=0x55e5e418b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 shutdown_connections 2024-04-22T00:02:45.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 1 -- 172.21.15.38:0/3793784375 wait complete. 2024-04-22T00:02:45.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.878+0000 7fe89358d740 20 client.15972 trim_cache size 0 max 16384 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 Processor -- start 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 -- start start 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 0x55e5e41a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e418b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1aeb10 con 0x55e5e418ef30 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1b0650 con 0x55e5e4198430 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe89358d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1af1e0 con 0x55e5e418f850 2024-04-22T00:02:45.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45074/0 (socket says 172.21.15.38:45074) 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 -- 172.21.15.38:0/3123095216 learned_addr learned my addr 172.21.15.38:0/3123095216 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 -- 172.21.15.38:0/3123095216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 msgr2=0x55e5e41a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe889d9b640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 0x55e5e41a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e418b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 0x55e5e41a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 -- 172.21.15.38:0/3123095216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 msgr2=0x55e5e418b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e418b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 -- 172.21.15.38:0/3123095216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5e4184f30 con 0x55e5e418ef30 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e418b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe889d9b640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 0x55e5e41a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.896+0000 7fe88a59c640 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0eebe0 tx=0x7fe87c0e83b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8ffa858ed272b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe873fff640 1 -- 172.21.15.38:0/3123095216 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0348f0 con 0x55e5e418ef30 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe873fff640 1 -- 172.21.15.38:0/3123095216 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c0bdf20 con 0x55e5e418ef30 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe873fff640 1 -- 172.21.15.38:0/3123095216 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c003970 con 0x55e5e418ef30 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 -- 172.21.15.38:0/3123095216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 msgr2=0x55e5e4197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0eebe0 tx=0x7fe87c0e83b0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 -- 172.21.15.38:0/3123095216 shutdown_connections 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e4198430 0x55e5e418b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418f850 0x55e5e41a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 --2- 172.21.15.38:0/3123095216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e418ef30 0x55e5e4197c20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 -- 172.21.15.38:0/3123095216 >> 172.21.15.38:0/3123095216 conn(0x55e5e4189660 msgr2=0x55e5e418aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 -- 172.21.15.38:0/3123095216 shutdown_connections 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.897+0000 7fe89358d740 1 -- 172.21.15.38:0/3123095216 wait complete. 2024-04-22T00:02:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 -- start start 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 Processor -- start 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e4192380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418f850 0x55e5e41928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1aff80 con 0x55e5e4198430 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1af8b0 con 0x55e5e418f850 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1acaf0 con 0x55e5e418ef30 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe89358d740 10 client.0 _ll_register_callbacks cb 0x7fe874005f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45088/0 (socket says 172.21.15.38:45088) 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 -- 172.21.15.38:0/1427284054 learned_addr learned my addr 172.21.15.38:0/1427284054 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe889d9b640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418f850 0x55e5e41928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 -- 172.21.15.38:0/1427284054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 msgr2=0x55e5e4192380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88a59c640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e4192380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e4192380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 -- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418f850 msgr2=0x55e5e41928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418f850 0x55e5e41928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 -- 172.21.15.38:0/1427284054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5e420dc30 con 0x55e5e4198430 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88a59c640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e4192380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe884010c00 tx=0x7fe88401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c5704fc78c75b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.898+0000 7fe871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88402f060 con 0x55e5e4198430 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe88401b040 con 0x55e5e4198430 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87c0bdf20 con 0x55e5e4198430 2024-04-22T00:02:45.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe884036050 con 0x55e5e4198430 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 10 client.15978 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e5e420ee50 con 0x55e5e4198430 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c0bdf20 con 0x55e5e4198430 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe88401f050 con 0x55e5e4198430 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 10 client.15978 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 20 client.15978 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e420f630 0x55e5e4211a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 4 client.15978 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.899+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e5e40fadf0 con 0x55e5e420f630 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe88a59c640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e420f630 0x55e5e4211a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe88a59c640 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e420f630 0x55e5e4211a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b9e819d38268504 server_cookie=cfdacfcf0d9a6488 in_seq=0 out_seq=0 2024-04-22T00:02:45.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe871ffb640 10 client.15978 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe871ffb640 1 -- 172.21.15.38:0/1427284054 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e5e40fadf0 con 0x55e5e420f630 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe871ffb640 10 client.15978 handle_command_reply: tid=0 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 client.15978 shutdown 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 msgr2=0x55e5e4192e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe884010c00 tx=0x7fe88401c000 comp rx=0 tx=0).stop 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 shutdown_connections 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e5e420f630 0x55e5e4211a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5e4198430 0x55e5e4192e00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe871ffb640 0 client.15978 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5e418f850 0x55e5e41928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 --2- 172.21.15.38:0/1427284054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5e418ef30 0x55e5e4192380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 >> 172.21.15.38:0/1427284054 conn(0x55e5e4189660 msgr2=0x55e5e41886e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 shutdown_connections 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 1 -- 172.21.15.38:0/1427284054 wait complete. 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.900+0000 7fe89358d740 20 client.15978 trim_cache size 0 max 16384 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:02:45.901 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:45.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 msgr2=0x7fe8780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 0x7fe8780a52e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe8840127b0 tx=0x7fe8840099d0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 msgr2=0x7fe88c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:45.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe87c004af0 tx=0x7fe87c032ec0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 shutdown_connections 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2e20 0x7fe8780a52e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c107720 0x7fe88c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c06a700 0x7fe88c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 --2- 172.21.15.38:0/778920581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c069d40 0x7fe88c198030 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 >> 172.21.15.38:0/778920581 conn(0x7fe88c076440 msgr2=0x7fe88c0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 shutdown_connections 2024-04-22T00:02:45.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:45.901+0000 7fe8909a1640 1 -- 172.21.15.38:0/778920581 wait complete. 2024-04-22T00:02:45.904 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:02:45.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 -- 172.21.15.38:0/2197614599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100560 msgr2=0x7f33dc100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 --2- 172.21.15.38:0/2197614599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100560 0x7f33dc100940 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009900 tx=0x7f33cc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 -- 172.21.15.38:0/2197614599 shutdown_connections 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 --2- 172.21.15.38:0/2197614599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33dc10e110 0x7f33dc110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 --2- 172.21.15.38:0/2197614599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100e80 0x7f33dc10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 --2- 172.21.15.38:0/2197614599 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100560 0x7f33dc100940 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 -- 172.21.15.38:0/2197614599 >> 172.21.15.38:0/2197614599 conn(0x7f33dc077740 msgr2=0x7f33dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 -- 172.21.15.38:0/2197614599 shutdown_connections 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.016+0000 7f33e29c9640 1 -- 172.21.15.38:0/2197614599 wait complete. 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 Processor -- start 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 -- start start 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100560 0x7f33dc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33dc10e110 0x7f33dc19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113d90 con 0x7f33dc100e80 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113c10 con 0x7f33dc10e110 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33e29c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc113f10 con 0x7f33dc100560 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33db7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33db7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45104/0 (socket says 172.21.15.38:45104) 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.017+0000 7f33db7fe640 1 -- 172.21.15.38:0/1345813188 learned_addr learned my addr 172.21.15.38:0/1345813188 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33dbfff640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100560 0x7f33dc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 -- 172.21.15.38:0/1345813188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100560 msgr2=0x7f33dc19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100560 0x7f33dc19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 -- 172.21.15.38:0/1345813188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33dc10e110 msgr2=0x7f33dc19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33dc10e110 0x7f33dc19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 -- 172.21.15.38:0/1345813188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc19f220 con 0x7f33dc100e80 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33db7fe640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d000cd80 con 0x7f33dc100e80 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d0010910 con 0x7f33dc100e80 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc19f510 con 0x7f33dc100e80 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0018990 con 0x7f33dc100e80 2024-04-22T00:02:46.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.018+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc1a6da0 con 0x7f33dc100e80 2024-04-22T00:02:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.019+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc19b580 con 0x7f33dc100e80 2024-04-22T00:02:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.020+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f33d0010ac0 con 0x7f33dc100e80 2024-04-22T00:02:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.021+0000 7f33d97fa640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 0x7f33c40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.021+0000 7f33dbfff640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 0x7f33c40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.021+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f33d00cfe90 con 0x7f33dc100e80 2024-04-22T00:02:46.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.022+0000 7f33dbfff640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 0x7f33c40a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33cc006010 tx=0x7f33cc02df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.026+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f33d009c520 con 0x7f33dc100e80 2024-04-22T00:02:46.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:02:46.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:02:46.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:02:46.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:45 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:02:46.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.252+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33dc02d170 con 0x7f33dc100e80 2024-04-22T00:02:46.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.252+0000 7f33d97fa640 1 -- 172.21.15.38:0/1345813188 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f33d009be20 con 0x7f33dc100e80 2024-04-22T00:02:46.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:02:46.245 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:02:03.216530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:02:46.245 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.254+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 msgr2=0x7f33c40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.254+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 0x7f33c40a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33cc006010 tx=0x7f33cc02df90 comp rx=0 tx=0).stop 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.254+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 msgr2=0x7f33dc19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.254+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f33d000e9f0 tx=0x7f33d000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 shutdown_connections 2024-04-22T00:02:46.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f33c40a2e20 0x7f33c40a52e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f33dc10e110 0x7f33dc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33dc100e80 0x7f33dc19a770 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 --2- 172.21.15.38:0/1345813188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc100560 0x7f33dc19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 >> 172.21.15.38:0/1345813188 conn(0x7f33dc077740 msgr2=0x7f33dc10f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 shutdown_connections 2024-04-22T00:02:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.255+0000 7f33e29c9640 1 -- 172.21.15.38:0/1345813188 wait complete. 2024-04-22T00:02:46.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:45 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:02:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:02:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:02:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:02:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:46.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:45 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 -- 172.21.15.38:0/1159134524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 msgr2=0x7f9b8010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/1159134524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b8010b7c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009900 tx=0x7f9b6c02bd60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 -- 172.21.15.38:0/1159134524 shutdown_connections 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/1159134524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b8010bd00 0x7f9b8010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/1159134524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b8010b7c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/1159134524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b801029c0 0x7f9b80102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 -- 172.21.15.38:0/1159134524 >> 172.21.15.38:0/1159134524 conn(0x7f9b80077740 msgr2=0x7f9b80077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 -- 172.21.15.38:0/1159134524 shutdown_connections 2024-04-22T00:02:46.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.377+0000 7f9b86ac6640 1 -- 172.21.15.38:0/1159134524 wait complete. 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 Processor -- start 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 -- start start 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b801029c0 0x7f9b80198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b8010bd00 0x7f9b8019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b80111b50 con 0x7f9b801032e0 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801119d0 con 0x7f9b801029c0 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b86ac6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b80111cd0 con 0x7f9b8010bd00 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45126/0 (socket says 172.21.15.38:45126) 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2068447302 learned_addr learned my addr 172.21.15.38:0/2068447302 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.378+0000 7f9b84862640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b801029c0 0x7f9b80198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b85063640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b8010bd00 0x7f9b8019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2068447302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b8010bd00 msgr2=0x7f9b8019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b8010bd00 0x7f9b8019c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2068447302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b801029c0 msgr2=0x7f9b80198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b801029c0 0x7f9b80198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2068447302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8019d080 con 0x7f9b801032e0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009a20 tx=0x7f9b6c02dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7dffb640 1 -- 172.21.15.38:0/2068447302 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c043020 con 0x7f9b801032e0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7dffb640 1 -- 172.21.15.38:0/2068447302 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b6c0054e0 con 0x7f9b801032e0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8019d310 con 0x7f9b801032e0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b7dffb640 1 -- 172.21.15.38:0/2068447302 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c003bd0 con 0x7f9b801032e0 2024-04-22T00:02:46.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.379+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801a4c00 con 0x7f9b801032e0 2024-04-22T00:02:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.381+0000 7f9b7dffb640 1 -- 172.21.15.38:0/2068447302 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f9b6c046020 con 0x7f9b801032e0 2024-04-22T00:02:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.382+0000 7f9b7dffb640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 0x7f9b700a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.382+0000 7f9b84862640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 0x7f9b700a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.382+0000 7f9b7dffb640 1 -- 172.21.15.38:0/2068447302 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9b6c0f1b10 con 0x7f9b801032e0 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 Processor -- start 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b84862640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 0x7f9b700a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9b740058c0 tx=0x7f9b7400a4c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 -- start start 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef16e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560eef0d2530 con 0x560eef177520 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560eef07a830 con 0x560eef16e020 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b89793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560eef0ce4f0 con 0x560eef16e940 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45130/0 (socket says 172.21.15.38:45130) 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 -- 172.21.15.38:0/2778647327 learned_addr learned my addr 172.21.15.38:0/2778647327 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b84862640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef16e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 -- 172.21.15.38:0/2778647327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 msgr2=0x560eef176fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 -- 172.21.15.38:0/2778647327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 msgr2=0x560eef16e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef16e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b85063640 1 -- 172.21.15.38:0/2778647327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eef09e2c0 con 0x560eef177520 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.383+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b85063640 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009e30 tx=0x7f9b7800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a804263eb2f68c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b6affd640 1 -- 172.21.15.38:0/2778647327 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78011640 con 0x560eef177520 2024-04-22T00:02:46.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b6affd640 1 -- 172.21.15.38:0/2778647327 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b78011cc0 con 0x560eef177520 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b6affd640 1 -- 172.21.15.38:0/2778647327 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7801bca0 con 0x560eef177520 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b89793740 1 -- 172.21.15.38:0/2778647327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 msgr2=0x560eef179910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.384+0000 7f9b89793740 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009e30 tx=0x7f9b7800f730 comp rx=0 tx=0).stop 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- 172.21.15.38:0/2778647327 shutdown_connections 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef179910 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- 172.21.15.38:0/2778647327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef16e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- 172.21.15.38:0/2778647327 >> 172.21.15.38:0/2778647327 conn(0x560eef168750 msgr2=0x560eef168b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- 172.21.15.38:0/2778647327 shutdown_connections 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- 172.21.15.38:0/2778647327 wait complete. 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- start start 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 Processor -- start 2024-04-22T00:02:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef174260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef174800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560eef17a310 con 0x560eef16e020 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560eef08df10 con 0x560eef177520 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560eef17a490 con 0x560eef16e940 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b89793740 10 client.0 _ll_register_callbacks cb 0x560eeede5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.385+0000 7f9b84862640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45132/0 (socket says 172.21.15.38:45132) 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 -- 172.21.15.38:0/1491465406 learned_addr learned my addr 172.21.15.38:0/1491465406 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef174260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b85063640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef174800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 -- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 msgr2=0x560eef174260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef174260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 -- 172.21.15.38:0/1491465406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 msgr2=0x560eef174800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef174800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 -- 172.21.15.38:0/1491465406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eef05fc80 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef174260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b85063640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef174800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b84862640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9b740109e0 tx=0x7f9b740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1d9f799f2bf525 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400fd50 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9b7401e430 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b78011cc0 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b7401e630 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b89793740 10 client.15996 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.386+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560eeede63f0 con 0x560eef16e020 2024-04-22T00:02:46.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b78011cc0 con 0x560eef16e020 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b74026850 con 0x560eef16e020 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 10 client.15996 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 10 client.15996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 20 client.15996 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef187610 0x560eef189a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 4 client.15996 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eef0dfa70 con 0x560eef187610 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef187610 0x560eef189a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.387+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef187610 0x560eef189a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=451ec23738479665 server_cookie=fa475199d7633760 in_seq=0 out_seq=0 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b5ffff640 10 client.15996 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b5ffff640 1 -- 172.21.15.38:0/1491465406 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560eef0dfa70 con 0x560eef187610 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b5ffff640 10 client.15996 handle_command_reply: tid=0 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 client.15996 shutdown 2024-04-22T00:02:46.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 msgr2=0x560eef176ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9b740109e0 tx=0x7f9b740046c0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 shutdown_connections 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef187610 0x560eef189a00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef174800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef174260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 --2- 172.21.15.38:0/1491465406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e020 0x560eef176ed0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 >> 172.21.15.38:0/1491465406 conn(0x560eef168750 msgr2=0x560eef16a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.388+0000 7f9b5ffff640 0 client.15996 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.389+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 shutdown_connections 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.389+0000 7f9b89793740 1 -- 172.21.15.38:0/1491465406 wait complete. 2024-04-22T00:02:46.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.389+0000 7f9b89793740 20 client.15996 trim_cache size 0 max 16384 2024-04-22T00:02:46.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 Processor -- start 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 -- start start 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef1831d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801aeb20 con 0x560eef177520 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801b0660 con 0x560eef16e020 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b89793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801af1f0 con 0x560eef16e940 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b85063640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b85063640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45146/0 (socket says 172.21.15.38:45146) 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.402+0000 7f9b85063640 1 -- 172.21.15.38:0/664208564 learned_addr learned my addr 172.21.15.38:0/664208564 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b84862640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef1831d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 -- 172.21.15.38:0/664208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 msgr2=0x560eef176990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 -- 172.21.15.38:0/664208564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 msgr2=0x560eef1831d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef1831d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 -- 172.21.15.38:0/664208564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eeeeb07d0 con 0x560eef177520 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b84862640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef1831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b85063640 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9b78000f80 tx=0x7f9b78005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabc443591595b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b6a7fc640 1 -- 172.21.15.38:0/664208564 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78003b70 con 0x560eef177520 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b6a7fc640 1 -- 172.21.15.38:0/664208564 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b78003d10 con 0x560eef177520 2024-04-22T00:02:46.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.403+0000 7f9b6a7fc640 1 -- 172.21.15.38:0/664208564 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b780257d0 con 0x560eef177520 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- 172.21.15.38:0/664208564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 msgr2=0x560eef176ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9b78000f80 tx=0x7f9b78005040 comp rx=0 tx=0).stop 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- 172.21.15.38:0/664208564 shutdown_connections 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef177520 0x560eef176ed0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e940 0x560eef176990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 --2- 172.21.15.38:0/664208564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef16e020 0x560eef1831d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- 172.21.15.38:0/664208564 >> 172.21.15.38:0/664208564 conn(0x560eef168750 msgr2=0x560eef167850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- 172.21.15.38:0/664208564 shutdown_connections 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- 172.21.15.38:0/664208564 wait complete. 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.404+0000 7f9b89793740 1 -- start start 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 Processor -- start 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e020 0x560eef17f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef17b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801aff90 con 0x560eef16e940 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801af8c0 con 0x560eef177520 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b801acb00 con 0x560eef16e020 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b89793740 10 client.0 _ll_register_callbacks cb 0x7f9b74014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45162/0 (socket says 172.21.15.38:45162) 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2963767142 learned_addr learned my addr 172.21.15.38:0/2963767142 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b85063640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef17b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b84862640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e020 0x560eef17f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e020 msgr2=0x560eef17f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e020 0x560eef17f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2963767142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 msgr2=0x560eef17b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef17b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b7ffff640 1 -- 172.21.15.38:0/2963767142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eef17b780 con 0x560eef16e940 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.405+0000 7f9b85063640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef17b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b7ffff640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0b9fb0 tx=0x7f9b6c009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b420e77ee8056c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0c67c0 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9b6c003d70 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b78003d10 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b6c0043c0 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 10 client.16002 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560eef1ed8e0 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b78003d10 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b6c0c6330 con 0x560eef16e940 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 10 client.16002 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 10 client.16002 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 20 client.16002 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef1ee000 0x560eef1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 4 client.16002 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.406+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560eef0d9dc0 con 0x560eef1ee000 2024-04-22T00:02:46.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b84862640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef1ee000 0x560eef1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b84862640 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef1ee000 0x560eef1f03f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=468130507d1549e server_cookie=b76c7b56087b7232 in_seq=0 out_seq=0 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b5ffff640 10 client.16002 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b5ffff640 1 -- 172.21.15.38:0/2963767142 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560eef0d9dc0 con 0x560eef1ee000 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b5ffff640 10 client.16002 handle_command_reply: tid=0 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b89793740 1 client.16002 shutdown 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 msgr2=0x560eef17ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.407+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0b9fb0 tx=0x7f9b6c009670 comp rx=0 tx=0).stop 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 shutdown_connections 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560eef1ee000 0x560eef1f03f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560eef177520 0x560eef17b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b5ffff640 0 client.16002 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560eef16e940 0x560eef17ac80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 --2- 172.21.15.38:0/2963767142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560eef16e020 0x560eef17f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 >> 172.21.15.38:0/2963767142 conn(0x560eef168750 msgr2=0x560eef16ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 shutdown_connections 2024-04-22T00:02:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 1 -- 172.21.15.38:0/2963767142 wait complete. 2024-04-22T00:02:46.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b89793740 20 client.16002 trim_cache size 0 max 16384 2024-04-22T00:02:46.407 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 msgr2=0x7f9b700a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 0x7f9b700a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9b740058c0 tx=0x7f9b7400a4c0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 msgr2=0x7f9b801985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009a20 tx=0x7f9b6c02dd30 comp rx=0 tx=0).stop 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 shutdown_connections 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9b700a2ea0 0x7f9b700a5360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b8010bd00 0x7f9b8019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9b801032e0 0x7f9b801985d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 --2- 172.21.15.38:0/2068447302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b801029c0 0x7f9b80198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.408+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 >> 172.21.15.38:0/2068447302 conn(0x7f9b80077740 msgr2=0x7f9b800fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.409+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 shutdown_connections 2024-04-22T00:02:46.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.409+0000 7f9b86ac6640 1 -- 172.21.15.38:0/2068447302 wait complete. 2024-04-22T00:02:46.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:02:46.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 -- 172.21.15.38:0/1529852834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d8100610 msgr2=0x7f08d8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 --2- 172.21.15.38:0/1529852834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d8100610 0x7f08d8100a90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009930 tx=0x7f08c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 -- 172.21.15.38:0/1529852834 shutdown_connections 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 --2- 172.21.15.38:0/1529852834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100fd0 0x7f08d810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 --2- 172.21.15.38:0/1529852834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d8100610 0x7f08d8100a90 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 --2- 172.21.15.38:0/1529852834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d81066b0 0x7f08d8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 -- 172.21.15.38:0/1529852834 >> 172.21.15.38:0/1529852834 conn(0x7f08d8077740 msgr2=0x7f08d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 -- 172.21.15.38:0/1529852834 shutdown_connections 2024-04-22T00:02:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.527+0000 7f08dcec9640 1 -- 172.21.15.38:0/1529852834 wait complete. 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 Processor -- start 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 -- start start 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100610 0x7f08d81022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d8100fd0 0x7f08d81027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81109f0 con 0x7f08d81066b0 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08d8110870 con 0x7f08d8100610 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08dcec9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08d8110b70 con 0x7f08d8100fd0 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45176/0 (socket says 172.21.15.38:45176) 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 -- 172.21.15.38:0/73753272 learned_addr learned my addr 172.21.15.38:0/73753272 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d659c640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100610 0x7f08d81022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d8100fd0 0x7f08d81027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 -- 172.21.15.38:0/73753272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d8100fd0 msgr2=0x7f08d81027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d8100fd0 0x7f08d81027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 -- 172.21.15.38:0/73753272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100610 msgr2=0x7f08d81022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100610 0x7f08d81022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 -- 172.21.15.38:0/73753272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d81a6480 con 0x7f08d81066b0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.528+0000 7f08d6d9d640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f08d000eb10 tx=0x7f08d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.529+0000 7f08cf7fe640 1 -- 172.21.15.38:0/73753272 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0004b20 con 0x7f08d81066b0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.529+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d81a6770 con 0x7f08d81066b0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.529+0000 7f08cf7fe640 1 -- 172.21.15.38:0/73753272 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08d0004cc0 con 0x7f08d81066b0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.529+0000 7f08cf7fe640 1 -- 172.21.15.38:0/73753272 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0010760 con 0x7f08d81066b0 2024-04-22T00:02:46.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.529+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d81a6c90 con 0x7f08d81066b0 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.531+0000 7f08dfb8e740 1 Processor -- start 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.531+0000 7f08cf7fe640 1 -- 172.21.15.38:0/73753272 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f08d0004fe0 con 0x7f08d81066b0 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.531+0000 7f08dfb8e740 1 -- start start 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563ba4b4f530 con 0x563ba4beb020 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ba4af7830 con 0x563ba4beb940 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ba4b4b4f0 con 0x563ba4bf4520 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45184/0 (socket says 172.21.15.38:45184) 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 -- 172.21.15.38:0/632462348 learned_addr learned my addr 172.21.15.38:0/632462348 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d659c640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08cf7fe640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 0x7f08c40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d6d9d640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 0x7f08c40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08cf7fe640 1 -- 172.21.15.38:0/73753272 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08d00d0120 con 0x7f08d81066b0 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 -- 172.21.15.38:0/632462348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 msgr2=0x563ba4bf6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 -- 172.21.15.38:0/632462348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 msgr2=0x563ba4bf3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d5d9b640 1 -- 172.21.15.38:0/632462348 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba4b1b2c0 con 0x563ba4beb020 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.532+0000 7f08d659c640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.533+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005940 tx=0x7f08c80058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f2a271a346722d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.533+0000 7f08d6d9d640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 0x7f08c40a5310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08d00cd410 tx=0x7f08d0091540 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.533+0000 7f08ccff9640 1 -- 172.21.15.38:0/632462348 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c802cc20 con 0x563ba4beb020 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.533+0000 7f08ccff9640 1 -- 172.21.15.38:0/632462348 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c8005040 con 0x563ba4beb020 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.533+0000 7f08ccff9640 1 -- 172.21.15.38:0/632462348 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c80049b0 con 0x563ba4beb020 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- 172.21.15.38:0/632462348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 msgr2=0x563ba4beb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005940 tx=0x7f08c80058b0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- 172.21.15.38:0/632462348 shutdown_connections 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/632462348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb020 0x563ba4beb400 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- 172.21.15.38:0/632462348 >> 172.21.15.38:0/632462348 conn(0x563ba4be5750 msgr2=0x563ba4be5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- 172.21.15.38:0/632462348 shutdown_connections 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- 172.21.15.38:0/632462348 wait complete. 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 -- start start 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 Processor -- start 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.534+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb020 0x563ba4bf3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563ba4bf7250 con 0x563ba4beb940 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ba4b0af10 con 0x563ba4beb020 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ba4bf73d0 con 0x563ba4bf4520 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45198/0 (socket says 172.21.15.38:45198) 2024-04-22T00:02:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 -- 172.21.15.38:0/1033287336 learned_addr learned my addr 172.21.15.38:0/1033287336 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 10 client.0 _ll_register_callbacks cb 0x563ba4862be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb020 0x563ba4bf3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 -- 172.21.15.38:0/1033287336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 msgr2=0x563ba4bf18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 -- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb020 msgr2=0x563ba4bf3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb020 0x563ba4bf3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 -- 172.21.15.38:0/1033287336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba4adcc80 con 0x563ba4beb940 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08d659c640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f08c000de50 tx=0x7f08c0012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb27fd50f0336fc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0005220 con 0x563ba4beb940 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08c00053c0 con 0x563ba4beb940 2024-04-22T00:02:46.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c8005040 con 0x563ba4beb940 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08c0025050 con 0x563ba4beb940 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 10 client.16014 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.535+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ba48633f0 con 0x563ba4beb940 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c8005040 con 0x563ba4beb940 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08c0010a50 con 0x563ba4beb940 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 10 client.16014 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 10 client.16014 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 20 client.16014 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c04560 0x563ba4c069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 4 client.16014 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba4b5ca70 con 0x563ba4c04560 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.536+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c04560 0x563ba4c069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.537+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c04560 0x563ba4c069d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86ed82080a340442 server_cookie=e9c8c98bd33066c3 in_seq=0 out_seq=0 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.537+0000 7f08bdffb640 10 client.16014 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:46.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.537+0000 7f08bdffb640 1 -- 172.21.15.38:0/1033287336 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ba4b5ca70 con 0x563ba4c04560 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.537+0000 7f08bdffb640 10 client.16014 handle_command_reply: tid=0 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.537+0000 7f08dfb8e740 1 client.16014 shutdown 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 msgr2=0x563ba4bf12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f08c000de50 tx=0x7f08c0012820 comp rx=0 tx=0).stop 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 shutdown_connections 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4bf4520 0x563ba4bf18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c04560 0x563ba4c069d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4bf12a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1033287336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb020 0x563ba4bf3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 >> 172.21.15.38:0/1033287336 conn(0x563ba4be5750 msgr2=0x563ba4be74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08bdffb640 0 client.16014 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 shutdown_connections 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1033287336 wait complete. 2024-04-22T00:02:46.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.538+0000 7f08dfb8e740 20 client.16014 trim_cache size 0 max 16384 2024-04-22T00:02:46.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 Processor -- start 2024-04-22T00:02:46.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 -- start start 2024-04-22T00:02:46.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4c00110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4bf4520 0x563ba4c0c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81b0c60 con 0x563ba4beb940 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81b27a0 con 0x563ba4bf4520 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81b1330 con 0x563ba4beb020 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45214/0 (socket says 172.21.15.38:45214) 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 -- 172.21.15.38:0/1894669707 learned_addr learned my addr 172.21.15.38:0/1894669707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4c00110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 -- 172.21.15.38:0/1894669707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 msgr2=0x563ba4c00110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4c00110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 -- 172.21.15.38:0/1894669707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4bf4520 msgr2=0x563ba4c0c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4bf4520 0x563ba4c0c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.552+0000 7f08d659c640 1 -- 172.21.15.38:0/1894669707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba4bf1c40 con 0x563ba4beb940 2024-04-22T00:02:46.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08d659c640 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f08c0010e20 tx=0x7f08c0012c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86998c6c7b407279 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08bffff640 1 -- 172.21.15.38:0/1894669707 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c002d020 con 0x563ba4beb940 2024-04-22T00:02:46.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08bffff640 1 -- 172.21.15.38:0/1894669707 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c0010ac0 con 0x563ba4beb940 2024-04-22T00:02:46.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08bffff640 1 -- 172.21.15.38:0/1894669707 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00052a0 con 0x563ba4beb940 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1894669707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 msgr2=0x563ba4c06570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f08c0010e20 tx=0x7f08c0012c60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1894669707 shutdown_connections 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4bf4520 0x563ba4c0c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4beb940 0x563ba4c06570 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.553+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/1894669707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4c00110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1894669707 >> 172.21.15.38:0/1894669707 conn(0x563ba4be5750 msgr2=0x563ba4befb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1894669707 shutdown_connections 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- 172.21.15.38:0/1894669707 wait complete. 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- start start 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 Processor -- start 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4bfc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81b20d0 con 0x563ba4bf4520 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81b1a00 con 0x563ba4beb940 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81aec40 con 0x563ba4beb020 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08dfb8e740 10 client.0 _ll_register_callbacks cb 0x7f08c000d6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45228/0 (socket says 172.21.15.38:45228) 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08d5d9b640 1 -- 172.21.15.38:0/309427137 learned_addr learned my addr 172.21.15.38:0/309427137 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.554+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4bfc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d659c640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 -- 172.21.15.38:0/309427137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 msgr2=0x563ba4bfc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4bfc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 -- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 msgr2=0x563ba4bf7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 -- 172.21.15.38:0/309427137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba492d7d0 con 0x563ba4bf4520 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d659c640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08d5d9b640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005940 tx=0x7f08c802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149b91991079b804 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c804f4d0 con 0x563ba4bf4520 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c0010ac0 con 0x563ba4bf4520 2024-04-22T00:02:46.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08c804f670 con 0x563ba4bf4520 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08c802ccc0 con 0x563ba4bf4520 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08dfb8e740 10 client.16020 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ba4c6a760 con 0x563ba4bf4520 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.555+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c0010ac0 con 0x563ba4bf4520 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08c8042440 con 0x563ba4bf4520 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 10 client.16020 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 10 client.16020 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 20 client.16020 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c6abb0 0x563ba4c6cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 4 client.16020 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ba4b56dc0 con 0x563ba4c6abb0 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.556+0000 7f08d659c640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c6abb0 0x563ba4c6cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08bdffb640 10 client.16020 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:46.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08d659c640 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c6abb0 0x563ba4c6cfa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6833b35893eda14 server_cookie=39ed57d7b28fab28 in_seq=0 out_seq=0 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08bdffb640 1 -- 172.21.15.38:0/309427137 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ba4b56dc0 con 0x563ba4c6abb0 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08bdffb640 10 client.16020 handle_command_reply: tid=0 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 client.16020 shutdown 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 msgr2=0x563ba4bf8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f08c8005940 tx=0x7f08c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 shutdown_connections 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x563ba4c6abb0 0x563ba4c6cfa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ba4bf4520 0x563ba4bf8120 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ba4beb940 0x563ba4bf7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08bdffb640 0 client.16020 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 --2- 172.21.15.38:0/309427137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ba4beb020 0x563ba4bfc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 >> 172.21.15.38:0/309427137 conn(0x563ba4be5750 msgr2=0x563ba4beffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 shutdown_connections 2024-04-22T00:02:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 1 -- 172.21.15.38:0/309427137 wait complete. 2024-04-22T00:02:46.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.557+0000 7f08dfb8e740 20 client.16020 trim_cache size 0 max 16384 2024-04-22T00:02:46.556 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 msgr2=0x7f08c40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 0x7f08c40a5310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08d00cd410 tx=0x7f08d0091540 comp rx=0 tx=0).stop 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 msgr2=0x7f08d8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f08d000eb10 tx=0x7f08d000c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 shutdown_connections 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08c40a2e50 0x7f08c40a5310 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08d81066b0 0x7f08d8102d30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d8100fd0 0x7f08d81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 --2- 172.21.15.38:0/73753272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08d8100610 0x7f08d81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 >> 172.21.15.38:0/73753272 conn(0x7f08d8077740 msgr2=0x7f08d80ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 shutdown_connections 2024-04-22T00:02:46.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.558+0000 7f08dcec9640 1 -- 172.21.15.38:0/73753272 wait complete. 2024-04-22T00:02:46.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:02:46.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 -- 172.21.15.38:0/756694117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd120100610 msgr2=0x7fd120100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 --2- 172.21.15.38:0/756694117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd120100610 0x7fd120100a90 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd108009930 tx=0x7fd10802bc30 comp rx=0 tx=0).stop 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 -- 172.21.15.38:0/756694117 shutdown_connections 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 --2- 172.21.15.38:0/756694117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100fd0 0x7fd12010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 --2- 172.21.15.38:0/756694117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd120100610 0x7fd120100a90 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 --2- 172.21.15.38:0/756694117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1201066b0 0x7fd120106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 -- 172.21.15.38:0/756694117 >> 172.21.15.38:0/756694117 conn(0x7fd120077740 msgr2=0x7fd120077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 -- 172.21.15.38:0/756694117 shutdown_connections 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.675+0000 7fd1254aa640 1 -- 172.21.15.38:0/756694117 wait complete. 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 Processor -- start 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 -- start start 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 0x7fd120197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120100fd0 0x7fd120198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd120110a30 con 0x7fd1201066b0 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201108b0 con 0x7fd120100610 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd1254aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd120110bb0 con 0x7fd120100fd0 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45246/0 (socket says 172.21.15.38:45246) 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.676+0000 7fd11f7fe640 1 -- 172.21.15.38:0/2841691783 learned_addr learned my addr 172.21.15.38:0/2841691783 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd117fff640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120100fd0 0x7fd120198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11effd640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 0x7fd120197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 -- 172.21.15.38:0/2841691783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120100fd0 msgr2=0x7fd120198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120100fd0 0x7fd120198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 -- 172.21.15.38:0/2841691783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 msgr2=0x7fd120197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 0x7fd120197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 -- 172.21.15.38:0/2841691783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12019cfd0 con 0x7fd1201066b0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11effd640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 0x7fd120197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd11800eb10 tx=0x7fd11800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11cff9640 1 -- 172.21.15.38:0/2841691783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118004b20 con 0x7fd1201066b0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11cff9640 1 -- 172.21.15.38:0/2841691783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd118004cc0 con 0x7fd1201066b0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12019d2c0 con 0x7fd1201066b0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd11cff9640 1 -- 172.21.15.38:0/2841691783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118010760 con 0x7fd1201066b0 2024-04-22T00:02:46.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.677+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1201a4b50 con 0x7fd1201066b0 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.679+0000 7fd11cff9640 1 -- 172.21.15.38:0/2841691783 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fd118005020 con 0x7fd1201066b0 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 Processor -- start 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 -- start start 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd11cff9640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 0x7fd10c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd11cff9640 1 -- 172.21.15.38:0/2841691783 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd1180d00a0 con 0x7fd1201066b0 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd117fff640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 0x7fd10c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb56400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563dffaba530 con 0x563dffb5f520 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563dffa62830 con 0x563dffb56940 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.680+0000 7fd128168740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563dffab64f0 con 0x563dffb56020 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45254/0 (socket says 172.21.15.38:45254) 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 -- 172.21.15.38:0/1327701170 learned_addr learned my addr 172.21.15.38:0/1327701170 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11effd640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb56400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd117fff640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 0x7fd10c0a52e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd108005ab0 tx=0x7fd108005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd117fff640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 -- 172.21.15.38:0/1327701170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 msgr2=0x563dffb56400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb56400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 -- 172.21.15.38:0/1327701170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 msgr2=0x563dffb5efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 -- 172.21.15.38:0/1327701170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dffa862c0 con 0x563dffb5f520 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd117fff640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd1180cd440 tx=0x7fd11809bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b8a73c86f5fa7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd1157fa640 1 -- 172.21.15.38:0/1327701170 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11809c030 con 0x563dffb5f520 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd1157fa640 1 -- 172.21.15.38:0/1327701170 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1180a03e0 con 0x563dffb5f520 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.681+0000 7fd1157fa640 1 -- 172.21.15.38:0/1327701170 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1180a4810 con 0x563dffb5f520 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 -- 172.21.15.38:0/1327701170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 msgr2=0x563dffb61910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd1180cd440 tx=0x7fd11809bb70 comp rx=0 tx=0).stop 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 -- 172.21.15.38:0/1327701170 shutdown_connections 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffb61910 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 --2- 172.21.15.38:0/1327701170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb56400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 -- 172.21.15.38:0/1327701170 >> 172.21.15.38:0/1327701170 conn(0x563dffb50750 msgr2=0x563dffb50b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 -- 172.21.15.38:0/1327701170 shutdown_connections 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.682+0000 7fd128168740 1 -- 172.21.15.38:0/1327701170 wait complete. 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 -- start start 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 Processor -- start 2024-04-22T00:02:46.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb5f520 0x563dffb5c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563dffb62310 con 0x563dffb56020 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563dffa75f10 con 0x563dffb56940 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563dffb62490 con 0x563dffb5f520 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd128168740 10 client.0 _ll_register_callbacks cb 0x563dff7cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd11effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd11effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45264/0 (socket says 172.21.15.38:45264) 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd11effd640 1 -- 172.21.15.38:0/3919934919 learned_addr learned my addr 172.21.15.38:0/3919934919 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd117fff640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb5f520 0x563dffb5c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.683+0000 7fd11effd640 1 -- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb5f520 msgr2=0x563dffb5c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd11effd640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb5f520 0x563dffb5c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd11effd640 1 -- 172.21.15.38:0/3919934919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 msgr2=0x563dffb5c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd11effd640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd11effd640 1 -- 172.21.15.38:0/3919934919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dffa47c80 con 0x563dffb56020 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd11effd640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd110013790 tx=0x7fd110013b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9131d0a7762e6765 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd110020060 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1180a03e0 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd110011dd0 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd110027050 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd128168740 10 client.16032 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563dff7ce3f0 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1180a03e0 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd11001a030 con 0x563dffb56020 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd128168740 10 client.16032 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.684+0000 7fd128168740 10 client.16032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd128168740 20 client.16032 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffb6f520 0x563dffb71950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd128168740 4 client.16032 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dffac7a70 con 0x563dffb6f520 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd117fff640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffb6f520 0x563dffb71950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd1067fc640 10 client.16032 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.685+0000 7fd117fff640 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffb6f520 0x563dffb71950 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2073a9971137ef3c server_cookie=ba25cc00a654476c in_seq=0 out_seq=0 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd1067fc640 1 -- 172.21.15.38:0/3919934919 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563dffac7a70 con 0x563dffb6f520 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd1067fc640 10 client.16032 handle_command_reply: tid=0 2024-04-22T00:02:46.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 client.16032 shutdown 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 msgr2=0x563dffb5eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd110013790 tx=0x7fd110013b90 comp rx=0 tx=0).stop 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 shutdown_connections 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffb6f520 0x563dffb71950 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb5f520 0x563dffb5c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd1067fc640 0 client.16032 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb5c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 --2- 172.21.15.38:0/3919934919 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56020 0x563dffb5eec0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 >> 172.21.15.38:0/3919934919 conn(0x563dffb50750 msgr2=0x563dffb524a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 shutdown_connections 2024-04-22T00:02:46.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 1 -- 172.21.15.38:0/3919934919 wait complete. 2024-04-22T00:02:46.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.686+0000 7fd128168740 20 client.16032 trim_cache size 0 max 16384 2024-04-22T00:02:46.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 Processor -- start 2024-04-22T00:02:46.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 -- start start 2024-04-22T00:02:46.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb5e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb5f520 0x563dffb57570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201aead0 con 0x563dffb56940 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b0610 con 0x563dffb5f520 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd128168740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201af1a0 con 0x563dffb56020 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd117fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd117fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45276/0 (socket says 172.21.15.38:45276) 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd117fff640 1 -- 172.21.15.38:0/928003257 learned_addr learned my addr 172.21.15.38:0/928003257 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.700+0000 7fd11effd640 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb5e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb5f520 0x563dffb57570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 -- 172.21.15.38:0/928003257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 msgr2=0x563dffb5e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb5e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 -- 172.21.15.38:0/928003257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb5f520 msgr2=0x563dffb57570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb5f520 0x563dffb57570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 -- 172.21.15.38:0/928003257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dff8987d0 con 0x563dffb56940 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd117fff640 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd108009900 tx=0x7fd10803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7972f8b9708496f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd114ff9640 1 -- 172.21.15.38:0/928003257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108004410 con 0x563dffb56940 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd114ff9640 1 -- 172.21.15.38:0/928003257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd108004ba0 con 0x563dffb56940 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.701+0000 7fd114ff9640 1 -- 172.21.15.38:0/928003257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1080035f0 con 0x563dffb56940 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- 172.21.15.38:0/928003257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 msgr2=0x563dffb5eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd108009900 tx=0x7fd10803f000 comp rx=0 tx=0).stop 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- 172.21.15.38:0/928003257 shutdown_connections 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb5f520 0x563dffb57570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb56940 0x563dffb5eec0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 --2- 172.21.15.38:0/928003257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb5e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- 172.21.15.38:0/928003257 >> 172.21.15.38:0/928003257 conn(0x563dffb50750 msgr2=0x563dffb56e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- 172.21.15.38:0/928003257 shutdown_connections 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- 172.21.15.38:0/928003257 wait complete. 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 -- start start 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.702+0000 7fd128168740 1 Processor -- start 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb63770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb63cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201aff40 con 0x563dffb5f520 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201af870 con 0x563dffb56940 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201acab0 con 0x563dffb56020 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd128168740 10 client.0 _ll_register_callbacks cb 0x7fd110009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45282/0 (socket says 172.21.15.38:45282) 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 -- 172.21.15.38:0/960938810 learned_addr learned my addr 172.21.15.38:0/960938810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11effd640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb63770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd117fff640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb63cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 -- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 msgr2=0x563dffb63770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb63770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 -- 172.21.15.38:0/960938810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 msgr2=0x563dffb63cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb63cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 -- 172.21.15.38:0/960938810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dffbd4fd0 con 0x563dffb5f520 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd11f7fe640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd11809bd30 tx=0x7fd11809bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9536d2d6afbf5bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd1067fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.703+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11809c030 con 0x563dffb5f520 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd1180a4810 con 0x563dffb5f520 2024-04-22T00:02:46.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd108004ba0 con 0x563dffb5f520 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd1180a4a10 con 0x563dffb5f520 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 10 client.16038 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563dffbd6230 con 0x563dffb5f520 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd108004ba0 con 0x563dffb5f520 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd118096cc0 con 0x563dffb5f520 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 10 client.16038 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 10 client.16038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 20 client.16038 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffbd6510 0x563dffbd8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 4 client.16038 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563dffac1dc0 con 0x563dffbd6510 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.704+0000 7fd11effd640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffbd6510 0x563dffbd8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd1067fc640 10 client.16038 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:46.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd11effd640 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffbd6510 0x563dffbd8900 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c4b542099d447c0 server_cookie=3357c2a83ffb362c in_seq=0 out_seq=0 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd1067fc640 1 -- 172.21.15.38:0/960938810 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563dffac1dc0 con 0x563dffbd6510 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd1067fc640 10 client.16038 handle_command_reply: tid=0 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 client.16038 shutdown 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 msgr2=0x563dffbd4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd11809bd30 tx=0x7fd11809bca0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 shutdown_connections 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563dffbd6510 0x563dffbd8900 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563dffb5f520 0x563dffbd4a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563dffb56940 0x563dffb63cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 --2- 172.21.15.38:0/960938810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dffb56020 0x563dffb63770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 >> 172.21.15.38:0/960938810 conn(0x563dffb50750 msgr2=0x563dffb576f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd1067fc640 0 client.16038 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:02:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 shutdown_connections 2024-04-22T00:02:46.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 1 -- 172.21.15.38:0/960938810 wait complete. 2024-04-22T00:02:46.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.705+0000 7fd128168740 20 client.16038 trim_cache size 0 max 16384 2024-04-22T00:02:46.705 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 msgr2=0x7fd10c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 0x7fd10c0a52e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd108005ab0 tx=0x7fd108005fd0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 msgr2=0x7fd12019c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd11800eb10 tx=0x7fd11800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 shutdown_connections 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd10c0a2e20 0x7fd10c0a52e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1201066b0 0x7fd12019c8a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120100fd0 0x7fd120198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 --2- 172.21.15.38:0/2841691783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120100610 0x7fd120197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 >> 172.21.15.38:0/2841691783 conn(0x7fd120077740 msgr2=0x7fd1200ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 shutdown_connections 2024-04-22T00:02:46.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.706+0000 7fd1254aa640 1 -- 172.21.15.38:0/2841691783 wait complete. 2024-04-22T00:02:46.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:02:46.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 -- 172.21.15.38:0/1793944681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49301032e0 msgr2=0x7f493010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 --2- 172.21.15.38:0/1793944681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49301032e0 0x7f493010b7c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4920009900 tx=0x7f492002bd60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 -- 172.21.15.38:0/1793944681 shutdown_connections 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 --2- 172.21.15.38:0/1793944681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493010bd00 0x7f493010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 --2- 172.21.15.38:0/1793944681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49301032e0 0x7f493010b7c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 --2- 172.21.15.38:0/1793944681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 0x7f4930102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 -- 172.21.15.38:0/1793944681 >> 172.21.15.38:0/1793944681 conn(0x7f4930077740 msgr2=0x7f4930077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 -- 172.21.15.38:0/1793944681 shutdown_connections 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 -- 172.21.15.38:0/1793944681 wait complete. 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.823+0000 7f493744e640 1 Processor -- start 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 -- start start 2024-04-22T00:02:46.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 0x7f4930100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49301032e0 0x7f4930100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111b50 con 0x7f493010bd00 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49301119d0 con 0x7f49301032e0 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f493744e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111cd0 con 0x7f49301029c0 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45300/0 (socket says 172.21.15.38:45300) 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 -- 172.21.15.38:0/806192343 learned_addr learned my addr 172.21.15.38:0/806192343 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49351ea640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 0x7f4930100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49349e9640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49301032e0 0x7f4930100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 -- 172.21.15.38:0/806192343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 msgr2=0x7f4930100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 0x7f4930100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 -- 172.21.15.38:0/806192343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49301032e0 msgr2=0x7f4930100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49301032e0 0x7f4930100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 -- 172.21.15.38:0/806192343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49301a6570 con 0x7f493010bd00 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.824+0000 7f49359eb640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f492800eaa0 tx=0x7f492800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.825+0000 7f49267fc640 1 -- 172.21.15.38:0/806192343 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492800cdb0 con 0x7f493010bd00 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.825+0000 7f49267fc640 1 -- 172.21.15.38:0/806192343 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4928005020 con 0x7f493010bd00 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.825+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49301a6860 con 0x7f493010bd00 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.825+0000 7f49267fc640 1 -- 172.21.15.38:0/806192343 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49280116b0 con 0x7f493010bd00 2024-04-22T00:02:46.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.825+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49301a6d30 con 0x7f493010bd00 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.827+0000 7f49267fc640 1 -- 172.21.15.38:0/806192343 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f4928005560 con 0x7f493010bd00 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 Processor -- start 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 -- start start 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 0x556478090fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478091520 0x556478093910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556477fec530 con 0x556478088020 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49267fc640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 0x7f491c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556477f94830 con 0x556478091520 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f493a036740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556477fe84f0 con 0x556478088940 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45316/0 (socket says 172.21.15.38:45316) 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49349e9640 1 -- 172.21.15.38:0/675067332 learned_addr learned my addr 172.21.15.38:0/675067332 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49359eb640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 0x7f491c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49351ea640 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 0x556478090fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.828+0000 7f49267fc640 1 -- 172.21.15.38:0/806192343 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f49280cfc30 con 0x7f493010bd00 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 -- 172.21.15.38:0/675067332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 msgr2=0x556478090fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 0x556478090fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 -- 172.21.15.38:0/675067332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478091520 msgr2=0x556478093910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478091520 0x556478093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 -- 172.21.15.38:0/675067332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556477fb82c0 con 0x556478088020 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49351ea640 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 0x556478090fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49359eb640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 0x7f491c0a52e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49280ccf20 tx=0x7f4928092040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f49349e9640 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4920000c00 tx=0x7f4920005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcfb44cc74fa09e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f491b7fe640 1 -- 172.21.15.38:0/675067332 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49200042f0 con 0x556478088020 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f491b7fe640 1 -- 172.21.15.38:0/675067332 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4920004490 con 0x556478088020 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.829+0000 7f491b7fe640 1 -- 172.21.15.38:0/675067332 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4920004920 con 0x556478088020 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- 172.21.15.38:0/675067332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 msgr2=0x556478088400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4920000c00 tx=0x7f4920005e60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- 172.21.15.38:0/675067332 shutdown_connections 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478091520 0x556478093910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478088940 0x556478090fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 --2- 172.21.15.38:0/675067332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478088400 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- 172.21.15.38:0/675067332 >> 172.21.15.38:0/675067332 conn(0x556478082750 msgr2=0x556478082b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- 172.21.15.38:0/675067332 shutdown_connections 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- 172.21.15.38:0/675067332 wait complete. 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 -- start start 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.830+0000 7f493a036740 1 Processor -- start 2024-04-22T00:02:46.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x55647808e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x55647808e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5564780942b0 con 0x556478088020 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556477fa7f10 con 0x556478088940 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556478094430 con 0x556478091520 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f493a036740 10 client.0 _ll_register_callbacks cb 0x556477cffbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45330/0 (socket says 172.21.15.38:45330) 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49349e9640 1 -- 172.21.15.38:0/1874769283 learned_addr learned my addr 172.21.15.38:0/1874769283 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x55647808e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49349e9640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x55647808e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 -- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 msgr2=0x55647808e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x55647808e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 -- 172.21.15.38:0/1874769283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 msgr2=0x556478090ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49351ea640 1 -- 172.21.15.38:0/1874769283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556477f79c80 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.831+0000 7f49349e9640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f49351ea640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x55647808e280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f492c005fd0 tx=0x7f492c00be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=551e8a98bbc1b923 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c003ea0 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4920004490 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f492c01b640 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f492c024050 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 10 client.25586 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556477d003f0 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4920004490 con 0x556478088940 2024-04-22T00:02:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f492c019020 con 0x556478088940 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 10 client.25586 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 10 client.25586 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.832+0000 7f493a036740 20 client.25586 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5564780a1570 0x5564780a3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f493a036740 4 client.25586 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556477ff9a70 con 0x5564780a1570 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f49349e9640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5564780a1570 0x5564780a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f4918ff9640 10 client.25586 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.833+0000 7f49349e9640 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5564780a1570 0x5564780a3980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2be06110808b861 server_cookie=6c3321f271289ed4 in_seq=0 out_seq=0 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f4918ff9640 1 -- 172.21.15.38:0/1874769283 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556477ff9a70 con 0x5564780a1570 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f4918ff9640 10 client.25586 handle_command_reply: tid=0 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 client.25586 shutdown 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 msgr2=0x55647808e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x55647808e280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f492c005fd0 tx=0x7f492c00be00 comp rx=0 tx=0).stop 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 shutdown_connections 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5564780a1570 0x5564780a3980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x55647808e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f4918ff9640 0 client.25586 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x55647808e280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 --2- 172.21.15.38:0/1874769283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x556478090ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 >> 172.21.15.38:0/1874769283 conn(0x556478082750 msgr2=0x5564780844a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 shutdown_connections 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.834+0000 7f493a036740 1 -- 172.21.15.38:0/1874769283 wait complete. 2024-04-22T00:02:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.835+0000 7f493a036740 20 client.25586 trim_cache size 0 max 16384 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 Processor -- start 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 -- start start 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x5564780909b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478090ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f49301b0c60 con 0x556478088020 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49301b27a0 con 0x556478088940 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f493a036740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49301b1330 con 0x556478091520 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f49349e9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45344/0 (socket says 172.21.15.38:45344) 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.849+0000 7f49349e9640 1 -- 172.21.15.38:0/3888895073 learned_addr learned my addr 172.21.15.38:0/3888895073 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49351ea640 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x5564780909b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478090ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 -- 172.21.15.38:0/3888895073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 msgr2=0x556478090ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478090ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 -- 172.21.15.38:0/3888895073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 msgr2=0x5564780909b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x5564780909b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 -- 172.21.15.38:0/3888895073 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55647808d230 con 0x556478088020 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f49349e9640 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4920009900 tx=0x7f4920005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a25b97ebcd46b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f491affd640 1 -- 172.21.15.38:0/3888895073 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4920054d50 con 0x556478088020 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f491affd640 1 -- 172.21.15.38:0/3888895073 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4920054040 con 0x556478088020 2024-04-22T00:02:46.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.850+0000 7f491affd640 1 -- 172.21.15.38:0/3888895073 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4920003850 con 0x556478088020 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- 172.21.15.38:0/3888895073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 msgr2=0x55647809d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4920009900 tx=0x7f4920005540 comp rx=0 tx=0).stop 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- 172.21.15.38:0/3888895073 shutdown_connections 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478090ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088940 0x5564780909b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 --2- 172.21.15.38:0/3888895073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088020 0x55647809d170 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- 172.21.15.38:0/3888895073 >> 172.21.15.38:0/3888895073 conn(0x556478082750 msgr2=0x556478081850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- 172.21.15.38:0/3888895073 shutdown_connections 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- 172.21.15.38:0/3888895073 wait complete. 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 -- start start 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.851+0000 7f493a036740 1 Processor -- start 2024-04-22T00:02:46.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088020 0x556478095430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f49301b20d0 con 0x556478088940 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f49301b1a00 con 0x556478088020 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49301aec40 con 0x556478091520 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f493a036740 10 client.0 _ll_register_callbacks cb 0x7f492c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45348/0 (socket says 172.21.15.38:45348) 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 -- 172.21.15.38:0/2899159399 learned_addr learned my addr 172.21.15.38:0/2899159399 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49349e9640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088020 0x556478095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 -- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 msgr2=0x556478106ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 -- 172.21.15.38:0/2899159399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088020 msgr2=0x556478095430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088020 0x556478095430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 -- 172.21.15.38:0/2899159399 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55647808ec00 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f49351ea640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x55647808f850 tx=0x7f492c005ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c829844f1ab809a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.852+0000 7f4918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c0039c0 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f492c011650 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4920054040 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f492c039050 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 10 client.16056 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556477dca7d0 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4920054040 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f492c019050 con 0x556478088940 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 10 client.16056 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 10 client.16056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 20 client.16056 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x556478108570 0x55647810a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 4 client.16056 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.853+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556477ff3dc0 con 0x556478108570 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f49349e9640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x556478108570 0x55647810a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f49349e9640 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x556478108570 0x55647810a990 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=547c38a1fce4a216 server_cookie=ca0d01558f7482f5 in_seq=0 out_seq=0 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f4918ff9640 10 client.16056 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f4918ff9640 1 -- 172.21.15.38:0/2899159399 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556477ff3dc0 con 0x556478108570 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f4918ff9640 10 client.16056 handle_command_reply: tid=0 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 client.16056 shutdown 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 msgr2=0x556478095970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x55647808f850 tx=0x7f492c005ca0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 shutdown_connections 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x556478108570 0x55647810a990 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556478091520 0x556478106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556478088940 0x556478095970 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 --2- 172.21.15.38:0/2899159399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556478088020 0x556478095430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 >> 172.21.15.38:0/2899159399 conn(0x556478082750 msgr2=0x556478080fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.854+0000 7f4918ff9640 0 client.16056 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 shutdown_connections 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493a036740 1 -- 172.21.15.38:0/2899159399 wait complete. 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493a036740 20 client.16056 trim_cache size 0 max 16384 2024-04-22T00:02:46.853 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 msgr2=0x7f491c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 0x7f491c0a52e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f49280ccf20 tx=0x7f4928092040 comp rx=0 tx=0).stop 2024-04-22T00:02:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 msgr2=0x7f4930101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f492800eaa0 tx=0x7f492800ef60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 shutdown_connections 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f491c0a2e20 0x7f491c0a52e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493010bd00 0x7f4930101090 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49301032e0 0x7f4930100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 --2- 172.21.15.38:0/806192343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49301029c0 0x7f4930100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 >> 172.21.15.38:0/806192343 conn(0x7f4930077740 msgr2=0x7f49300fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 shutdown_connections 2024-04-22T00:02:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.855+0000 7f493744e640 1 -- 172.21.15.38:0/806192343 wait complete. 2024-04-22T00:02:46.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 -- 172.21.15.38:0/3080676826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 msgr2=0x7f3164102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 --2- 172.21.15.38:0/3080676826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164102da0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3154009900 tx=0x7f315402bd60 comp rx=0 tx=0).stop 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 -- 172.21.15.38:0/3080676826 shutdown_connections 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 --2- 172.21.15.38:0/3080676826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f316410bd00 0x7f316410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 --2- 172.21.15.38:0/3080676826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641032e0 0x7f316410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 --2- 172.21.15.38:0/3080676826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164102da0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 -- 172.21.15.38:0/3080676826 >> 172.21.15.38:0/3080676826 conn(0x7f3164077740 msgr2=0x7f3164077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 -- 172.21.15.38:0/3080676826 shutdown_connections 2024-04-22T00:02:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.979+0000 7f316aa3b640 1 -- 172.21.15.38:0/3080676826 wait complete. 2024-04-22T00:02:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 Processor -- start 2024-04-22T00:02:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 -- start start 2024-04-22T00:02:46.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31641032e0 0x7f3164198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f316410bd00 0x7f316419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3164111b50 con 0x7f31641029c0 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31641119d0 con 0x7f31641032e0 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f316aa3b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3164111cd0 con 0x7f316410bd00 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45356/0 (socket says 172.21.15.38:45356) 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 -- 172.21.15.38:0/857415190 learned_addr learned my addr 172.21.15.38:0/857415190 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 -- 172.21.15.38:0/857415190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f316410bd00 msgr2=0x7f316419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f316410bd00 0x7f316419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 -- 172.21.15.38:0/857415190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31641032e0 msgr2=0x7f3164198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31641032e0 0x7f3164198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.980+0000 7f3163fff640 1 -- 172.21.15.38:0/857415190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316419cfe0 con 0x7f31641029c0 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f3163fff640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3154009a20 tx=0x7f315402df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f31617fa640 1 -- 172.21.15.38:0/857415190 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154043020 con 0x7f31641029c0 2024-04-22T00:02:46.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f31617fa640 1 -- 172.21.15.38:0/857415190 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31540054e0 con 0x7f31641029c0 2024-04-22T00:02:46.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316419d270 con 0x7f31641029c0 2024-04-22T00:02:46.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f31617fa640 1 -- 172.21.15.38:0/857415190 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154003ce0 con 0x7f31641029c0 2024-04-22T00:02:46.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.981+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31641a4b60 con 0x7f31641029c0 2024-04-22T00:02:46.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.983+0000 7f316d61a740 1 Processor -- start 2024-04-22T00:02:46.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.983+0000 7f316d61a740 1 -- start start 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.983+0000 7f31617fa640 1 -- 172.21.15.38:0/857415190 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f3154046020 con 0x7f31641029c0 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55ef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dda5553530 con 0x55dda55f8520 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dda54fb830 con 0x55dda55ef020 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f316d61a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dda554f4f0 con 0x55dda55ef940 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45368/0 (socket says 172.21.15.38:45368) 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 -- 172.21.15.38:0/114057711 learned_addr learned my addr 172.21.15.38:0/114057711 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f3168fd8640 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f3163fff640 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 -- 172.21.15.38:0/114057711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 msgr2=0x55dda55f7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 -- 172.21.15.38:0/114057711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 msgr2=0x55dda55ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55ef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 -- 172.21.15.38:0/114057711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda551f2c0 con 0x55dda55f8520 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31617fa640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 0x7f314c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f3168fd8640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 0x7f314c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.984+0000 7f31637fe640 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f315800bae0 tx=0x7f315800d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358feaa420ac79a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f31617fa640 1 -- 172.21.15.38:0/857415190 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f31540f1c60 con 0x7f31641029c0 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f314a7fc640 1 -- 172.21.15.38:0/114057711 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158016070 con 0x55dda55f8520 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f314a7fc640 1 -- 172.21.15.38:0/114057711 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3158005500 con 0x55dda55f8520 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f314a7fc640 1 -- 172.21.15.38:0/114057711 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158010460 con 0x55dda55f8520 2024-04-22T00:02:46.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f316d61a740 1 -- 172.21.15.38:0/114057711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 msgr2=0x55dda55fa910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f3168fd8640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 0x7f314c0a52e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f315c005cc0 tx=0x7f315c00a330 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.985+0000 7f316d61a740 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f315800bae0 tx=0x7f315800d2e0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- 172.21.15.38:0/114057711 shutdown_connections 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55fa910 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- 172.21.15.38:0/114057711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- 172.21.15.38:0/114057711 >> 172.21.15.38:0/114057711 conn(0x55dda55e9750 msgr2=0x55dda55e9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- 172.21.15.38:0/114057711 shutdown_connections 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- 172.21.15.38:0/114057711 wait complete. 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- start start 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 Processor -- start 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55f7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dda55fb310 con 0x55dda55f8520 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55dda550ef10 con 0x55dda55ef020 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dda55fb490 con 0x55dda55ef940 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.986+0000 7f316d61a740 10 client.0 _ll_register_callbacks cb 0x55dda5266be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45382/0 (socket says 172.21.15.38:45382) 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 -- 172.21.15.38:0/3679450516 learned_addr learned my addr 172.21.15.38:0/3679450516 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3163fff640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55f7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f31637fe640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 -- 172.21.15.38:0/3679450516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 msgr2=0x55dda55f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 -- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 msgr2=0x55dda55f7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55f7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 -- 172.21.15.38:0/3679450516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda54e0c80 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f3168fd8640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f315c010e30 tx=0x7f315c005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15aa71427b7517c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f31437fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c021070 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f315c01d450 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.987+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3158005500 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f315c02d050 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 10 client.16068 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dda52673f0 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3158005500 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f315c025540 con 0x55dda55f8520 2024-04-22T00:02:46.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 10 client.16068 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 20 client.16068 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda5608520 0x55dda560a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 4 client.16068 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dda5560a70 con 0x55dda5608520 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.988+0000 7f3163fff640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda5608520 0x55dda560a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.989+0000 7f3163fff640 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda5608520 0x55dda560a970 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2745ea502147e86 server_cookie=61b6fa14e0de73e6 in_seq=0 out_seq=0 2024-04-22T00:02:46.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.989+0000 7f31437fe640 10 client.16068 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:46.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f31437fe640 1 -- 172.21.15.38:0/3679450516 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dda5560a70 con 0x55dda5608520 2024-04-22T00:02:46.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f31437fe640 10 client.16068 handle_command_reply: tid=0 2024-04-22T00:02:46.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 client.16068 shutdown 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 msgr2=0x55dda55f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f315c010e30 tx=0x7f315c005c10 comp rx=0 tx=0).stop 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 shutdown_connections 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda5608520 0x55dda560a970 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f5700 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f31437fe640 0 client.16068 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 --2- 172.21.15.38:0/3679450516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda55f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 >> 172.21.15.38:0/3679450516 conn(0x55dda55e9750 msgr2=0x55dda55eb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 shutdown_connections 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 1 -- 172.21.15.38:0/3679450516 wait complete. 2024-04-22T00:02:46.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:46.990+0000 7f316d61a740 20 client.16068 trim_cache size 0 max 16384 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.003+0000 7f316d61a740 1 Processor -- start 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.003+0000 7f316d61a740 1 -- start start 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda56041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55eb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31641aea30 con 0x55dda55f8520 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31641b0570 con 0x55dda55ef020 2024-04-22T00:02:46.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f316d61a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31641af100 con 0x55dda55ef940 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45398/0 (socket says 172.21.15.38:45398) 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 -- 172.21.15.38:0/2159936549 learned_addr learned my addr 172.21.15.38:0/2159936549 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f31637fe640 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3163fff640 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda56041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 -- 172.21.15.38:0/2159936549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 msgr2=0x55dda55eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 -- 172.21.15.38:0/2159936549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 msgr2=0x55dda56041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda56041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 -- 172.21.15.38:0/2159936549 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dda53317d0 con 0x55dda55f8520 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.004+0000 7f3168fd8640 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f315c00ebb0 tx=0x7f315c00a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa90043bffcea97d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f3149ffb640 1 -- 172.21.15.38:0/2159936549 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c035030 con 0x55dda55f8520 2024-04-22T00:02:46.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f3149ffb640 1 -- 172.21.15.38:0/2159936549 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c025b80 con 0x55dda55f8520 2024-04-22T00:02:46.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f3149ffb640 1 -- 172.21.15.38:0/2159936549 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c021050 con 0x55dda55f8520 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 -- 172.21.15.38:0/2159936549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 msgr2=0x55dda55f78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f315c00ebb0 tx=0x7f315c00a460 comp rx=0 tx=0).stop 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 -- 172.21.15.38:0/2159936549 shutdown_connections 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55f8520 0x55dda55f78f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef940 0x55dda55eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 --2- 172.21.15.38:0/2159936549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55ef020 0x55dda56041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 -- 172.21.15.38:0/2159936549 >> 172.21.15.38:0/2159936549 conn(0x55dda55e9750 msgr2=0x55dda55eab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 -- 172.21.15.38:0/2159936549 shutdown_connections 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.005+0000 7f316d61a740 1 -- 172.21.15.38:0/2159936549 wait complete. 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 -- start start 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 Processor -- start 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef020 0x55dda55f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55f8520 0x55dda55f2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31641afea0 con 0x55dda55ef940 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f31641af7d0 con 0x55dda55f8520 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31641aca10 con 0x55dda55ef020 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f316d61a740 10 client.0 _ll_register_callbacks cb 0x7f315c014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f31637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:47.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f31637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45404/0 (socket says 172.21.15.38:45404) 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f31637fe640 1 -- 172.21.15.38:0/1586423314 learned_addr learned my addr 172.21.15.38:0/1586423314 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.006+0000 7f3163fff640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef020 0x55dda55f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f3168fd8640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55f8520 0x55dda55f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 -- 172.21.15.38:0/1586423314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef020 msgr2=0x55dda55f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef020 0x55dda55f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 -- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55f8520 msgr2=0x55dda55f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55f8520 0x55dda55f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 -- 172.21.15.38:0/1586423314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dda566db00 con 0x55dda55ef940 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31637fe640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f315800bae0 tx=0x7f3158004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc028ccae7088ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31437fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3158016030 con 0x55dda55ef940 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3158004040 con 0x55dda55ef940 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315c025b80 con 0x55dda55ef940 2024-04-22T00:02:47.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3158004960 con 0x55dda55ef940 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f316d61a740 10 client.16074 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dda566ee30 con 0x55dda55ef940 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.007+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c025b80 con 0x55dda55ef940 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3158004cd0 con 0x55dda55ef940 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 10 client.16074 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 20 client.16074 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda566f4e0 0x55dda5671900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 4 client.16074 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dda555adc0 con 0x55dda566f4e0 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f3163fff640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda566f4e0 0x55dda5671900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f3163fff640 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda566f4e0 0x55dda5671900 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b0e05dcff0e2649 server_cookie=5a2799aa53160ddc in_seq=0 out_seq=0 2024-04-22T00:02:47.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f31437fe640 10 client.16074 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f31437fe640 1 -- 172.21.15.38:0/1586423314 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dda555adc0 con 0x55dda566f4e0 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f31437fe640 10 client.16074 handle_command_reply: tid=0 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.008+0000 7f316d61a740 1 client.16074 shutdown 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 msgr2=0x55dda55f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f315800bae0 tx=0x7f3158004b00 comp rx=0 tx=0).stop 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 shutdown_connections 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dda55f8520 0x55dda55f2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55dda566f4e0 0x55dda5671900 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dda55ef940 0x55dda55f2410 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 --2- 172.21.15.38:0/1586423314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dda55ef020 0x55dda55f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f31437fe640 0 client.16074 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 >> 172.21.15.38:0/1586423314 conn(0x55dda55e9750 msgr2=0x55dda55eb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 shutdown_connections 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 1 -- 172.21.15.38:0/1586423314 wait complete. 2024-04-22T00:02:47.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316d61a740 20 client.16074 trim_cache size 0 max 16384 2024-04-22T00:02:47.008 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 msgr2=0x7f314c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 0x7f314c0a52e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f315c005cc0 tx=0x7f315c00a330 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 msgr2=0x7f3164197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3154009a20 tx=0x7f315402df60 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 shutdown_connections 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f314c0a2e20 0x7f314c0a52e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f316410bd00 0x7f316419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31641032e0 0x7f3164198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 --2- 172.21.15.38:0/857415190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f31641029c0 0x7f3164197ff0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.009+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 >> 172.21.15.38:0/857415190 conn(0x7f3164077740 msgr2=0x7f31640fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.010+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 shutdown_connections 2024-04-22T00:02:47.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:47.010+0000 7f316aa3b640 1 -- 172.21.15.38:0/857415190 wait complete. 2024-04-22T00:02:47.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1345813188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:47.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1345813188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:47.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1345813188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:02:48.014 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:02:48.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:02:48.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:02:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 -- 172.21.15.38:0/712322879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e8102150 msgr2=0x7fc9e810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/712322879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e8102150 0x7fc9e810e660 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009cd0 tx=0x7fc9e002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 -- 172.21.15.38:0/712322879 shutdown_connections 2024-04-22T00:02:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/712322879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e8102150 0x7fc9e810e660 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/712322879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8101790 0x7fc9e8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/712322879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e81077a0 0x7fc9e8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.130+0000 7fc9ed09d640 1 -- 172.21.15.38:0/712322879 >> 172.21.15.38:0/712322879 conn(0x7fc9e80fd2a0 msgr2=0x7fc9e80ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.131+0000 7fc9ed09d640 1 -- 172.21.15.38:0/712322879 shutdown_connections 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.131+0000 7fc9ed09d640 1 -- 172.21.15.38:0/712322879 wait complete. 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.131+0000 7fc9ed09d640 1 Processor -- start 2024-04-22T00:02:48.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.131+0000 7fc9ed09d640 1 -- start start 2024-04-22T00:02:48.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e8101790 0x7fc9e8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8102150 0x7fc9e8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8111b70 con 0x7fc9e81077a0 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81119f0 con 0x7fc9e8102150 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9ed09d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8111cf0 con 0x7fc9e8101790 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45424/0 (socket says 172.21.15.38:45424) 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 -- 172.21.15.38:0/4112891099 learned_addr learned my addr 172.21.15.38:0/4112891099 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e8101790 0x7fc9e8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e659c640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8102150 0x7fc9e8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 -- 172.21.15.38:0/4112891099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e8101790 msgr2=0x7fc9e8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e8101790 0x7fc9e8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 -- 172.21.15.38:0/4112891099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8102150 msgr2=0x7fc9e8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8102150 0x7fc9e8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 -- 172.21.15.38:0/4112891099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e819cfe0 con 0x7fc9e81077a0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9e759e640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00348f0 tx=0x7fc9e0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.132+0000 7fc9cffff640 1 -- 172.21.15.38:0/4112891099 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0041020 con 0x7fc9e81077a0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.133+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e819d270 con 0x7fc9e81077a0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.133+0000 7fc9cffff640 1 -- 172.21.15.38:0/4112891099 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9e0034cd0 con 0x7fc9e81077a0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.133+0000 7fc9cffff640 1 -- 172.21.15.38:0/4112891099 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0039040 con 0x7fc9e81077a0 2024-04-22T00:02:48.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.133+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e81a4b60 con 0x7fc9e81077a0 2024-04-22T00:02:48.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9cffff640 1 -- 172.21.15.38:0/4112891099 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7fc9e0044020 con 0x7fc9e81077a0 2024-04-22T00:02:48.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 Processor -- start 2024-04-22T00:02:48.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 -- start start 2024-04-22T00:02:48.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed3c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed3cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56376eca0170 con 0x56376ed45db0 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56376ec487c0 con 0x56376ed3d720 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.135+0000 7fc9efd66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56376ec9c130 con 0x56376ed3ca40 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9cffff640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 0x7fc9d00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45440/0 (socket says 172.21.15.38:45440) 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 -- 172.21.15.38:0/3090831824 learned_addr learned my addr 172.21.15.38:0/3090831824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed3c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e759e640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 0x7fc9d00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed3cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 -- 172.21.15.38:0/3090831824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 msgr2=0x56376ed3cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed3cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 -- 172.21.15.38:0/3090831824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 msgr2=0x56376ed3c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed3c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9cffff640 1 -- 172.21.15.38:0/4112891099 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc9e00f1890 con 0x7fc9e81077a0 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 -- 172.21.15.38:0/3090831824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376ec59e60 con 0x56376ed45db0 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed3cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0149c0 tx=0x7fc9dc00c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288de43bd4cf1898 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9e759e640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 0x7fc9d00a52e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00eec90 tx=0x7fc9e008f8f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9cd7fa640 1 -- 172.21.15.38:0/3090831824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0142d0 con 0x56376ed45db0 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9cd7fa640 1 -- 172.21.15.38:0/3090831824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9dc00fc40 con 0x56376ed45db0 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.136+0000 7fc9cd7fa640 1 -- 172.21.15.38:0/3090831824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc012bc0 con 0x56376ed45db0 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- 172.21.15.38:0/3090831824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 msgr2=0x56376ed481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0149c0 tx=0x7fc9dc00c4d0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- 172.21.15.38:0/3090831824 shutdown_connections 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed481a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed3cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3090831824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed3c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- 172.21.15.38:0/3090831824 >> 172.21.15.38:0/3090831824 conn(0x56376ed034f0 msgr2=0x56376ed37490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- 172.21.15.38:0/3090831824 shutdown_connections 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- 172.21.15.38:0/3090831824 wait complete. 2024-04-22T00:02:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 -- start start 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.137+0000 7fc9efd66740 1 Processor -- start 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed43300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed43840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56376ed48c20 con 0x56376ed3ca40 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56376ec5bd40 con 0x56376ed3d720 2024-04-22T00:02:48.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56376ed48da0 con 0x56376ed45db0 2024-04-22T00:02:48.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9efd66740 10 client.0 _ll_register_callbacks cb 0x56376ec97820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:48.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45450/0 (socket says 172.21.15.38:45450) 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 -- 172.21.15.38:0/4107113363 learned_addr learned my addr 172.21.15.38:0/4107113363 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed43300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 -- 172.21.15.38:0/4107113363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 msgr2=0x56376ed43840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed43840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 -- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 msgr2=0x56376ed43300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed43300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 -- 172.21.15.38:0/4107113363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56376ea7c7e0 con 0x56376ed3ca40 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9e659c640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc005d00 tx=0x7fc9dc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83820a248ec9ee7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9caff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.138+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0142d0 con 0x56376ed3ca40 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc9dc00f730 con 0x56376ed3ca40 2024-04-22T00:02:48.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9dc00fc40 con 0x56376ed3ca40 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc9dc031050 con 0x56376ed3ca40 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 10 client.16086 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56376eadcf70 con 0x56376ed3ca40 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9dc00fc40 con 0x56376ed3ca40 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc9dc0035f0 con 0x56376ed3ca40 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 10 client.16086 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 20 client.16086 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376ed55cf0 0x56376ed580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 4 client.16086 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.139+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56376ec74450 con 0x56376ed55cf0 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.140+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376ed55cf0 0x56376ed580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.140+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376ed55cf0 0x56376ed580e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=624c02ea4f7842b6 server_cookie=953fbad1410799b2 in_seq=0 out_seq=0 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.140+0000 7fc9caff5640 10 client.16086 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9caff5640 1 -- 172.21.15.38:0/4107113363 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56376ec74450 con 0x56376ed55cf0 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9caff5640 10 client.16086 handle_command_reply: tid=0 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 client.16086 shutdown 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 msgr2=0x56376ed42dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc005d00 tx=0x7fc9dc005740 comp rx=0 tx=0).stop 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 shutdown_connections 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed43840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed43300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376ed55cf0 0x56376ed580e0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 --2- 172.21.15.38:0/4107113363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3ca40 0x56376ed42dc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9caff5640 0 client.16086 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 >> 172.21.15.38:0/4107113363 conn(0x56376ed034f0 msgr2=0x56376ed391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 shutdown_connections 2024-04-22T00:02:48.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 1 -- 172.21.15.38:0/4107113363 wait complete. 2024-04-22T00:02:48.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.141+0000 7fc9efd66740 20 client.16086 trim_cache size 0 max 16384 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 Processor -- start 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 -- start start 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed51b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81ae9c0 con 0x56376ed45db0 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81b0500 con 0x56376ed3d720 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9efd66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81af090 con 0x56376ed3ca40 2024-04-22T00:02:48.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9e659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45456/0 (socket says 172.21.15.38:45456) 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9e659c640 1 -- 172.21.15.38:0/3975972148 learned_addr learned my addr 172.21.15.38:0/3975972148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed51b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.159+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 -- 172.21.15.38:0/3975972148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 msgr2=0x56376ed454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 -- 172.21.15.38:0/3975972148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 msgr2=0x56376ed51b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed51b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 -- 172.21.15.38:0/3975972148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376e974170 con 0x56376ed45db0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc003f10 tx=0x7fc9dc005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f97c54b0bf2ec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9ccff9640 1 -- 172.21.15.38:0/3975972148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00c8d0 con 0x56376ed45db0 2024-04-22T00:02:48.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9ccff9640 1 -- 172.21.15.38:0/3975972148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9dc004470 con 0x56376ed45db0 2024-04-22T00:02:48.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.160+0000 7fc9ccff9640 1 -- 172.21.15.38:0/3975972148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0140e0 con 0x56376ed45db0 2024-04-22T00:02:48.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- 172.21.15.38:0/3975972148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 msgr2=0x56376ed36370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc003f10 tx=0x7fc9dc005b90 comp rx=0 tx=0).stop 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- 172.21.15.38:0/3975972148 shutdown_connections 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed45db0 0x56376ed36370 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3d720 0x56376ed51b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3975972148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed3ca40 0x56376ed454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- 172.21.15.38:0/3975972148 >> 172.21.15.38:0/3975972148 conn(0x56376ed034f0 msgr2=0x56376ed35c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- 172.21.15.38:0/3975972148 shutdown_connections 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- 172.21.15.38:0/3975972148 wait complete. 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- start start 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 Processor -- start 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 0x56376ed4e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed40580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81afe30 con 0x56376ed3d720 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81af760 con 0x56376ed3ca40 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81ac9a0 con 0x56376ed45db0 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.161+0000 7fc9efd66740 10 client.0 _ll_register_callbacks cb 0x7fc9dc00cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45462/0 (socket says 172.21.15.38:45462) 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 -- 172.21.15.38:0/3471106235 learned_addr learned my addr 172.21.15.38:0/3471106235 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 0x56376ed4e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 -- 172.21.15.38:0/3471106235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 msgr2=0x56376ed40580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed40580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 -- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 msgr2=0x56376ed4e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 0x56376ed4e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 -- 172.21.15.38:0/3471106235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56376ed40d60 con 0x56376ed3d720 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 0x56376ed4e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9e6d9d640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8005fd0 tx=0x7fc9d8012740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab181f4acd05d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9caff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8004030 con 0x56376ed3d720 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc9d80041d0 con 0x56376ed3d720 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9dc004470 con 0x56376ed3d720 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc9d802b050 con 0x56376ed3d720 2024-04-22T00:02:48.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9efd66740 10 client.16092 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.162+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56376edc0590 con 0x56376ed3d720 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9dc004470 con 0x56376ed3d720 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc9d8003a60 con 0x56376ed3d720 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 10 client.16092 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 20 client.16092 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376edc0d40 0x56376edc3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 4 client.16092 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56376ecad6b0 con 0x56376edc0d40 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.163+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376edc0d40 0x56376edc3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.164+0000 7fc9caff5640 10 client.16092 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.164+0000 7fc9e659c640 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376edc0d40 0x56376edc3160 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f83338b1f4d54a2e server_cookie=33957153e231767e in_seq=0 out_seq=0 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9caff5640 1 -- 172.21.15.38:0/3471106235 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56376ecad6b0 con 0x56376edc0d40 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9caff5640 10 client.16092 handle_command_reply: tid=0 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 client.16092 shutdown 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 msgr2=0x56376ed40040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8005fd0 tx=0x7fc9d8012740 comp rx=0 tx=0).stop 2024-04-22T00:02:48.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 shutdown_connections 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56376ed45db0 0x56376ed40580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56376edc0d40 0x56376edc3160 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56376ed3d720 0x56376ed40040 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 --2- 172.21.15.38:0/3471106235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56376ed3ca40 0x56376ed4e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 >> 172.21.15.38:0/3471106235 conn(0x56376ed034f0 msgr2=0x56376ed41df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9caff5640 0 client.16092 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 shutdown_connections 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 1 -- 172.21.15.38:0/3471106235 wait complete. 2024-04-22T00:02:48.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.165+0000 7fc9efd66740 20 client.16092 trim_cache size 0 max 16384 2024-04-22T00:02:48.169 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:48.169 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:02:48.169 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "eb44f82e-bb21-44db-bb92-d4c5686d315a", 2024-04-22T00:02:48.169 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:02:48.169 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:48.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 msgr2=0x7fc9d00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 0x7fc9d00a52e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00eec90 tx=0x7fc9e008f8f0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 msgr2=0x7fc9e819c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc9e00348f0 tx=0x7fc9e0032ec0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 shutdown_connections 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc9d00a2e20 0x7fc9d00a52e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9e81077a0 0x7fc9e819c8b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9e8102150 0x7fc9e8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 --2- 172.21.15.38:0/4112891099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e8101790 0x7fc9e8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 >> 172.21.15.38:0/4112891099 conn(0x7fc9e80fd2a0 msgr2=0x7fc9e80fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 shutdown_connections 2024-04-22T00:02:48.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.166+0000 7fc9ed09d640 1 -- 172.21.15.38:0/4112891099 wait complete. 2024-04-22T00:02:48.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:02:48.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:02:48.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:47 smithi038 ceph-mon[27972]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:02:48.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:47 smithi203 ceph-mon[31431]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:02:48.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 -- 172.21.15.38:0/3606194108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 msgr2=0x7ff4b410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 --2- 172.21.15.38:0/3606194108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b410da80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4009900 tx=0x7ff4a402bcd0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 -- 172.21.15.38:0/3606194108 shutdown_connections 2024-04-22T00:02:48.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 --2- 172.21.15.38:0/3606194108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 0x7ff4b41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 --2- 172.21.15.38:0/3606194108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b410da80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 --2- 172.21.15.38:0/3606194108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 0x7ff4b4073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 -- 172.21.15.38:0/3606194108 >> 172.21.15.38:0/3606194108 conn(0x7ff4b4078b40 msgr2=0x7ff4b4078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 -- 172.21.15.38:0/3606194108 shutdown_connections 2024-04-22T00:02:48.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.289+0000 7ff4bb572640 1 -- 172.21.15.38:0/3606194108 wait complete. 2024-04-22T00:02:48.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 Processor -- start 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 -- start start 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 0x7ff4b419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 0x7ff4b419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b4113d20 con 0x7ff4b4074430 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b4113ba0 con 0x7ff4b4073b10 2024-04-22T00:02:48.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4bb572640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b4113ea0 con 0x7ff4b410dfc0 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4b8b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4b8b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45476/0 (socket says 172.21.15.38:45476) 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/1778820565 learned_addr learned my addr 172.21.15.38:0/1778820565 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.290+0000 7ff4b930e640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 0x7ff4b419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 0x7ff4b419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/1778820565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 msgr2=0x7ff4b419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 0x7ff4b419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/1778820565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 msgr2=0x7ff4b419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 0x7ff4b419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/1778820565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b419f2b0 con 0x7ff4b4074430 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005940 tx=0x7ff4a402cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4aa7fc640 1 -- 172.21.15.38:0/1778820565 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4041070 con 0x7ff4b4074430 2024-04-22T00:02:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b419f540 con 0x7ff4b4074430 2024-04-22T00:02:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b41a6e30 con 0x7ff4b4074430 2024-04-22T00:02:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4aa7fc640 1 -- 172.21.15.38:0/1778820565 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a4004450 con 0x7ff4b4074430 2024-04-22T00:02:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.291+0000 7ff4aa7fc640 1 -- 172.21.15.38:0/1778820565 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4034440 con 0x7ff4b4074430 2024-04-22T00:02:48.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.294+0000 7ff4aa7fc640 1 -- 172.21.15.38:0/1778820565 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7ff4a4044070 con 0x7ff4b4074430 2024-04-22T00:02:48.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.294+0000 7ff4aa7fc640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 0x7ff4a00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.294+0000 7ff4aa7fc640 1 -- 172.21.15.38:0/1778820565 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff4a40f1380 con 0x7ff4b4074430 2024-04-22T00:02:48.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.294+0000 7ff4b930e640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 0x7ff4a00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 Processor -- start 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 -- start start 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4b930e640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 0x7ff4a00a52e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00058c0 tx=0x7ff4b000a2d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca77310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca80430 0x55edeca82820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55edec9db560 con 0x55edeca77850 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55edec983730 con 0x55edeca80430 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4be230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55edec9d7520 con 0x55edeca76f30 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4b8b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.295+0000 7ff4b8b0d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45488/0 (socket says 172.21.15.38:45488) 2024-04-22T00:02:48.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/2272423337 learned_addr learned my addr 172.21.15.38:0/2272423337 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca80430 0x55edeca82820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca77310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/2272423337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 msgr2=0x55edeca77310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca77310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/2272423337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca80430 msgr2=0x55edeca82820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca80430 0x55edeca82820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 -- 172.21.15.38:0/2272423337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edec7b97d0 con 0x55edeca77850 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca77310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff4a402c8c0 tx=0x7ff4a40bd360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d2a64d0e00d6b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff49f7fe640 1 -- 172.21.15.38:0/2272423337 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a404b040 con 0x55edeca77850 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff49f7fe640 1 -- 172.21.15.38:0/2272423337 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a40c17d0 con 0x55edeca77850 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.296+0000 7ff49f7fe640 1 -- 172.21.15.38:0/2272423337 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40ee8f0 con 0x55edeca77850 2024-04-22T00:02:48.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- 172.21.15.38:0/2272423337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 msgr2=0x55edeca7fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff4a402c8c0 tx=0x7ff4a40bd360 comp rx=0 tx=0).stop 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- 172.21.15.38:0/2272423337 shutdown_connections 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca80430 0x55edeca82820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca77850 0x55edeca7fef0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 --2- 172.21.15.38:0/2272423337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca77310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- 172.21.15.38:0/2272423337 >> 172.21.15.38:0/2272423337 conn(0x55edeca71660 msgr2=0x55edeca71a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- 172.21.15.38:0/2272423337 shutdown_connections 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- 172.21.15.38:0/2272423337 wait complete. 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 -- start start 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.297+0000 7ff4be230740 1 Processor -- start 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca76f30 0x55edeca7d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca77850 0x55edeca7d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55edeca82fe0 con 0x55edeca80430 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55edec996e00 con 0x55edeca76f30 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55edeca83160 con 0x55edeca77850 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4be230740 10 client.0 _ll_register_callbacks cb 0x55edec9d1eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45504/0 (socket says 172.21.15.38:45504) 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2970652237 learned_addr learned my addr 172.21.15.38:0/2970652237 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca77850 0x55edeca7d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca76f30 0x55edeca7d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2970652237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca77850 msgr2=0x55edeca7d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca77850 0x55edeca7d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca76f30 msgr2=0x55edeca7d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca76f30 0x55edeca7d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.298+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2970652237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edec6ef3f0 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac012700 tx=0x7ff4ac012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b17411b1b0af66d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac003d80 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff4ac004b20 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4a40c17d0 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4ac004d20 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 10 client.16104 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55edec6af0f0 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a40c17d0 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4ac019070 con 0x55edeca80430 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 10 client.16104 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:48.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 20 client.16104 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edeca90610 0x55edeca92a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 4 client.16104 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.299+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edec9e8aa0 con 0x55edeca90610 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.300+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edeca90610 0x55edeca92a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.300+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edeca90610 0x55edeca92a50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=106c9e8f5489967b server_cookie=e6a7bd818279caaa in_seq=0 out_seq=0 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.300+0000 7ff49cff9640 10 client.16104 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff49cff9640 1 -- 172.21.15.38:0/2970652237 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edec9e8aa0 con 0x55edeca90610 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff49cff9640 10 client.16104 handle_command_reply: tid=0 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 client.16104 shutdown 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 msgr2=0x55edeca7def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac012700 tx=0x7ff4ac012bc0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 shutdown_connections 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edeca90610 0x55edeca92a50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca7def0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca77850 0x55edeca7d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 --2- 172.21.15.38:0/2970652237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca76f30 0x55edeca7d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 >> 172.21.15.38:0/2970652237 conn(0x55edeca71660 msgr2=0x55edeca733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff49cff9640 0 client.16104 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 shutdown_connections 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 1 -- 172.21.15.38:0/2970652237 wait complete. 2024-04-22T00:02:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.301+0000 7ff4be230740 20 client.16104 trim_cache size 0 max 16384 2024-04-22T00:02:48.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.323+0000 7ff4be230740 1 Processor -- start 2024-04-22T00:02:48.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.323+0000 7ff4be230740 1 -- start start 2024-04-22T00:02:48.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca7fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca8bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff49800fda0 con 0x55edeca80430 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff4980118e0 con 0x55edeca77850 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4be230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff498010470 con 0x55edeca76f30 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45520/0 (socket says 172.21.15.38:45520) 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/3388935467 learned_addr learned my addr 172.21.15.38:0/3388935467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca8bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b930e640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca7fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/3388935467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 msgr2=0x55edeca7fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca7fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/3388935467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 msgr2=0x55edeca8bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca8bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/3388935467 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edeca7ba00 con 0x55edeca80430 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.324+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac015040 tx=0x7ff4ac009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac6e08ca2689112 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff49effd640 1 -- 172.21.15.38:0/3388935467 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac02c030 con 0x55edeca80430 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff49effd640 1 -- 172.21.15.38:0/3388935467 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4ac01a530 con 0x55edeca80430 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff49effd640 1 -- 172.21.15.38:0/3388935467 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac028680 con 0x55edeca80430 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 -- 172.21.15.38:0/3388935467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 msgr2=0x55edeca98320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac015040 tx=0x7ff4ac009d70 comp rx=0 tx=0).stop 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 -- 172.21.15.38:0/3388935467 shutdown_connections 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca98320 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 --2- 172.21.15.38:0/3388935467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca7fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 -- 172.21.15.38:0/3388935467 >> 172.21.15.38:0/3388935467 conn(0x55edeca71660 msgr2=0x55edeca7bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 -- 172.21.15.38:0/3388935467 shutdown_connections 2024-04-22T00:02:48.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.325+0000 7ff4be230740 1 -- 172.21.15.38:0/3388935467 wait complete. 2024-04-22T00:02:48.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 -- start start 2024-04-22T00:02:48.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 Processor -- start 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca883b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca888f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff498011210 con 0x55edeca80430 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff498010b40 con 0x55edeca77850 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff49800dd80 con 0x55edeca76f30 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4be230740 10 client.0 _ll_register_callbacks cb 0x7ff4ac005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4b9b0f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:45526/0 (socket says 172.21.15.38:45526) 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.326+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2520919098 learned_addr learned my addr 172.21.15.38:0/2520919098 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b8b0d640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca888f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2520919098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 msgr2=0x55edeca883b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca883b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 msgr2=0x55edeca888f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca888f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 -- 172.21.15.38:0/2520919098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edeca79ce0 con 0x55edeca80430 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4b9b0f640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009c50 tx=0x7ff4ac012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56fd9c6aec88fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff49cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac035070 con 0x55edeca80430 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff4ac01a6e0 con 0x55edeca80430 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4ac01a530 con 0x55edeca80430 2024-04-22T00:02:48.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4ac041050 con 0x55edeca80430 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4be230740 10 client.16110 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.327+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55edecaf6400 con 0x55edeca80430 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4ac01a530 con 0x55edeca80430 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4ac019050 con 0x55edeca80430 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 10 client.16110 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 20 client.16110 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edecaf6a10 0x55edecaf8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 4 client.16110 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55edec9e2df0 con 0x55edecaf6a10 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edecaf6a10 0x55edecaf8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff49cff9640 10 client.16110 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.328+0000 7ff4b930e640 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edecaf6a10 0x55edecaf8e30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab621e793ff776f6 server_cookie=815024dae408c6d8 in_seq=0 out_seq=0 2024-04-22T00:02:48.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff49cff9640 1 -- 172.21.15.38:0/2520919098 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55edec9e2df0 con 0x55edecaf6a10 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff49cff9640 10 client.16110 handle_command_reply: tid=0 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 client.16110 shutdown 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 msgr2=0x55edeca79720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009c50 tx=0x7ff4ac012bc0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 shutdown_connections 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55edecaf6a10 0x55edecaf8e30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55edeca80430 0x55edeca79720 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55edeca77850 0x55edeca888f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 --2- 172.21.15.38:0/2520919098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edeca76f30 0x55edeca883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 >> 172.21.15.38:0/2520919098 conn(0x55edeca71660 msgr2=0x55edeca7d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff49cff9640 0 client.16110 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 shutdown_connections 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 1 -- 172.21.15.38:0/2520919098 wait complete. 2024-04-22T00:02:48.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.329+0000 7ff4be230740 20 client.16110 trim_cache size 0 max 16384 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "eb44f82e-bb21-44db-bb92-d4c5686d315a": { 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "eb44f82e-bb21-44db-bb92-d4c5686d315a", 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:02:48.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:02:48.330 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:02:48.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 msgr2=0x7ff4a00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 0x7ff4a00a52e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff4b00058c0 tx=0x7ff4b000a2d0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 msgr2=0x7ff4b419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005940 tx=0x7ff4a402cd10 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 shutdown_connections 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4a00a2e20 0x7ff4a00a52e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4b410dfc0 0x7ff4b419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b4074430 0x7ff4b419a800 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 --2- 172.21.15.38:0/1778820565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4b4073b10 0x7ff4b419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 >> 172.21.15.38:0/1778820565 conn(0x7ff4b4078b40 msgr2=0x7ff4b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 shutdown_connections 2024-04-22T00:02:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:02:48.330+0000 7ff4bb572640 1 -- 172.21.15.38:0/1778820565 wait complete. 2024-04-22T00:02:48.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:47 smithi111 ceph-mon[31543]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:02:48.333 INFO:tasks.cephfs.filesystem:scrub status for tag:eb44f82e-bb21-44db-bb92-d4c5686d315a - {'path': '/', 'tag': 'eb44f82e-bb21-44db-bb92-d4c5686d315a', 'options': 'recursive,force'} 2024-04-22T00:02:49.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:48 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:02:49.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:48 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:48 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:02:49.280 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:48 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:49.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:48 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:02:49.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:48 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:49.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:48 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:02:49.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:48 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:49.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:48 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:02:49.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:48 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:48 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:02:49.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:48 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:02:50.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:49 smithi203 ceph-mon[31431]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:02:50.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:49 smithi111 ceph-mon[31543]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:02:50.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:49 smithi038 ceph-mon[27972]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:02:51.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:50 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:02:51.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:50 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:02:51.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:50 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:02:52.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:51 smithi203 ceph-mon[31431]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-22T00:02:52.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:51 smithi111 ceph-mon[31543]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-22T00:02:52.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:51 smithi038 ceph-mon[27972]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-22T00:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 -- 172.21.15.38:0/1410381603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242410df60 msgr2=0x7f24241103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 --2- 172.21.15.38:0/1410381603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242410df60 0x7f24241103b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f241c009ca0 tx=0x7f241c02c180 comp rx=0 tx=0).stop 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 -- 172.21.15.38:0/1410381603 shutdown_connections 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 --2- 172.21.15.38:0/1410381603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242410df60 0x7f24241103b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 --2- 172.21.15.38:0/1410381603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2424074490 0x7f242410da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 --2- 172.21.15.38:0/1410381603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424073b70 0x7f2424073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 -- 172.21.15.38:0/1410381603 >> 172.21.15.38:0/1410381603 conn(0x7f24240fd1f0 msgr2=0x7f24240ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 -- 172.21.15.38:0/1410381603 shutdown_connections 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.139+0000 7f242a337640 1 -- 172.21.15.38:0/1410381603 wait complete. 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 Processor -- start 2024-04-22T00:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 -- start start 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424074490 0x7f242419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 0x7f242419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2424113d00 con 0x7f2424073b70 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2424113b80 con 0x7f2424074490 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f242a337640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2424113e80 con 0x7f242410df60 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36480/0 (socket says 172.21.15.38:36480) 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 -- 172.21.15.38:0/2665250331 learned_addr learned my addr 172.21.15.38:0/2665250331 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f24288d4640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 0x7f242419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f24237fe640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424074490 0x7f242419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 -- 172.21.15.38:0/2665250331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 msgr2=0x7f242419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 0x7f242419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 -- 172.21.15.38:0/2665250331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424074490 msgr2=0x7f242419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424074490 0x7f242419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f2423fff640 1 -- 172.21.15.38:0/2665250331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242419f100 con 0x7f2424073b70 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.140+0000 7f24288d4640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 0x7f242419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f2423fff640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f241400e9f0 tx=0x7f241400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f24217fa640 1 -- 172.21.15.38:0/2665250331 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241400cd80 con 0x7f2424073b70 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242419f3f0 con 0x7f2424073b70 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f24217fa640 1 -- 172.21.15.38:0/2665250331 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2414010910 con 0x7f2424073b70 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f24217fa640 1 -- 172.21.15.38:0/2665250331 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24140188f0 con 0x7f2424073b70 2024-04-22T00:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.141+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24241a6c80 con 0x7f2424073b70 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.143+0000 7f24217fa640 1 -- 172.21.15.38:0/2665250331 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) ==== 138060+0+0 (secure 0 0 0) 0x7f2414010ac0 con 0x7f2424073b70 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f24217fa640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 0x7f240c0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f24237fe640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 0x7f240c0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f24217fa640 1 -- 172.21.15.38:0/2665250331 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f24140cff10 con 0x7f2424073b70 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f24237fe640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 0x7f240c0a5380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2418005d80 tx=0x7f2418005c90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f242cff9740 1 Processor -- start 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.144+0000 7f242cff9740 1 -- start start 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd86cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd89600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f20cce1b70 con 0x55f20cd7db20 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f20cc8a470 con 0x55f20cd87210 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f242cff9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f20ccddb30 con 0x55f20cd7e440 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36488/0 (socket says 172.21.15.38:36488) 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 -- 172.21.15.38:0/589548913 learned_addr learned my addr 172.21.15.38:0/589548913 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f24237fe640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd86cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f24288d4640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd89600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 -- 172.21.15.38:0/589548913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 msgr2=0x55f20cd86cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd86cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 -- 172.21.15.38:0/589548913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 msgr2=0x55f20cd89600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd89600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 -- 172.21.15.38:0/589548913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f20c9f6210 con 0x55f20cd7db20 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f24237fe640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd86cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f24288d4640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd89600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f2423fff640 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f241400e9c0 tx=0x7f241409c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb5c7852fa665dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.145+0000 7f240a7fc640 1 -- 172.21.15.38:0/589548913 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24140a4b10 con 0x55f20cd7db20 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f240a7fc640 1 -- 172.21.15.38:0/589548913 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241409a1a0 con 0x55f20cd7db20 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f240a7fc640 1 -- 172.21.15.38:0/589548913 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414018db0 con 0x55f20cd7db20 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 -- 172.21.15.38:0/589548913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 msgr2=0x55f20cd7df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f241400e9c0 tx=0x7f241409c660 comp rx=0 tx=0).stop 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 -- 172.21.15.38:0/589548913 shutdown_connections 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd89600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd86cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 --2- 172.21.15.38:0/589548913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd7df00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 -- 172.21.15.38:0/589548913 >> 172.21.15.38:0/589548913 conn(0x55f20cd78440 msgr2=0x55f20cd78820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 -- 172.21.15.38:0/589548913 shutdown_connections 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.146+0000 7f242cff9740 1 -- 172.21.15.38:0/589548913 wait complete. 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 -- start start 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 Processor -- start 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7db20 0x55f20cd842b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd84d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f20cd89de0 con 0x55f20cd7e440 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f20cc9da10 con 0x55f20cd87210 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f20cd89f60 con 0x55f20cd7db20 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f242cff9740 10 client.0 _ll_register_callbacks cb 0x55f20ccd8b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.147+0000 7f24237fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36490/0 (socket says 172.21.15.38:36490) 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 -- 172.21.15.38:0/3964009936 learned_addr learned my addr 172.21.15.38:0/3964009936 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f2423fff640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7db20 0x55f20cd842b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24288d4640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd84d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 -- 172.21.15.38:0/3964009936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7db20 msgr2=0x55f20cd842b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7db20 0x55f20cd842b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 -- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 msgr2=0x55f20cd84d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd84d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 -- 172.21.15.38:0/3964009936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f20cac2620 con 0x55f20cd7e440 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24237fe640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f24180117a0 tx=0x7f24180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c0fbca88a528eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24037fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418023070 con 0x55f20cd7e440 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2418010b50 con 0x55f20cd7e440 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f241409a1a0 con 0x55f20cd7e440 2024-04-22T00:02:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f241802e050 con 0x55f20cd7e440 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f242cff9740 10 client.16122 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.148+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f20cd96070 con 0x55f20cd7e440 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241409a1a0 con 0x55f20cd7e440 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f241800f040 con 0x55f20cd7e440 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 10 client.16122 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 20 client.16122 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20cd999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 4 client.16122 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f20ccef0b0 con 0x55f20cd97580 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.149+0000 7f2423fff640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20cd999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.150+0000 7f2423fff640 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20cd999b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d53589fe073470c server_cookie=f21c130159db0d6a in_seq=0 out_seq=0 2024-04-22T00:02:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.150+0000 7f24037fe640 10 client.16122 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.150+0000 7f24037fe640 1 -- 172.21.15.38:0/3964009936 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f20ccef0b0 con 0x55f20cd97580 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.150+0000 7f24037fe640 10 client.16122 handle_command_reply: tid=0 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.150+0000 7f242cff9740 1 client.16122 shutdown 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 msgr2=0x55f20cd847f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f24180117a0 tx=0x7f24180046c0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 shutdown_connections 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd84d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20cd999b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7e440 0x55f20cd847f0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f24037fe640 0 client.16122 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 --2- 172.21.15.38:0/3964009936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7db20 0x55f20cd842b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 >> 172.21.15.38:0/3964009936 conn(0x55f20cd78440 msgr2=0x55f20cd7a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 shutdown_connections 2024-04-22T00:02:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 1 -- 172.21.15.38:0/3964009936 wait complete. 2024-04-22T00:02:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.151+0000 7f242cff9740 20 client.16122 trim_cache size 0 max 16384 2024-04-22T00:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 Processor -- start 2024-04-22T00:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 -- start start 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd937a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd92ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f24241b0d40 con 0x55f20cd7db20 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f24241b2880 con 0x55f20cd87210 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f242cff9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24241b1410 con 0x55f20cd7e440 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36492/0 (socket says 172.21.15.38:36492) 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 -- 172.21.15.38:0/1832172404 learned_addr learned my addr 172.21.15.38:0/1832172404 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f24237fe640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd937a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f24288d4640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd92ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 -- 172.21.15.38:0/1832172404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 msgr2=0x55f20cd937a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd937a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 -- 172.21.15.38:0/1832172404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 msgr2=0x55f20cd92ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd92ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.172+0000 7f2423fff640 1 -- 172.21.15.38:0/1832172404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f20cd82800 con 0x55f20cd7db20 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.173+0000 7f24288d4640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd92ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.173+0000 7f2423fff640 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f241409be30 tx=0x7f2414098270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d79170b3bf44f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.173+0000 7f2409ffb640 1 -- 172.21.15.38:0/1832172404 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414098510 con 0x55f20cd7db20 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.173+0000 7f2409ffb640 1 -- 172.21.15.38:0/1832172404 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24140cd480 con 0x55f20cd7db20 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.173+0000 7f2409ffb640 1 -- 172.21.15.38:0/1832172404 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241409b550 con 0x55f20cd7db20 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- 172.21.15.38:0/1832172404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 msgr2=0x55f20cd869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f241409be30 tx=0x7f2414098270 comp rx=0 tx=0).stop 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- 172.21.15.38:0/1832172404 shutdown_connections 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd87210 0x55f20cd92ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd7e440 0x55f20cd937a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- 172.21.15.38:0/1832172404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd869e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- 172.21.15.38:0/1832172404 >> 172.21.15.38:0/1832172404 conn(0x55f20cd78440 msgr2=0x55f20cd78820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- 172.21.15.38:0/1832172404 shutdown_connections 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- 172.21.15.38:0/1832172404 wait complete. 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 -- start start 2024-04-22T00:02:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 Processor -- start 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 0x55f20cd8b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.174+0000 7f242cff9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 0x55f20cd8b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f24241b21b0 con 0x55f20cd7db20 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f24241b1ae0 con 0x55f20cd7e440 2024-04-22T00:02:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24241aed20 con 0x55f20cd87210 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 10 client.0 _ll_register_callbacks cb 0x7f241800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36508/0 (socket says 172.21.15.38:36508) 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 -- 172.21.15.38:0/659682826 learned_addr learned my addr 172.21.15.38:0/659682826 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24237fe640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 0x55f20cd8b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24288d4640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 0x55f20cd8b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 -- 172.21.15.38:0/659682826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 msgr2=0x55f20cd8b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 0x55f20cd8b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 -- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 msgr2=0x55f20cd8b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 0x55f20cd8b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 -- 172.21.15.38:0/659682826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f20cdfce00 con 0x55f20cd7db20 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24237fe640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 0x55f20cd8b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:02:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24288d4640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 0x55f20cd8b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f2423fff640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f24140a45e0 tx=0x7f241409a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234a7355cc8abeb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24037fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414098510 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f241409b2f0 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24140cd480 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f24140986b0 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 10 client.16128 fetch_fsmap learned FSMap version 28 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.175+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f20cdfe080 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24140cd480 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2414097d60 con 0x55f20cd7db20 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 10 client.16128 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 20 client.16128 populate_metadata read hostname 'smithi038' 2024-04-22T00:02:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20ce00310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:02:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 4 client.16128 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:02:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f20cd73900 con 0x55f20cd97580 2024-04-22T00:02:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.176+0000 7f24237fe640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20ce00310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:02:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.177+0000 7f24237fe640 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20ce00310 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54d27cf53acb1d12 server_cookie=7864e4866e203d4 in_seq=0 out_seq=0 2024-04-22T00:02:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.177+0000 7f24037fe640 10 client.16128 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f24037fe640 1 -- 172.21.15.38:0/659682826 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f20cd73900 con 0x55f20cd97580 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f24037fe640 10 client.16128 handle_command_reply: tid=0 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 client.16128 shutdown 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 msgr2=0x55f20cd8ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f24140a45e0 tx=0x7f241409a1a0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 shutdown_connections 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f20cd87210 0x55f20cd8b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f20cd7e440 0x55f20cd8b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f20cd97580 0x55f20ce00310 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 --2- 172.21.15.38:0/659682826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f20cd7db20 0x55f20cd8ae90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f24037fe640 0 client.16128 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:02:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 >> 172.21.15.38:0/659682826 conn(0x55f20cd78440 msgr2=0x55f20cd76c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 shutdown_connections 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 1 -- 172.21.15.38:0/659682826 wait complete. 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.184+0000 7f242cff9740 20 client.16128 trim_cache size 0 max 16384 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:02:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:02:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 msgr2=0x7f240c0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 0x7f240c0a5380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2418005d80 tx=0x7f2418005c90 comp rx=0 tx=0).stop 2024-04-22T00:02:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 msgr2=0x7f242419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:02:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f241400e9f0 tx=0x7f241400eeb0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 shutdown_connections 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f240c0a2ec0 0x7f240c0a5380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242410df60 0x7f242419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2424074490 0x7f242419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 --2- 172.21.15.38:0/2665250331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2424073b70 0x7f242419a160 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 >> 172.21.15.38:0/2665250331 conn(0x7f24240fd1f0 msgr2=0x7f24240fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 shutdown_connections 2024-04-22T00:02:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:02:54.185+0000 7f242a337640 1 -- 172.21.15.38:0/2665250331 wait complete. 2024-04-22T00:02:54.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:53 smithi203 ceph-mon[31431]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T00:02:54.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:53 smithi111 ceph-mon[31543]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T00:02:54.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:53 smithi038 ceph-mon[27972]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-22T00:02:56.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:55 smithi203 ceph-mon[31431]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-22T00:02:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:55 smithi111 ceph-mon[31543]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-22T00:02:56.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:55 smithi038 ceph-mon[27972]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-22T00:02:58.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:02:58 smithi038 ceph-mon[27972]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-04-22T00:02:58.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:02:58 smithi203 ceph-mon[31431]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-04-22T00:02:58.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:02:58 smithi111 ceph-mon[31543]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-04-22T00:03:00.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:00 smithi203 ceph-mon[31431]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:03:00.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:00 smithi111 ceph-mon[31543]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:03:00.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-mon[27972]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-22T00:03:01.207 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:03:00.884+0000 7fea465d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:03:01.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:03:00.884+0000 7fea465d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:03:01.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:03:00.884+0000 7fea465d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:03:01.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:03:00.884+0000 7fea465d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:03:01.208 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:03:00 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:03:00.884+0000 7fea465d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T00:03:02.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:02 smithi203 ceph-mon[31431]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T00:03:02.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:02 smithi111 ceph-mon[31543]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T00:03:02.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:02 smithi038 ceph-mon[27972]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-22T00:03:04.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:04 smithi203 ceph-mon[31431]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:03:04.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:04 smithi203 ceph-mon[31431]: mgrmap e25: x(active, since 58s), standbys: z, y 2024-04-22T00:03:04.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:04 smithi111 ceph-mon[31543]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:03:04.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:04 smithi111 ceph-mon[31543]: mgrmap e25: x(active, since 58s), standbys: z, y 2024-04-22T00:03:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:04 smithi038 ceph-mon[27972]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-04-22T00:03:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:04 smithi038 ceph-mon[27972]: mgrmap e25: x(active, since 58s), standbys: z, y 2024-04-22T00:03:06.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:06 smithi203 ceph-mon[31431]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:03:06.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:06 smithi111 ceph-mon[31543]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:03:06.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:06 smithi038 ceph-mon[27972]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-22T00:03:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 -- 172.21.15.38:0/22336513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060101030 msgr2=0x7f906010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 --2- 172.21.15.38:0/22336513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060101030 0x7f906010d4f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9058009cd0 tx=0x7f905802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 -- 172.21.15.38:0/22336513 shutdown_connections 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 --2- 172.21.15.38:0/22336513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060101030 0x7f906010d4f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 --2- 172.21.15.38:0/22336513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060100670 0x7f9060100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 --2- 172.21.15.38:0/22336513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 0x7f9060106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 -- 172.21.15.38:0/22336513 >> 172.21.15.38:0/22336513 conn(0x7f90600777c0 msgr2=0x7f9060077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 -- 172.21.15.38:0/22336513 shutdown_connections 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 -- 172.21.15.38:0/22336513 wait complete. 2024-04-22T00:03:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.320+0000 7f9065d65640 1 Processor -- start 2024-04-22T00:03:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 -- start start 2024-04-22T00:03:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060101030 0x7f9060197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 0x7f90601975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9060110a00 con 0x7f9060100670 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9060110880 con 0x7f9060106630 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f9065d65640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9060110b80 con 0x7f9060101030 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35538/0 (socket says 172.21.15.38:35538) 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f905f7fe640 1 -- 172.21.15.38:0/2891529237 learned_addr learned my addr 172.21.15.38:0/2891529237 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f905effd640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060101030 0x7f9060197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.321+0000 7f905ffff640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 0x7f90601975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 -- 172.21.15.38:0/2891529237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060101030 msgr2=0x7f9060197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060101030 0x7f9060197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 -- 172.21.15.38:0/2891529237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 msgr2=0x7f90601975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 0x7f90601975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 -- 172.21.15.38:0/2891529237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90601a63c0 con 0x7f9060100670 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905f7fe640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f905000e9f0 tx=0x7f905000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905cff9640 1 -- 172.21.15.38:0/2891529237 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905000cd80 con 0x7f9060100670 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905cff9640 1 -- 172.21.15.38:0/2891529237 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9050010910 con 0x7f9060100670 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90601a66b0 con 0x7f9060100670 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f905cff9640 1 -- 172.21.15.38:0/2891529237 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050018990 con 0x7f9060100670 2024-04-22T00:03:07.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.322+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90601a6b10 con 0x7f9060100670 2024-04-22T00:03:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.325+0000 7f9068a2b740 1 Processor -- start 2024-04-22T00:03:07.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.325+0000 7f9068a2b740 1 -- start start 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905cff9640 1 -- 172.21.15.38:0/2891529237 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f9050010ac0 con 0x7f9060100670 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 0x561e7d221cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d224600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561e7d17cb70 con 0x561e7d218b20 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35552/0 (socket says 172.21.15.38:35552) 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905ffff640 1 -- 172.21.15.38:0/1141818113 learned_addr learned my addr 172.21.15.38:0/1141818113 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561e7d125470 con 0x561e7d219440 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e7d178b30 con 0x561e7d222210 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905f7fe640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 0x561e7d221cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.326+0000 7f905effd640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d224600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 -- 172.21.15.38:0/1141818113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 msgr2=0x561e7d224600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d224600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 -- 172.21.15.38:0/1141818113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 msgr2=0x561e7d221cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 0x561e7d221cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 -- 172.21.15.38:0/1141818113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7ce91210 con 0x561e7d218b20 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905f7fe640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 0x561e7d221cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905ffff640 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9058004af0 tx=0x7f9058005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44acd8dc4d1891d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905cff9640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 0x7f90480a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f9045ffb640 1 -- 172.21.15.38:0/1141818113 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058043070 con 0x561e7d218b20 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f9045ffb640 1 -- 172.21.15.38:0/1141818113 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905802cca0 con 0x561e7d218b20 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f9045ffb640 1 -- 172.21.15.38:0/1141818113 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058039040 con 0x561e7d218b20 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.327+0000 7f905f7fe640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 0x7f90480a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f905cff9640 1 -- 172.21.15.38:0/2891529237 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f90500cfbe0 con 0x7f9060100670 2024-04-22T00:03:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 msgr2=0x561e7d218f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9058004af0 tx=0x7f9058005f80 comp rx=0 tx=0).stop 2024-04-22T00:03:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f905f7fe640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 0x7f90480a5560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90500cce10 tx=0x7f90500d3690 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 shutdown_connections 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d224600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d219440 0x561e7d221cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1141818113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d218b20 0x561e7d218f00 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 >> 172.21.15.38:0/1141818113 conn(0x561e7d213440 msgr2=0x561e7d213820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 shutdown_connections 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- 172.21.15.38:0/1141818113 wait complete. 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.328+0000 7f9068a2b740 1 -- start start 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 Processor -- start 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 0x561e7d21f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d21fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561e7d224de0 con 0x561e7d219440 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561e7d138a10 con 0x561e7d218b20 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e7d224f60 con 0x561e7d222210 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f9068a2b740 10 client.0 _ll_register_callbacks cb 0x561e7d173b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35566/0 (socket says 172.21.15.38:35566) 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f905effd640 1 -- 172.21.15.38:0/3399499578 learned_addr learned my addr 172.21.15.38:0/3399499578 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.329+0000 7f905ffff640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 0x561e7d21f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 -- 172.21.15.38:0/3399499578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 msgr2=0x561e7d21fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d21fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 -- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 msgr2=0x561e7d21f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 0x561e7d21f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 -- 172.21.15.38:0/3399499578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7cf5d620 con 0x561e7d219440 2024-04-22T00:03:07.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905ffff640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 0x561e7d21f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f905effd640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f905400bb00 tx=0x7f90540044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2b47728a12ff9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f903effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054003e20 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9054010440 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905802cca0 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9054010660 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f9068a2b740 10 client.16146 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.330+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e7d2310c0 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905802cca0 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f90540239d0 con 0x561e7d219440 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 10 client.16146 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 20 client.16146 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d232640 0x561e7d234a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 4 client.16146 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e7d18a0b0 con 0x561e7d232640 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.331+0000 7f905ffff640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d232640 0x561e7d234a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.332+0000 7f903effd640 10 client.16146 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.332+0000 7f905ffff640 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d232640 0x561e7d234a30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5721d09ffd898e39 server_cookie=5d4e18756f1898c5 in_seq=0 out_seq=0 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.332+0000 7f903effd640 1 -- 172.21.15.38:0/3399499578 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e7d18a0b0 con 0x561e7d232640 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.332+0000 7f903effd640 10 client.16146 handle_command_reply: tid=0 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.332+0000 7f9068a2b740 1 client.16146 shutdown 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 msgr2=0x561e7d21f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f905400bb00 tx=0x7f90540044c0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 shutdown_connections 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d222210 0x561e7d21fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d232640 0x561e7d234a30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f903effd640 0 client.16146 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21f790 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 --2- 172.21.15.38:0/3399499578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d218b20 0x561e7d21f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 >> 172.21.15.38:0/3399499578 conn(0x561e7d213440 msgr2=0x561e7d215190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 shutdown_connections 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 1 -- 172.21.15.38:0/3399499578 wait complete. 2024-04-22T00:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.333+0000 7f9068a2b740 20 client.16146 trim_cache size 0 max 16384 2024-04-22T00:03:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 Processor -- start 2024-04-22T00:03:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 -- start start 2024-04-22T00:03:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d221980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d2199a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90601ac7c0 con 0x561e7d219440 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90601ae300 con 0x561e7d222210 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.348+0000 7f9068a2b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f90601ace90 con 0x561e7d218b20 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35574/0 (socket says 172.21.15.38:35574) 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 -- 172.21.15.38:0/1271225742 learned_addr learned my addr 172.21.15.38:0/1271225742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905ffff640 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d221980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905ffff640 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d2199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 -- 172.21.15.38:0/1271225742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 msgr2=0x561e7d221980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d221980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 -- 172.21.15.38:0/1271225742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 msgr2=0x561e7d2199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d2199a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 -- 172.21.15.38:0/1271225742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e7d212c40 con 0x561e7d219440 2024-04-22T00:03:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f905effd640 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9054023ec0 tx=0x7f90540043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989e3236916ecf58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f90457fa640 1 -- 172.21.15.38:0/1271225742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054035020 con 0x561e7d219440 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f90457fa640 1 -- 172.21.15.38:0/1271225742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9054003e20 con 0x561e7d219440 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.349+0000 7f90457fa640 1 -- 172.21.15.38:0/1271225742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054023920 con 0x561e7d219440 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- 172.21.15.38:0/1271225742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 msgr2=0x561e7d22dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9054023ec0 tx=0x7f90540043e0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- 172.21.15.38:0/1271225742 shutdown_connections 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d2199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d22dca0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1271225742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d221980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- 172.21.15.38:0/1271225742 >> 172.21.15.38:0/1271225742 conn(0x561e7d213440 msgr2=0x561e7d20e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- 172.21.15.38:0/1271225742 shutdown_connections 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- 172.21.15.38:0/1271225742 wait complete. 2024-04-22T00:03:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.350+0000 7f9068a2b740 1 -- start start 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 Processor -- start 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d22a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d21c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90601adc30 con 0x561e7d219440 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90601ad560 con 0x561e7d222210 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f90601aa7e0 con 0x561e7d218b20 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f9068a2b740 10 client.0 _ll_register_callbacks cb 0x7f9054004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35576/0 (socket says 172.21.15.38:35576) 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 -- 172.21.15.38:0/1788635377 learned_addr learned my addr 172.21.15.38:0/1788635377 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905ffff640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d22a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905ffff640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d21c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 -- 172.21.15.38:0/1788635377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 msgr2=0x561e7d22a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d22a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 -- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 msgr2=0x561e7d21c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d21c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.351+0000 7f905effd640 1 -- 172.21.15.38:0/1788635377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e7d21c700 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f905effd640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f905400bb00 tx=0x7f9054023810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e374036e1d71252 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540104c0 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9054003e20 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9054023a30 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9054010d90 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 10 client.16152 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e7d298a70 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9054003e20 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9054004a60 con 0x561e7d219440 2024-04-22T00:03:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 10 client.16152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 20 client.16152 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d298f50 0x561e7d29b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.352+0000 7f9068a2b740 4 client.16152 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.353+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e7d184400 con 0x561e7d298f50 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.353+0000 7f905ffff640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d298f50 0x561e7d29b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.353+0000 7f903effd640 10 client.16152 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.353+0000 7f905ffff640 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d298f50 0x561e7d29b370 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cf502b49f367b36 server_cookie=238ffd4a763cfed0 in_seq=0 out_seq=0 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f903effd640 1 -- 172.21.15.38:0/1788635377 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e7d184400 con 0x561e7d298f50 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f903effd640 10 client.16152 handle_command_reply: tid=0 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 client.16152 shutdown 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 msgr2=0x561e7d21bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f905400bb00 tx=0x7f9054023810 comp rx=0 tx=0).stop 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 shutdown_connections 2024-04-22T00:03:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561e7d222210 0x561e7d21c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561e7d298f50 0x561e7d29b370 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561e7d219440 0x561e7d21bb00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 --2- 172.21.15.38:0/1788635377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e7d218b20 0x561e7d22a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 >> 172.21.15.38:0/1788635377 conn(0x561e7d213440 msgr2=0x561e7d2199a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.359+0000 7f903effd640 0 client.16152 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 shutdown_connections 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9068a2b740 1 -- 172.21.15.38:0/1788635377 wait complete. 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9068a2b740 20 client.16152 trim_cache size 0 max 16384 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:03:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:03:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:03:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 msgr2=0x7f90480a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 0x7f90480a5560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90500cce10 tx=0x7f90500d3690 comp rx=0 tx=0).stop 2024-04-22T00:03:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 msgr2=0x7f9060196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f905000e9f0 tx=0x7f905000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 shutdown_connections 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90480a30a0 0x7f90480a5560 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9060106630 0x7f90601975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9060101030 0x7f9060197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 --2- 172.21.15.38:0/2891529237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9060100670 0x7f9060196b50 secure :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f905000e9f0 tx=0x7f905000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.360+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 >> 172.21.15.38:0/2891529237 conn(0x7f90600777c0 msgr2=0x7f90601030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.361+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 shutdown_connections 2024-04-22T00:03:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:07.361+0000 7f9065d65640 1 -- 172.21.15.38:0/2891529237 wait complete. 2024-04-22T00:03:08.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:08 smithi203 ceph-mon[31431]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:08.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:08 smithi111 ceph-mon[31543]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:08.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:08 smithi038 ceph-mon[27972]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:10.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:10 smithi111 ceph-mon[31543]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:10.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:10 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:03:10.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:10 smithi038 ceph-mon[27972]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:10.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:10 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:03:10.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:10 smithi203 ceph-mon[31431]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:10.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:10 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:03:11.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:03:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:11 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:03:11.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:03:11.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:11 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:11.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:03:11.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:03:11.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:11 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:03:12.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:12 smithi038 ceph-mon[27972]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:12.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:12 smithi203 ceph-mon[31431]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:12.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:12 smithi111 ceph-mon[31543]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:14.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:14 smithi038 ceph-mon[27972]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-22T00:03:14.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:14 smithi203 ceph-mon[31431]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-22T00:03:14.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:14 smithi111 ceph-mon[31543]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-04-22T00:03:16.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:16 smithi038 ceph-mon[27972]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:16.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:16 smithi203 ceph-mon[31431]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:16.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:16 smithi111 ceph-mon[31543]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-22T00:03:18.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:03:18.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 -- 172.21.15.38:0/2998823499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 msgr2=0x7fe850100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 --2- 172.21.15.38:0/2998823499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850100a90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009930 tx=0x7fe83c02bc20 comp rx=0 tx=0).stop 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 -- 172.21.15.38:0/2998823499 shutdown_connections 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 --2- 172.21.15.38:0/2998823499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe85010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 --2- 172.21.15.38:0/2998823499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850100a90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 --2- 172.21.15.38:0/2998823499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe850106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 -- 172.21.15.38:0/2998823499 >> 172.21.15.38:0/2998823499 conn(0x7fe850077740 msgr2=0x7fe850077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 -- 172.21.15.38:0/2998823499 shutdown_connections 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.459+0000 7fe8571fe640 1 -- 172.21.15.38:0/2998823499 wait complete. 2024-04-22T00:03:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 Processor -- start 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 -- start start 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe8501985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe85019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe850110aa0 con 0x7fe850100610 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe850110920 con 0x7fe850100fd0 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe8571fe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe850110c20 con 0x7fe8501066b0 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe854f9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe85579b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe85019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe854f9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46666/0 (socket says 172.21.15.38:46666) 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe854f9a640 1 -- 172.21.15.38:0/3461483110 learned_addr learned my addr 172.21.15.38:0/3461483110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.460+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe8501985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 -- 172.21.15.38:0/3461483110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 msgr2=0x7fe85019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe85019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 -- 172.21.15.38:0/3461483110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 msgr2=0x7fe8501985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe8501985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 -- 172.21.15.38:0/3461483110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85019d070 con 0x7fe850100610 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe85579b640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe85019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe84ffff640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe8501985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe854f9a640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe84400b450 tx=0x7fe84400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe84dffb640 1 -- 172.21.15.38:0/3461483110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844015020 con 0x7fe850100610 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85019d360 con 0x7fe850100610 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe84dffb640 1 -- 172.21.15.38:0/3461483110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe844011d70 con 0x7fe850100610 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe84dffb640 1 -- 172.21.15.38:0/3461483110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84400f340 con 0x7fe850100610 2024-04-22T00:03:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.461+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8501074b0 con 0x7fe850100610 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe84dffb640 1 -- 172.21.15.38:0/3461483110 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fe844018020 con 0x7fe850100610 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 Processor -- start 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 -- start start 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56196007ef30 0x56196007f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 0x561960087ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56195ffe3560 con 0x561960088430 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56195ff8b730 con 0x56196007f850 2024-04-22T00:03:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.464+0000 7fe859de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56195ffdf520 con 0x56196007ef30 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84dffb640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 0x7fe8400a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46674/0 (socket says 172.21.15.38:46674) 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe85579b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56196007ef30 0x56196007f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 -- 172.21.15.38:0/953240973 learned_addr learned my addr 172.21.15.38:0/953240973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe85579b640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 0x7fe8400a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84dffb640 1 -- 172.21.15.38:0/3461483110 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe8440cf1b0 con 0x7fe850100610 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe854f9a640 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 0x561960087ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 -- 172.21.15.38:0/953240973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56196007ef30 msgr2=0x56196007f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56196007ef30 0x56196007f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 -- 172.21.15.38:0/953240973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 msgr2=0x561960087ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 0x561960087ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 -- 172.21.15.38:0/953240973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195fdc17d0 con 0x561960088430 2024-04-22T00:03:18.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe854f9a640 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 0x561960087ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.465+0000 7fe84ffff640 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe83c000c00 tx=0x7fe83c02c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1192c0742ce75135 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe85579b640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 0x7fe8400a5370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe848009fd0 tx=0x7fe848009f60 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe83affd640 1 -- 172.21.15.38:0/953240973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c045070 con 0x561960088430 2024-04-22T00:03:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe83affd640 1 -- 172.21.15.38:0/953240973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c004bb0 con 0x561960088430 2024-04-22T00:03:18.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe83affd640 1 -- 172.21.15.38:0/953240973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c005240 con 0x561960088430 2024-04-22T00:03:18.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe859de2740 1 -- 172.21.15.38:0/953240973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 msgr2=0x56196008a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.466+0000 7fe859de2740 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe83c000c00 tx=0x7fe83c02c670 comp rx=0 tx=0).stop 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- 172.21.15.38:0/953240973 shutdown_connections 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561960088430 0x56196008a820 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007f850 0x561960087ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- 172.21.15.38:0/953240973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56196007ef30 0x56196007f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- 172.21.15.38:0/953240973 >> 172.21.15.38:0/953240973 conn(0x561960079660 msgr2=0x561960079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- 172.21.15.38:0/953240973 shutdown_connections 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- 172.21.15.38:0/953240973 wait complete. 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- start start 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 Processor -- start 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007ef30 0x5619600854b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960085f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56196008b1c0 con 0x56196007f850 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56195ff9ee00 con 0x56196007ef30 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56196008b340 con 0x561960088430 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.467+0000 7fe859de2740 10 client.0 _ll_register_callbacks cb 0x56195ffd9eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46682/0 (socket says 172.21.15.38:46682) 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 -- 172.21.15.38:0/475696471 learned_addr learned my addr 172.21.15.38:0/475696471 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe85579b640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960085f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe854f9a640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007ef30 0x5619600854b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 -- 172.21.15.38:0/475696471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 msgr2=0x561960085f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960085f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 -- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007ef30 msgr2=0x5619600854b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007ef30 0x5619600854b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 -- 172.21.15.38:0/475696471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195fcf73f0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe84ffff640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe83c004270 tx=0x7fe83c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b4a88e027ccfc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe82ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c045070 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.468+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe83c0051c0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83c004bb0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe83c058050 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 10 client.16164 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56195fcb70f0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c004bb0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe83c003ce0 con 0x56196007f850 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 10 client.16164 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:18.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 20 client.16164 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600986d0 0x56196009aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 4 client.16164 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.469+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56195fff0aa0 con 0x5619600986d0 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.470+0000 7fe854f9a640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600986d0 0x56196009aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.470+0000 7fe82ffff640 10 client.16164 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.470+0000 7fe854f9a640 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600986d0 0x56196009aac0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e22fe83ee5b4d7b6 server_cookie=5f13f4d7321074eb in_seq=0 out_seq=0 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe82ffff640 1 -- 172.21.15.38:0/475696471 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56195fff0aa0 con 0x5619600986d0 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe82ffff640 10 client.16164 handle_command_reply: tid=0 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 client.16164 shutdown 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 msgr2=0x5619600859f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe83c004270 tx=0x7fe83c005fd0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 shutdown_connections 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960085f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600986d0 0x56196009aac0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007f850 0x5619600859f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 --2- 172.21.15.38:0/475696471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56196007ef30 0x5619600854b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 >> 172.21.15.38:0/475696471 conn(0x561960079660 msgr2=0x56196007b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe82ffff640 0 client.16164 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 shutdown_connections 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 1 -- 172.21.15.38:0/475696471 wait complete. 2024-04-22T00:03:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.471+0000 7fe859de2740 20 client.16164 trim_cache size 0 max 16384 2024-04-22T00:03:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 Processor -- start 2024-04-22T00:03:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 -- start start 2024-04-22T00:03:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960079660 0x561960094080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960088430 0x5619600a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501b0cf0 con 0x56196007ef30 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501b2830 con 0x561960088430 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe859de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501b13c0 con 0x561960079660 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46684/0 (socket says 172.21.15.38:46684) 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 -- 172.21.15.38:0/889329022 learned_addr learned my addr 172.21.15.38:0/889329022 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe85579b640 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960088430 0x5619600a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe854f9a640 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960079660 0x561960094080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 -- 172.21.15.38:0/889329022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960079660 msgr2=0x561960094080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960079660 0x561960094080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 -- 172.21.15.38:0/889329022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960088430 msgr2=0x5619600a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.489+0000 7fe84ffff640 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960088430 0x5619600a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.490+0000 7fe84ffff640 1 -- 172.21.15.38:0/889329022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561960095000 con 0x56196007ef30 2024-04-22T00:03:18.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.490+0000 7fe84ffff640 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe83c04a610 tx=0x7fe83c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb25e5251d398c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.490+0000 7fe83a7fc640 1 -- 172.21.15.38:0/889329022 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c045070 con 0x56196007ef30 2024-04-22T00:03:18.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.490+0000 7fe83a7fc640 1 -- 172.21.15.38:0/889329022 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c02cb80 con 0x56196007ef30 2024-04-22T00:03:18.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.490+0000 7fe83a7fc640 1 -- 172.21.15.38:0/889329022 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c04a9c0 con 0x56196007ef30 2024-04-22T00:03:18.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- 172.21.15.38:0/889329022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 msgr2=0x5619600a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe83c04a610 tx=0x7fe83c005a20 comp rx=0 tx=0).stop 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- 172.21.15.38:0/889329022 shutdown_connections 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960088430 0x5619600a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x5619600a0060 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 --2- 172.21.15.38:0/889329022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960079660 0x561960094080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- 172.21.15.38:0/889329022 >> 172.21.15.38:0/889329022 conn(0x5619600840f0 msgr2=0x5619600786e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- 172.21.15.38:0/889329022 shutdown_connections 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- 172.21.15.38:0/889329022 wait complete. 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 -- start start 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.491+0000 7fe859de2740 1 Processor -- start 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 0x561960082420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960082ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501b2160 con 0x56196007ef30 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501b1a90 con 0x561960079660 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8501aecd0 con 0x561960088430 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe859de2740 10 client.0 _ll_register_callbacks cb 0x7fe848012bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 0x561960082420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46696/0 (socket says 172.21.15.38:46696) 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe84ffff640 1 -- 172.21.15.38:0/822131066 learned_addr learned my addr 172.21.15.38:0/822131066 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe85579b640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960082ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 -- 172.21.15.38:0/822131066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 msgr2=0x561960082ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960082ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 -- 172.21.15.38:0/822131066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 msgr2=0x561960082960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe854f9a640 1 -- 172.21.15.38:0/822131066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619600fdb70 con 0x561960079660 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe85579b640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960082ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.492+0000 7fe84ffff640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe854f9a640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 0x561960082420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8440cc490 tx=0x7fe8440bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5455f4c65d1bb1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8440a3ed0 con 0x561960079660 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83c02cb80 con 0x561960079660 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe844010570 con 0x561960079660 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe8440b4240 con 0x561960079660 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 10 client.25682 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:18.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5619600feeb0 con 0x561960079660 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c02cb80 con 0x561960079660 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe8440bb140 con 0x561960079660 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 10 client.25682 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 10 client.25682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.493+0000 7fe859de2740 20 client.25682 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600ff600 0x561960101a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 4 client.25682 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56195ffeadf0 con 0x5619600ff600 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe84ffff640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600ff600 0x561960101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe82ffff640 10 client.25682 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe84ffff640 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600ff600 0x561960101a20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de37f7eb600ea101 server_cookie=5fbe94ea334c7418 in_seq=0 out_seq=0 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe82ffff640 1 -- 172.21.15.38:0/822131066 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56195ffeadf0 con 0x5619600ff600 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe82ffff640 10 client.25682 handle_command_reply: tid=0 2024-04-22T00:03:18.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 client.25682 shutdown 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 msgr2=0x561960082420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 0x561960082420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8440cc490 tx=0x7fe8440bb040 comp rx=0 tx=0).stop 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 shutdown_connections 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561960088430 0x561960082ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5619600ff600 0x561960101a20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56196007ef30 0x561960082960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe859de2740 1 --2- 172.21.15.38:0/822131066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561960079660 0x561960082420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.494+0000 7fe82ffff640 0 client.25682 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 >> 172.21.15.38:0/822131066 conn(0x5619600840f0 msgr2=0x561960077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 shutdown_connections 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe859de2740 1 -- 172.21.15.38:0/822131066 wait complete. 2024-04-22T00:03:18.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe859de2740 20 client.25682 trim_cache size 0 max 16384 2024-04-22T00:03:18.494 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:03:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:03:18.494 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:18.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 msgr2=0x7fe8400a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 0x7fe8400a5370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe848009fd0 tx=0x7fe848009f60 comp rx=0 tx=0).stop 2024-04-22T00:03:18.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 msgr2=0x7fe850198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe84400b450 tx=0x7fe84400b910 comp rx=0 tx=0).stop 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 shutdown_connections 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.495+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8400a2eb0 0x7fe8400a5370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8501066b0 0x7fe85019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe850100fd0 0x7fe8501985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 --2- 172.21.15.38:0/3461483110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe850100610 0x7fe850198080 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 >> 172.21.15.38:0/3461483110 conn(0x7fe850077740 msgr2=0x7fe8500fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 shutdown_connections 2024-04-22T00:03:18.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.496+0000 7fe8571fe640 1 -- 172.21.15.38:0/3461483110 wait complete. 2024-04-22T00:03:18.499 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:03:18.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3206061232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 msgr2=0x7f2e08102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3206061232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08102db0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009900 tx=0x7f2e0402bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3206061232 shutdown_connections 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3206061232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3206061232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 0x7f2e0810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3206061232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08102db0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3206061232 >> 172.21.15.38:0/3206061232 conn(0x7f2e08077740 msgr2=0x7f2e08077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3206061232 shutdown_connections 2024-04-22T00:03:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.611+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3206061232 wait complete. 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.612+0000 7f2e0fe15640 1 Processor -- start 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.612+0000 7f2e0fe15640 1 -- start start 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.612+0000 7f2e0fe15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.612+0000 7f2e0fe15640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 0x7f2e081985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.612+0000 7f2e0fe15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0fe15640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08111b60 con 0x7f2e081029d0 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0dbb1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0dbb1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46708/0 (socket says 172.21.15.38:46708) 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0dbb1640 1 -- 172.21.15.38:0/3181199114 learned_addr learned my addr 172.21.15.38:0/3181199114 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0fe15640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e081119e0 con 0x7f2e081032f0 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e08111ce0 con 0x7f2e0810bd10 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0e3b2640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.613+0000 7f2e0d3b0640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 0x7f2e081985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 -- 172.21.15.38:0/3181199114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 msgr2=0x7f2e081985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 0x7f2e081985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 -- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 msgr2=0x7f2e08198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0819d020 con 0x7f2e0810bd10 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0dbb1640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0e3b2640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0819c8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009fd0 tx=0x7f2e0000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0000ce20 con 0x7f2e0810bd10 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0819d310 con 0x7f2e0810bd10 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.614+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e00005020 con 0x7f2e0810bd10 2024-04-22T00:03:18.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.615+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00010720 con 0x7f2e0810bd10 2024-04-22T00:03:18.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.615+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e081a4ba0 con 0x7f2e0810bd10 2024-04-22T00:03:18.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.616+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e080690d0 con 0x7f2e0810bd10 2024-04-22T00:03:18.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.618+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f2e000051d0 con 0x7f2e0810bd10 2024-04-22T00:03:18.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.618+0000 7f2dfeffd640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 0x7f2df40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:18.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.619+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2e000cffa0 con 0x7f2e0810bd10 2024-04-22T00:03:18.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.619+0000 7f2e0dbb1640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 0x7f2df40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:18.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.619+0000 7f2e0dbb1640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 0x7f2df40a52d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e04005c40 tx=0x7f2e04005b50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:18.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.623+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2e0009c640 con 0x7f2e0810bd10 2024-04-22T00:03:18.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:18 smithi038 ceph-mon[27972]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:18.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:18 smithi203 ceph-mon[31431]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:18.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:18 smithi111 ceph-mon[31543]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-04-22T00:03:18.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.896+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e081993b0 con 0x7f2e0810bd10 2024-04-22T00:03:18.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.897+0000 7f2dfeffd640 1 -- 172.21.15.38:0/3181199114 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f2e081993b0 con 0x7f2e0810bd10 2024-04-22T00:03:18.890 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:03:18.890 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:02:03.216530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:03:18.890 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 28 2024-04-22T00:03:18.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 msgr2=0x7f2df40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 0x7f2df40a52d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e04005c40 tx=0x7f2e04005b50 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 msgr2=0x7f2e0819c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0819c8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009fd0 tx=0x7f2e0000ef10 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 shutdown_connections 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2df40a2e10 0x7f2df40a52d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0810bd10 0x7f2e0819c8f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e081032f0 0x7f2e081985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 --2- 172.21.15.38:0/3181199114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e081029d0 0x7f2e08198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 >> 172.21.15.38:0/3181199114 conn(0x7f2e08077740 msgr2=0x7f2e080fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 shutdown_connections 2024-04-22T00:03:18.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:18.899+0000 7f2e0fe15640 1 -- 172.21.15.38:0/3181199114 wait complete. 2024-04-22T00:03:18.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/2777209238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 msgr2=0x7fbd58108d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/2777209238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd58108d30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009900 tx=0x7fbd4802bc30 comp rx=0 tx=0).stop 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/2777209238 shutdown_connections 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/2777209238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58114890 0x7fbd58116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/2777209238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58109270 0x7fbd58114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/2777209238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd58108d30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/2777209238 >> 172.21.15.38:0/2777209238 conn(0x7fbd5806c850 msgr2=0x7fbd5806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/2777209238 shutdown_connections 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.029+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/2777209238 wait complete. 2024-04-22T00:03:19.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 Processor -- start 2024-04-22T00:03:19.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 -- start start 2024-04-22T00:03:19.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 0x7fbd5810a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 0x7fbd5810ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5811a360 con 0x7fbd58108950 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5811a1e0 con 0x7fbd58109270 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd5e0ed640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5811a4e0 con 0x7fbd58114890 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46736/0 (socket says 172.21.15.38:46736) 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd577fe640 1 -- 172.21.15.38:0/3174151733 learned_addr learned my addr 172.21.15.38:0/3174151733 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 0x7fbd5810ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.030+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 0x7fbd5810a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 -- 172.21.15.38:0/3174151733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 msgr2=0x7fbd5810ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 0x7fbd5810ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 -- 172.21.15.38:0/3174151733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 msgr2=0x7fbd5810a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 0x7fbd5810a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 -- 172.21.15.38:0/3174151733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd581aedc0 con 0x7fbd58108950 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 0x7fbd5810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 0x7fbd5810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbd4802c240 tx=0x7fbd48004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd54ff9640 1 -- 172.21.15.38:0/3174151733 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48043020 con 0x7fbd58108950 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd581af050 con 0x7fbd58108950 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd54ff9640 1 -- 172.21.15.38:0/3174151733 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd480054e0 con 0x7fbd58108950 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.031+0000 7fbd54ff9640 1 -- 172.21.15.38:0/3174151733 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48003cf0 con 0x7fbd58108950 2024-04-22T00:03:19.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.032+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd581af5f0 con 0x7fbd58108950 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.034+0000 7fbd60db5740 1 Processor -- start 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.034+0000 7fbd60db5740 1 -- start start 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.034+0000 7fbd54ff9640 1 -- 172.21.15.38:0/3174151733 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fbd48046020 con 0x7fbd58108950 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.034+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf020 0x5622bd7cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.034+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7d8520 0x5622bd7da910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd60db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5622bd733530 con 0x5622bd7cf940 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd60db5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622bd6db830 con 0x5622bd7d8520 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd60db5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5622bd72f4f0 con 0x5622bd7cf020 2024-04-22T00:03:19.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46746/0 (socket says 172.21.15.38:46746) 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 -- 172.21.15.38:0/1569407277 learned_addr learned my addr 172.21.15.38:0/1569407277 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7d8520 0x5622bd7da910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd57fff640 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf020 0x5622bd7cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 -- 172.21.15.38:0/1569407277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf020 msgr2=0x5622bd7cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf020 0x5622bd7cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 -- 172.21.15.38:0/1569407277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7d8520 msgr2=0x5622bd7da910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7d8520 0x5622bd7da910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 -- 172.21.15.38:0/1569407277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bd6ff2c0 con 0x5622bd7cf940 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd54ff9640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 0x7fbd400a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.035+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbd480eee50 tx=0x7fbd480f5cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fa41437dbfd60a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd3dffb640 1 -- 172.21.15.38:0/1569407277 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48101020 con 0x5622bd7cf940 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 0x7fbd400a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd3dffb640 1 -- 172.21.15.38:0/1569407277 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd480be380 con 0x5622bd7cf940 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd3dffb640 1 -- 172.21.15.38:0/1569407277 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd480bd5e0 con 0x5622bd7cf940 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd60db5740 1 -- 172.21.15.38:0/1569407277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 msgr2=0x5622bd7d7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.036+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbd480eee50 tx=0x7fbd480f5cb0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd54ff9640 1 -- 172.21.15.38:0/3174151733 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbd480f1bd0 con 0x7fbd58108950 2024-04-22T00:03:19.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 0x7fbd400a5370 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd50005fd0 tx=0x7fbd50009780 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 -- 172.21.15.38:0/1569407277 shutdown_connections 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7d8520 0x5622bd7da910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d7fe0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1569407277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf020 0x5622bd7cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 -- 172.21.15.38:0/1569407277 >> 172.21.15.38:0/1569407277 conn(0x5622bd7c9750 msgr2=0x5622bd7c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 -- 172.21.15.38:0/1569407277 shutdown_connections 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 -- 172.21.15.38:0/1569407277 wait complete. 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 -- start start 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.037+0000 7fbd60db5740 1 Processor -- start 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7d7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7d56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5622bd7db0d0 con 0x5622bd7d8520 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622bd6eef10 con 0x5622bd7cf020 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5622bd7db250 con 0x5622bd7cf940 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd60db5740 10 client.0 _ll_register_callbacks cb 0x5622bd446be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47894/0 (socket says 172.21.15.38:47894) 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 -- 172.21.15.38:0/1415584347 learned_addr learned my addr 172.21.15.38:0/1415584347 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd57fff640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7d56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7d7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 -- 172.21.15.38:0/1415584347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 msgr2=0x5622bd7d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7d7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 -- 172.21.15.38:0/1415584347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 msgr2=0x5622bd7d56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7d56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd56ffd640 1 -- 172.21.15.38:0/1415584347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bd6c0c80 con 0x5622bd7cf940 2024-04-22T00:03:19.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7d7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.038+0000 7fbd57fff640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7d56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00bb40 tx=0x7fbd4c0044f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbba6dd3ca1a5b19 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c01b640 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbd4c01b7e0 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd480be380 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd4c024050 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd60db5740 10 client.25576 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.039+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622bd4473f0 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd480be380 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbd4c019020 con 0x5622bd7cf940 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 10 client.25576 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 10 client.25576 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 20 client.25576 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd7e8760 0x5622bd7eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 4 client.25576 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622bd740a70 con 0x5622bd7e8760 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.040+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd7e8760 0x5622bd7eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.041+0000 7fbd36ffd640 10 client.25576 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:19.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.041+0000 7fbd577fe640 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd7e8760 0x5622bd7eaba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85b1506c616ad7e5 server_cookie=eead52e8af8ab7aa in_seq=0 out_seq=0 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.041+0000 7fbd36ffd640 1 -- 172.21.15.38:0/1415584347 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622bd740a70 con 0x5622bd7e8760 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.041+0000 7fbd36ffd640 10 client.25576 handle_command_reply: tid=0 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.041+0000 7fbd60db5740 1 client.25576 shutdown 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 msgr2=0x5622bd7d51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00bb40 tx=0x7fbd4c0044f0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 shutdown_connections 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd7e8760 0x5622bd7eaba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7d56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7d51b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 --2- 172.21.15.38:0/1415584347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7d7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 >> 172.21.15.38:0/1415584347 conn(0x5622bd7c9750 msgr2=0x5622bd7cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd36ffd640 0 client.25576 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 shutdown_connections 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 1 -- 172.21.15.38:0/1415584347 wait complete. 2024-04-22T00:03:19.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.042+0000 7fbd60db5740 20 client.25576 trim_cache size 0 max 16384 2024-04-22T00:03:19.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.069+0000 7fbd60db5740 1 Processor -- start 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 -- start start 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7d8520 0x5622bd7d78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581b9540 con 0x5622bd7cf940 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581bb080 con 0x5622bd7cf020 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd60db5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581b9c10 con 0x5622bd7d8520 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46760/0 (socket says 172.21.15.38:46760) 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 -- 172.21.15.38:0/162239788 learned_addr learned my addr 172.21.15.38:0/162239788 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd577fe640 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 -- 172.21.15.38:0/162239788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7d8520 msgr2=0x5622bd7d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7d8520 0x5622bd7d78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 -- 172.21.15.38:0/162239788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 msgr2=0x5622bd7e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd56ffd640 1 -- 172.21.15.38:0/162239788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622bd5117d0 con 0x5622bd7cf940 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.070+0000 7fbd577fe640 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.071+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c011d40 tx=0x7fbd4c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7168cbf0e8a9033d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.071+0000 7fbd3d7fa640 1 -- 172.21.15.38:0/162239788 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c02c020 con 0x5622bd7cf940 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.071+0000 7fbd3d7fa640 1 -- 172.21.15.38:0/162239788 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd4c011920 con 0x5622bd7cf940 2024-04-22T00:03:19.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.071+0000 7fbd3d7fa640 1 -- 172.21.15.38:0/162239788 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004680 con 0x5622bd7cf940 2024-04-22T00:03:19.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- 172.21.15.38:0/162239788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 msgr2=0x5622bd7d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c011d40 tx=0x7fbd4c005970 comp rx=0 tx=0).stop 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- 172.21.15.38:0/162239788 shutdown_connections 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7d8520 0x5622bd7d78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7cf940 0x5622bd7d0690 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 --2- 172.21.15.38:0/162239788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- 172.21.15.38:0/162239788 >> 172.21.15.38:0/162239788 conn(0x5622bd7c9750 msgr2=0x5622bd7cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- 172.21.15.38:0/162239788 shutdown_connections 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- 172.21.15.38:0/162239788 wait complete. 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.072+0000 7fbd60db5740 1 -- start start 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 Processor -- start 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7db460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7db9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581ba9b0 con 0x5622bd7d8520 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581ba2e0 con 0x5622bd7cf020 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd581b7520 con 0x5622bd7cf940 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd60db5740 10 client.0 _ll_register_callbacks cb 0x7fbd4c0043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd57fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd57fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46768/0 (socket says 172.21.15.38:46768) 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd57fff640 1 -- 172.21.15.38:0/3187179248 learned_addr learned my addr 172.21.15.38:0/3187179248 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd56ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7db9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.073+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7db460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 -- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 msgr2=0x5622bd7db9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7db9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 -- 172.21.15.38:0/3187179248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 msgr2=0x5622bd7db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7db460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 -- 172.21.15.38:0/3187179248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622bd84cd10 con 0x5622bd7d8520 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd56ffd640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd57fff640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fbd50021430 tx=0x7fbd50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf5d654c1bf8d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50004d20 con 0x5622bd7d8520 2024-04-22T00:03:19.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbd500106a0 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd4c011920 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd500108a0 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd60db5740 10 client.16194 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.074+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622bd84e020 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd4c011920 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbd50028070 con 0x5622bd7d8520 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 10 client.16194 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 10 client.16194 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:03:19.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 20 client.16194 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd84e7f0 0x5622bd850c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 4 client.16194 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622bd73adc0 con 0x5622bd84e7f0 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.075+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd84e7f0 0x5622bd850c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd36ffd640 10 client.16194 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd577fe640 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd84e7f0 0x5622bd850c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b4d427b055a59132 server_cookie=11db00b83ab089f in_seq=0 out_seq=0 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd36ffd640 1 -- 172.21.15.38:0/3187179248 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622bd73adc0 con 0x5622bd84e7f0 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd36ffd640 10 client.16194 handle_command_reply: tid=0 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd60db5740 1 client.16194 shutdown 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 msgr2=0x5622bd7dbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.076+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fbd50021430 tx=0x7fbd50004ab0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 shutdown_connections 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5622bd84e7f0 0x5622bd850c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622bd7d8520 0x5622bd7dbee0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622bd7cf940 0x5622bd7db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd36ffd640 0 client.16194 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 --2- 172.21.15.38:0/3187179248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622bd7cf020 0x5622bd7db460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 >> 172.21.15.38:0/3187179248 conn(0x5622bd7c9750 msgr2=0x5622bd7cb350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 shutdown_connections 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 1 -- 172.21.15.38:0/3187179248 wait complete. 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd60db5740 20 client.16194 trim_cache size 0 max 16384 2024-04-22T00:03:19.076 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 msgr2=0x7fbd400a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.077+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 0x7fbd400a5370 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd50005fd0 tx=0x7fbd50009780 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 msgr2=0x7fbd5810a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fbd4802c240 tx=0x7fbd48004430 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 shutdown_connections 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd400a2eb0 0x7fbd400a5370 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd58114890 0x7fbd5810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd58109270 0x7fbd5810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 --2- 172.21.15.38:0/3174151733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd58108950 0x7fbd5810a1b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 >> 172.21.15.38:0/3174151733 conn(0x7fbd5806c850 msgr2=0x7fbd58070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 shutdown_connections 2024-04-22T00:03:19.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.078+0000 7fbd5e0ed640 1 -- 172.21.15.38:0/3174151733 wait complete. 2024-04-22T00:03:19.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 -- 172.21.15.38:0/2333910110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 msgr2=0x7fc190102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 --2- 172.21.15.38:0/2333910110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190102da0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc180009900 tx=0x7fc18002bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 -- 172.21.15.38:0/2333910110 shutdown_connections 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 --2- 172.21.15.38:0/2333910110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc19010bd00 0x7fc19010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 --2- 172.21.15.38:0/2333910110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1901032e0 0x7fc19010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 --2- 172.21.15.38:0/2333910110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190102da0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 -- 172.21.15.38:0/2333910110 >> 172.21.15.38:0/2333910110 conn(0x7fc190077740 msgr2=0x7fc190077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 -- 172.21.15.38:0/2333910110 shutdown_connections 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.198+0000 7fc195c35640 1 -- 172.21.15.38:0/2333910110 wait complete. 2024-04-22T00:03:19.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 Processor -- start 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 -- start start 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1901032e0 0x7fc190198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 0x7fc19019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc18f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc18f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46780/0 (socket says 172.21.15.38:46780) 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc18f7fe640 1 -- 172.21.15.38:0/133833188 learned_addr learned my addr 172.21.15.38:0/133833188 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc190111b50 con 0x7fc1901029c0 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc1901119d0 con 0x7fc1901032e0 2024-04-22T00:03:19.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc190111cd0 con 0x7fc19010bd00 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.199+0000 7fc18effd640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1901032e0 0x7fc190198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18ffff640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 0x7fc19019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 -- 172.21.15.38:0/133833188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 msgr2=0x7fc19019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 0x7fc19019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 -- 172.21.15.38:0/133833188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1901032e0 msgr2=0x7fc190198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1901032e0 0x7fc190198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19019d040 con 0x7fc1901029c0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18ffff640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 0x7fc19019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc180009a20 tx=0x7fc18002dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18cff9640 1 -- 172.21.15.38:0/133833188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180043020 con 0x7fc1901029c0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19019d2d0 con 0x7fc1901029c0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc18cff9640 1 -- 172.21.15.38:0/133833188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1800054e0 con 0x7fc1901029c0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.200+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1901a4bc0 con 0x7fc1901029c0 2024-04-22T00:03:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.201+0000 7fc18cff9640 1 -- 172.21.15.38:0/133833188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180003bd0 con 0x7fc1901029c0 2024-04-22T00:03:19.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.204+0000 7fc18cff9640 1 -- 172.21.15.38:0/133833188 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fc180046020 con 0x7fc1901029c0 2024-04-22T00:03:19.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.204+0000 7fc19881e740 1 Processor -- start 2024-04-22T00:03:19.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 -- start start 2024-04-22T00:03:19.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18cff9640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 0x7fc1780a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d9520 0x55857d9db910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55857d934530 con 0x55857d9d0940 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55857d8dc830 con 0x55857d9d9520 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc19881e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55857d9304f0 con 0x55857d9d0020 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46792/0 (socket says 172.21.15.38:46792) 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 -- 172.21.15.38:0/3733743453 learned_addr learned my addr 172.21.15.38:0/3733743453 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18cff9640 1 -- 172.21.15.38:0/133833188 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc1800f1a00 con 0x7fc1901029c0 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18ffff640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 0x7fc1780a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 -- 172.21.15.38:0/3733743453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 msgr2=0x55857d9d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 -- 172.21.15.38:0/3733743453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d9520 msgr2=0x55857d9db910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:19.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d9520 0x55857d9db910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.205+0000 7fc18effd640 1 -- 172.21.15.38:0/3733743453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55857d9002c0 con 0x55857d9d0940 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.206+0000 7fc18ffff640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 0x7fc1780a5370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc188009370 tx=0x7fc188009300 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.206+0000 7fc18effd640 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ca0 tx=0x7fc184011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4c1e120b6498f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.206+0000 7fc175ffb640 1 -- 172.21.15.38:0/3733743453 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184005220 con 0x55857d9d0940 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.206+0000 7fc175ffb640 1 -- 172.21.15.38:0/3733743453 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc184009e50 con 0x55857d9d0940 2024-04-22T00:03:19.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.206+0000 7fc175ffb640 1 -- 172.21.15.38:0/3733743453 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184016810 con 0x55857d9d0940 2024-04-22T00:03:19.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 -- 172.21.15.38:0/3733743453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 msgr2=0x55857d9d8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ca0 tx=0x7fc184011810 comp rx=0 tx=0).stop 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 -- 172.21.15.38:0/3733743453 shutdown_connections 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d9520 0x55857d9db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d0940 0x55857d9d8fe0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 --2- 172.21.15.38:0/3733743453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 -- 172.21.15.38:0/3733743453 >> 172.21.15.38:0/3733743453 conn(0x55857d9ca750 msgr2=0x55857d9cab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 -- 172.21.15.38:0/3733743453 shutdown_connections 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.207+0000 7fc19881e740 1 -- 172.21.15.38:0/3733743453 wait complete. 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 -- start start 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 Processor -- start 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9d8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9e5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55857d9dc310 con 0x55857d9d9520 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55857d8eff10 con 0x55857d9d0020 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55857d9dc490 con 0x55857d9d68f0 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc19881e740 10 client.0 _ll_register_callbacks cb 0x55857d647be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc18effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc18effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46806/0 (socket says 172.21.15.38:46806) 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.208+0000 7fc18effd640 1 -- 172.21.15.38:0/2226986413 learned_addr learned my addr 172.21.15.38:0/2226986413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9d8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18ffff640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 -- 172.21.15.38:0/2226986413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 msgr2=0x55857d9e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9e5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 -- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 msgr2=0x55857d9d8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9d8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 -- 172.21.15.38:0/2226986413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55857d8c1c80 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc18effd640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc184020040 tx=0x7fc1840048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e026b4729bf0e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc16effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184004cf0 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc184004020 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.209+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc184009e50 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc184004240 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 10 client.16206 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55857d6483f0 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc184009e50 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc184016550 con 0x55857d9d9520 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 10 client.16206 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 10 client.16206 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:03:19.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 20 client.16206 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857d9e9a20 0x55857d9ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 4 client.16206 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.210+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55857d941a70 con 0x55857d9e9a20 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.211+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857d9e9a20 0x55857d9ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.211+0000 7fc16effd640 10 client.16206 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.211+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857d9e9a20 0x55857d9ebe10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d822497105817bda server_cookie=9a13795200b8714a in_seq=0 out_seq=0 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc16effd640 1 -- 172.21.15.38:0/2226986413 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55857d941a70 con 0x55857d9e9a20 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc16effd640 10 client.16206 handle_command_reply: tid=0 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 client.16206 shutdown 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 msgr2=0x55857d9d6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc184020040 tx=0x7fc1840048f0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 shutdown_connections 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857d9e9a20 0x55857d9ebe10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc16effd640 0 client.16206 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9d6270 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 --2- 172.21.15.38:0/2226986413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9d8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 >> 172.21.15.38:0/2226986413 conn(0x55857d9ca750 msgr2=0x55857d9cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 shutdown_connections 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 1 -- 172.21.15.38:0/2226986413 wait complete. 2024-04-22T00:03:19.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.212+0000 7fc19881e740 20 client.16206 trim_cache size 0 max 16384 2024-04-22T00:03:19.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 Processor -- start 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 -- start start 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c00fd70 con 0x55857d9d9520 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c0118b0 con 0x55857d9d0020 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.226+0000 7fc19881e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c010440 con 0x55857d9d68f0 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46820/0 (socket says 172.21.15.38:46820) 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 -- 172.21.15.38:0/3929012640 learned_addr learned my addr 172.21.15.38:0/3929012640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18effd640 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 -- 172.21.15.38:0/3929012640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 msgr2=0x55857d9f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 -- 172.21.15.38:0/3929012640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 msgr2=0x55857d9f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9f14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 -- 172.21.15.38:0/3929012640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55857d7127d0 con 0x55857d9d9520 2024-04-22T00:03:19.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc18ffff640 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc188011580 tx=0x7fc188012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be17137441a30d5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.227+0000 7fc1757fa640 1 -- 172.21.15.38:0/3929012640 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18801e620 con 0x55857d9d9520 2024-04-22T00:03:19.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc1757fa640 1 -- 172.21.15.38:0/3929012640 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc18801eca0 con 0x55857d9d9520 2024-04-22T00:03:19.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc1757fa640 1 -- 172.21.15.38:0/3929012640 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1880103d0 con 0x55857d9d9520 2024-04-22T00:03:19.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 -- 172.21.15.38:0/3929012640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 msgr2=0x55857d9c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc188011580 tx=0x7fc188012000 comp rx=0 tx=0).stop 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 -- 172.21.15.38:0/3929012640 shutdown_connections 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857d9c96d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d68f0 0x55857d9f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 --2- 172.21.15.38:0/3929012640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d0020 0x55857d9f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 -- 172.21.15.38:0/3929012640 >> 172.21.15.38:0/3929012640 conn(0x55857d9ca750 msgr2=0x55857d9c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 -- 172.21.15.38:0/3929012640 shutdown_connections 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.228+0000 7fc19881e740 1 -- 172.21.15.38:0/3929012640 wait complete. 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.229+0000 7fc19881e740 1 -- start start 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.229+0000 7fc19881e740 1 Processor -- start 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.229+0000 7fc19881e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9dd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.229+0000 7fc19881e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 0x55857d9dda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc19881e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc19881e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c0111e0 con 0x55857d9d9520 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc19881e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c010b10 con 0x55857d9d68f0 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc19881e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c00dd50 con 0x55857d9d0020 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46826/0 (socket says 172.21.15.38:46826) 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 -- 172.21.15.38:0/905117685 learned_addr learned my addr 172.21.15.38:0/905117685 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 0x55857d9dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9dd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc19881e740 10 client.0 _ll_register_callbacks cb 0x7fc184009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 -- 172.21.15.38:0/905117685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 msgr2=0x55857d9dd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9dd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 -- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 msgr2=0x55857d9dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 0x55857d9dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 -- 172.21.15.38:0/905117685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55857d9de340 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18effd640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 0x55857d9dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc18ffff640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc188006010 tx=0x7fc1880095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50e0d5c86e9ffbd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.230+0000 7fc16effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188026610 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc188026b40 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc18801eca0 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc188026d40 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 10 client.16212 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55857da50370 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc18801eca0 con 0x55857d9d9520 2024-04-22T00:03:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc188005040 con 0x55857d9d9520 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 10 client.16212 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 10 client.16212 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 20 client.16212 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857da509d0 0x55857da52df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 4 client.16212 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.231+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55857d93bdc0 con 0x55857da509d0 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857da509d0 0x55857da52df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc16effd640 10 client.16212 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc18f7fe640 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857da509d0 0x55857da52df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d6ef344c8910dfa server_cookie=70917e5dfdba5ce2 in_seq=0 out_seq=0 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc16effd640 1 -- 172.21.15.38:0/905117685 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55857d93bdc0 con 0x55857da509d0 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc16effd640 10 client.16212 handle_command_reply: tid=0 2024-04-22T00:03:19.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.232+0000 7fc19881e740 1 client.16212 shutdown 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 msgr2=0x55857da4ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc188006010 tx=0x7fc1880095e0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 shutdown_connections 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55857da509d0 0x55857da52df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc16effd640 0 client.16212 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55857d9d9520 0x55857da4ed40 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55857d9d68f0 0x55857d9dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 --2- 172.21.15.38:0/905117685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55857d9d0020 0x55857d9dd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 >> 172.21.15.38:0/905117685 conn(0x55857d9ca750 msgr2=0x55857d9c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 shutdown_connections 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 1 -- 172.21.15.38:0/905117685 wait complete. 2024-04-22T00:03:19.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.233+0000 7fc19881e740 20 client.16212 trim_cache size 0 max 16384 2024-04-22T00:03:19.232 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 msgr2=0x7fc1780a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 0x7fc1780a5370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc188009370 tx=0x7fc188009300 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 msgr2=0x7fc190198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc180009a20 tx=0x7fc18002dd30 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 shutdown_connections 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc1780a2eb0 0x7fc1780a5370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc19010bd00 0x7fc19019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1901032e0 0x7fc190198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 --2- 172.21.15.38:0/133833188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc1901029c0 0x7fc190198050 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 >> 172.21.15.38:0/133833188 conn(0x7fc190077740 msgr2=0x7fc1900fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 shutdown_connections 2024-04-22T00:03:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.234+0000 7fc195c35640 1 -- 172.21.15.38:0/133833188 wait complete. 2024-04-22T00:03:19.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 -- 172.21.15.38:0/457164981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3181066b0 msgr2=0x7ff318106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/457164981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3181066b0 0x7ff318106a90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff308009930 tx=0x7ff30802bc30 comp rx=0 tx=0).stop 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 -- 172.21.15.38:0/457164981 shutdown_connections 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/457164981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100fd0 0x7ff31810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/457164981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff318100610 0x7ff318100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/457164981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3181066b0 0x7ff318106a90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 -- 172.21.15.38:0/457164981 >> 172.21.15.38:0/457164981 conn(0x7ff318077740 msgr2=0x7ff318077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 -- 172.21.15.38:0/457164981 shutdown_connections 2024-04-22T00:03:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.356+0000 7ff31d1e5640 1 -- 172.21.15.38:0/457164981 wait complete. 2024-04-22T00:03:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.357+0000 7ff31d1e5640 1 Processor -- start 2024-04-22T00:03:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.357+0000 7ff31d1e5640 1 -- start start 2024-04-22T00:03:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.357+0000 7ff31d1e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 0x7ff318198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.357+0000 7ff31d1e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.357+0000 7ff31d1e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 0x7ff3181a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31d1e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff318110a30 con 0x7ff318100fd0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31d1e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181108b0 con 0x7ff3181066b0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31d1e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff318110bb0 con 0x7ff318100610 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46844/0 (socket says 172.21.15.38:46844) 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 -- 172.21.15.38:0/4266544571 learned_addr learned my addr 172.21.15.38:0/4266544571 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31759e640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 0x7ff3181a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 0x7ff318198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 -- 172.21.15.38:0/4266544571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 msgr2=0x7ff318198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 0x7ff318198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 -- 172.21.15.38:0/4266544571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 msgr2=0x7ff3181a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 0x7ff3181a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 -- 172.21.15.38:0/4266544571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3181a4390 con 0x7ff318100fd0 2024-04-22T00:03:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31759e640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 0x7ff3181a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 0x7ff318198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.358+0000 7ff31659c640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00d950 tx=0x7ff30c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.359+0000 7ff2fffff640 1 -- 172.21.15.38:0/4266544571 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c017030 con 0x7ff318100fd0 2024-04-22T00:03:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.359+0000 7ff2fffff640 1 -- 172.21.15.38:0/4266544571 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff30c004020 con 0x7ff318100fd0 2024-04-22T00:03:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.359+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3181a4560 con 0x7ff318100fd0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.359+0000 7ff2fffff640 1 -- 172.21.15.38:0/4266544571 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0047a0 con 0x7ff318100fd0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.359+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3181a4a70 con 0x7ff318100fd0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.361+0000 7ff2fffff640 1 -- 172.21.15.38:0/4266544571 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7ff30c0041d0 con 0x7ff318100fd0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.362+0000 7ff2fffff640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 0x7ff3000a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.362+0000 7ff31fdd5740 1 Processor -- start 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.362+0000 7ff31fdd5740 1 -- start start 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff2fffff640 1 -- 172.21.15.38:0/4266544571 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff30c0cf3e0 con 0x7ff318100fd0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 0x7ff3000a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b052025fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b052028910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b051f81530 con 0x55b05201d020 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b051f29830 con 0x55b052026520 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b051f7d4f0 con 0x55b05201d940 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46848/0 (socket says 172.21.15.38:46848) 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 -- 172.21.15.38:0/2071003367 learned_addr learned my addr 172.21.15.38:0/2071003367 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31659c640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b052025fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff31759e640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b052028910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 -- 172.21.15.38:0/2071003367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 msgr2=0x55b052025fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b052025fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 -- 172.21.15.38:0/2071003367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 msgr2=0x55b052028910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b052028910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.363+0000 7ff316d9d640 1 -- 172.21.15.38:0/2071003367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b051f4d2c0 con 0x55b05201d020 2024-04-22T00:03:19.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff31759e640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b052028910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 0x7ff3000a53f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff318101750 tx=0x7ff30803a000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff316d9d640 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff3080096c0 tx=0x7ff3080040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1967fdb68ee415f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff2fd7fa640 1 -- 172.21.15.38:0/2071003367 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308050070 con 0x55b05201d020 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff2fd7fa640 1 -- 172.21.15.38:0/2071003367 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff308047070 con 0x55b05201d020 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.364+0000 7ff2fd7fa640 1 -- 172.21.15.38:0/2071003367 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3080035f0 con 0x55b05201d020 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- 172.21.15.38:0/2071003367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 msgr2=0x55b05201d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff3080096c0 tx=0x7ff3080040c0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- 172.21.15.38:0/2071003367 shutdown_connections 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b052028910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b052025fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/2071003367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b05201d400 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- 172.21.15.38:0/2071003367 >> 172.21.15.38:0/2071003367 conn(0x55b052017750 msgr2=0x55b052017b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- 172.21.15.38:0/2071003367 shutdown_connections 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- 172.21.15.38:0/2071003367 wait complete. 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 -- start start 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.365+0000 7ff31fdd5740 1 Processor -- start 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b0520232a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b0520238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b0520292b0 con 0x55b05201d020 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b051f3cf10 con 0x55b052026520 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b052029430 con 0x55b05201d940 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31fdd5740 10 client.0 _ll_register_callbacks cb 0x55b051c94be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46856/0 (socket says 172.21.15.38:46856) 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 -- 172.21.15.38:0/1108922464 learned_addr learned my addr 172.21.15.38:0/1108922464 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31659c640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b0520232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff31759e640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b0520238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 -- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 msgr2=0x55b0520232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b0520232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 -- 172.21.15.38:0/1108922464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 msgr2=0x55b0520238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b0520238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 -- 172.21.15.38:0/1108922464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b051f0ec80 con 0x55b05201d020 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff316d9d640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff308009850 tx=0x7ff308009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0b7f0bc7584a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.366+0000 7ff2faff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308050040 con 0x55b05201d020 2024-04-22T00:03:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3080046c0 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff308047070 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff308062050 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 10 client.16224 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b051c953f0 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff308047070 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff308053070 con 0x55b05201d020 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 10 client.16224 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 10 client.16224 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 20 client.16224 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.367+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b052036390 0x55b052038780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.368+0000 7ff31fdd5740 4 client.16224 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.368+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b051f8ea70 con 0x55b052036390 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.368+0000 7ff31659c640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b052036390 0x55b052038780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.368+0000 7ff31659c640 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b052036390 0x55b052038780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f5f0eb5b6017a95c server_cookie=b8315e04816444ed in_seq=0 out_seq=0 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.368+0000 7ff2faff5640 10 client.16224 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff2faff5640 1 -- 172.21.15.38:0/1108922464 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b051f8ea70 con 0x55b052036390 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff2faff5640 10 client.16224 handle_command_reply: tid=0 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 client.16224 shutdown 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 msgr2=0x55b052025f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff308009850 tx=0x7ff308009260 comp rx=0 tx=0).stop 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 shutdown_connections 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b052036390 0x55b052038780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052026520 0x55b0520238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff2faff5640 0 client.16224 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:19.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b05201d940 0x55b0520232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/1108922464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d020 0x55b052025f10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 >> 172.21.15.38:0/1108922464 conn(0x55b052017750 msgr2=0x55b0520194a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 shutdown_connections 2024-04-22T00:03:19.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 1 -- 172.21.15.38:0/1108922464 wait complete. 2024-04-22T00:03:19.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.369+0000 7ff31fdd5740 20 client.16224 trim_cache size 0 max 16384 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 Processor -- start 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 -- start start 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520259d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b05203dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181aea40 con 0x55b05201d940 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181b0580 con 0x55b052025f10 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.383+0000 7ff31fdd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181af110 con 0x55b052026520 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46872/0 (socket says 172.21.15.38:46872) 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 -- 172.21.15.38:0/405486778 learned_addr learned my addr 172.21.15.38:0/405486778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff31659c640 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520259d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff31759e640 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b05203dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 -- 172.21.15.38:0/405486778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 msgr2=0x55b0520259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520259d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 -- 172.21.15.38:0/405486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 msgr2=0x55b05203dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b05203dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 -- 172.21.15.38:0/405486778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b05201d770 con 0x55b05201d940 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff316d9d640 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff308006010 tx=0x7ff30802c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159d913de0dfd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff2fcff9640 1 -- 172.21.15.38:0/405486778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308050050 con 0x55b05201d940 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff2fcff9640 1 -- 172.21.15.38:0/405486778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff30806b070 con 0x55b05201d940 2024-04-22T00:03:19.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.384+0000 7ff2fcff9640 1 -- 172.21.15.38:0/405486778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3080035f0 con 0x55b05201d940 2024-04-22T00:03:19.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 -- 172.21.15.38:0/405486778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 msgr2=0x55b052032170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff308006010 tx=0x7ff30802c6e0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 -- 172.21.15.38:0/405486778 shutdown_connections 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b05203dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520259d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/405486778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b052032170 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 -- 172.21.15.38:0/405486778 >> 172.21.15.38:0/405486778 conn(0x55b052017750 msgr2=0x55b052019410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 -- 172.21.15.38:0/405486778 shutdown_connections 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.385+0000 7ff31fdd5740 1 -- 172.21.15.38:0/405486778 wait complete. 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 -- start start 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 Processor -- start 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b052020390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520208d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff316d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46884/0 (socket says 172.21.15.38:46884) 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff316d9d640 1 -- 172.21.15.38:0/4253011245 learned_addr learned my addr 172.21.15.38:0/4253011245 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181afeb0 con 0x55b05201d940 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181af7e0 con 0x55b052025f10 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3181aca20 con 0x55b052026520 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.386+0000 7ff31fdd5740 10 client.0 _ll_register_callbacks cb 0x7ff30804e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31759e640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520208d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 -- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 msgr2=0x55b0520208d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520208d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31659c640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b052020390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 -- 172.21.15.38:0/4253011245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 msgr2=0x55b052020390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b052020390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b051d5f7d0 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31759e640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520208d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31659c640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b052020390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff316d9d640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff308003c60 tx=0x7ff308009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d059a10017a59e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff2faff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308050070 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3080048c0 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff30806b070 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff308078050 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31fdd5740 10 client.16230 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.387+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b05209c5b0 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.388+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff30806b070 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.388+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff308053040 con 0x55b05201d940 2024-04-22T00:03:19.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 10 client.16230 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 10 client.16230 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 20 client.16230 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b05209cc40 0x55b05209f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 4 client.16230 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b051f88dc0 con 0x55b05209cc40 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31659c640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b05209cc40 0x55b05209f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff31659c640 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b05209cc40 0x55b05209f060 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2c85ce815794dd2 server_cookie=404741088ae16e7d in_seq=0 out_seq=0 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.389+0000 7ff2faff5640 10 client.16230 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff2faff5640 1 -- 172.21.15.38:0/4253011245 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b051f88dc0 con 0x55b05209cc40 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff2faff5640 10 client.16230 handle_command_reply: tid=0 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 client.16230 shutdown 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 msgr2=0x55b05201fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff308003c60 tx=0x7ff308009650 comp rx=0 tx=0).stop 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 shutdown_connections 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55b05209cc40 0x55b05209f060 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b052026520 0x55b0520208d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b052025f10 0x55b052020390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 --2- 172.21.15.38:0/4253011245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b05201d940 0x55b05201fe50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 >> 172.21.15.38:0/4253011245 conn(0x55b052017750 msgr2=0x55b052015fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff2faff5640 0 client.16230 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 shutdown_connections 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 1 -- 172.21.15.38:0/4253011245 wait complete. 2024-04-22T00:03:19.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.390+0000 7ff31fdd5740 20 client.16230 trim_cache size 0 max 16384 2024-04-22T00:03:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:19.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 msgr2=0x7ff3000a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 0x7ff3000a53f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff318101750 tx=0x7ff30803a000 comp rx=0 tx=0).stop 2024-04-22T00:03:19.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 msgr2=0x7ff318199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff30c00d950 tx=0x7ff30c00de10 comp rx=0 tx=0).stop 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 shutdown_connections 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3000a2f30 0x7ff3000a53f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181066b0 0x7ff3181a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff318100fd0 0x7ff318199310 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 --2- 172.21.15.38:0/4266544571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff318100610 0x7ff318198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 >> 172.21.15.38:0/4266544571 conn(0x7ff318077740 msgr2=0x7ff3180ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 shutdown_connections 2024-04-22T00:03:19.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.391+0000 7ff31d1e5640 1 -- 172.21.15.38:0/4266544571 wait complete. 2024-04-22T00:03:19.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.512+0000 7f64bb325640 1 -- 172.21.15.38:0/4037640905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 msgr2=0x7f64b4102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.512+0000 7f64bb325640 1 --2- 172.21.15.38:0/4037640905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b4102da0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009900 tx=0x7f64b002bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 -- 172.21.15.38:0/4037640905 shutdown_connections 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 --2- 172.21.15.38:0/4037640905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b410bd00 0x7f64b410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 --2- 172.21.15.38:0/4037640905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b41032e0 0x7f64b410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 --2- 172.21.15.38:0/4037640905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b4102da0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 -- 172.21.15.38:0/4037640905 >> 172.21.15.38:0/4037640905 conn(0x7f64b4077740 msgr2=0x7f64b4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 -- 172.21.15.38:0/4037640905 shutdown_connections 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 -- 172.21.15.38:0/4037640905 wait complete. 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 Processor -- start 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.513+0000 7f64bb325640 1 -- start start 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b41032e0 0x7f64b41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b410bd00 0x7f64b419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4111b50 con 0x7f64b41029c0 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41119d0 con 0x7f64b41032e0 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64bb325640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4111cd0 con 0x7f64b410bd00 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46892/0 (socket says 172.21.15.38:46892) 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 -- 172.21.15.38:0/3952841783 learned_addr learned my addr 172.21.15.38:0/3952841783 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b98c2640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b410bd00 0x7f64b419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b88c0640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b41032e0 0x7f64b41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 -- 172.21.15.38:0/3952841783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b410bd00 msgr2=0x7f64b419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b410bd00 0x7f64b419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 -- 172.21.15.38:0/3952841783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b41032e0 msgr2=0x7f64b41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b41032e0 0x7f64b41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 -- 172.21.15.38:0/3952841783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b419d0a0 con 0x7f64b41029c0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.514+0000 7f64b90c1640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009a20 tx=0x7f64b002dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.515+0000 7f64aa7fc640 1 -- 172.21.15.38:0/3952841783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0043020 con 0x7f64b41029c0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.515+0000 7f64aa7fc640 1 -- 172.21.15.38:0/3952841783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b00054e0 con 0x7f64b41029c0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.515+0000 7f64aa7fc640 1 -- 172.21.15.38:0/3952841783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0003bd0 con 0x7f64b41029c0 2024-04-22T00:03:19.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.515+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419d330 con 0x7f64b41029c0 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.515+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b41a4c20 con 0x7f64b41029c0 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.517+0000 7f64bdfed740 1 Processor -- start 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.517+0000 7f64bdfed740 1 -- start start 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64aa7fc640 1 -- 172.21.15.38:0/3952841783 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f64b0046020 con 0x7f64b41029c0 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56066d514530 con 0x56066d5c1c50 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56066d4bc830 con 0x56066d5afa90 2024-04-22T00:03:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64bdfed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56066d5104f0 con 0x56066d5b0440 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46902/0 (socket says 172.21.15.38:46902) 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 -- 172.21.15.38:0/761150696 learned_addr learned my addr 172.21.15.38:0/761150696 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b90c1640 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b98c2640 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 -- 172.21.15.38:0/761150696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 msgr2=0x56066d5b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 -- 172.21.15.38:0/761150696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 msgr2=0x56066d5afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64b88c0640 1 -- 172.21.15.38:0/761150696 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56066d4e02c0 con 0x56066d5c1c50 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.518+0000 7f64aa7fc640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 0x7f64a00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f64b88c0640 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f64a400bc90 tx=0x7f64a400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7027bafcc02d3f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f64b98c2640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 0x7f64a00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f64aa7fc640 1 -- 172.21.15.38:0/3952841783 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f64b00f1a50 con 0x7f64b41029c0 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f649f7fe640 1 -- 172.21.15.38:0/761150696 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4015070 con 0x56066d5c1c50 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f649f7fe640 1 -- 172.21.15.38:0/761150696 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a401a940 con 0x56066d5c1c50 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.519+0000 7f649f7fe640 1 -- 172.21.15.38:0/761150696 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4011910 con 0x56066d5c1c50 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64b98c2640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 0x7f64a00a53f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64ac005fd0 tx=0x7f64ac009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- 172.21.15.38:0/761150696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 msgr2=0x56066d5c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f64a400bc90 tx=0x7f64a400f730 comp rx=0 tx=0).stop 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- 172.21.15.38:0/761150696 shutdown_connections 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5c4040 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 --2- 172.21.15.38:0/761150696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- 172.21.15.38:0/761150696 >> 172.21.15.38:0/761150696 conn(0x56066d5aa750 msgr2=0x56066d5aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- 172.21.15.38:0/761150696 shutdown_connections 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- 172.21.15.38:0/761150696 wait complete. 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 -- start start 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.520+0000 7f64bdfed740 1 Processor -- start 2024-04-22T00:03:19.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56066d5c4b00 con 0x56066d5c1c50 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56066d4cff10 con 0x56066d5afa90 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56066d5c4c80 con 0x56066d5b0440 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64bdfed740 10 client.0 _ll_register_callbacks cb 0x56066d227be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46916/0 (socket says 172.21.15.38:46916) 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 -- 172.21.15.38:0/192427183 learned_addr learned my addr 172.21.15.38:0/192427183 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b90c1640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b88c0640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 -- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 msgr2=0x56066d5b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 -- 172.21.15.38:0/192427183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 msgr2=0x56066d5b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 -- 172.21.15.38:0/192427183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56066d4a1c80 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b90c1640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f64b98c2640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f64ac003f70 tx=0x7f64ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1459dbe3ddb8c21 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.521+0000 7f649cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac02d060 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f64ac018070 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a401a940 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64ac034050 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 10 client.16242 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56066d2283f0 con 0x56066d5c1c50 2024-04-22T00:03:19.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a401a940 con 0x56066d5c1c50 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f64ac01eb90 con 0x56066d5c1c50 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 10 client.16242 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 10 client.16242 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 20 client.16242 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.522+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d5d1d70 0x56066d5d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.523+0000 7f64bdfed740 4 client.16242 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.523+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56066d521a70 con 0x56066d5d1d70 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.523+0000 7f64b90c1640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d5d1d70 0x56066d5d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.523+0000 7f64b90c1640 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d5d1d70 0x56066d5d41a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1e9b739c6550c93 server_cookie=24c242b2084b970 in_seq=0 out_seq=0 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.523+0000 7f649cff9640 10 client.16242 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f649cff9640 1 -- 172.21.15.38:0/192427183 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56066d521a70 con 0x56066d5d1d70 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f649cff9640 10 client.16242 handle_command_reply: tid=0 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 client.16242 shutdown 2024-04-22T00:03:19.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 msgr2=0x56066d5b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f64ac003f70 tx=0x7f64ac004ab0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 shutdown_connections 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d5d1d70 0x56066d5d41a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b21e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5b0440 0x56066d5b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 --2- 172.21.15.38:0/192427183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5afa90 0x56066d5b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 >> 172.21.15.38:0/192427183 conn(0x56066d5aa750 msgr2=0x56066d5aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f649cff9640 0 client.16242 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 shutdown_connections 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 1 -- 172.21.15.38:0/192427183 wait complete. 2024-04-22T00:03:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.524+0000 7f64bdfed740 20 client.16242 trim_cache size 0 max 16384 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 Processor -- start 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 -- start start 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 0x56066d5b0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ac8c0 con 0x56066d5c1c50 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ae400 con 0x56066d5b0440 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.538+0000 7f64bdfed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41acf90 con 0x56066d5afa90 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46930/0 (socket says 172.21.15.38:46930) 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 -- 172.21.15.38:0/1340848977 learned_addr learned my addr 172.21.15.38:0/1340848977 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b88c0640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 0x56066d5b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b90c1640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 -- 172.21.15.38:0/1340848977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 msgr2=0x56066d5cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 -- 172.21.15.38:0/1340848977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 msgr2=0x56066d5b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 0x56066d5b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 -- 172.21.15.38:0/1340848977 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56066d5b0ff0 con 0x56066d5c1c50 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b88c0640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 0x56066d5b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f64b98c2640 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00a010 tx=0x7f64ac009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4068c96c418bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.539+0000 7f649effd640 1 -- 172.21.15.38:0/1340848977 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac028700 con 0x56066d5c1c50 2024-04-22T00:03:19.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f649effd640 1 -- 172.21.15.38:0/1340848977 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ac004890 con 0x56066d5c1c50 2024-04-22T00:03:19.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f649effd640 1 -- 172.21.15.38:0/1340848977 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac020ba0 con 0x56066d5c1c50 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 -- 172.21.15.38:0/1340848977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 msgr2=0x56066d5b43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00a010 tx=0x7f64ac009e60 comp rx=0 tx=0).stop 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 -- 172.21.15.38:0/1340848977 shutdown_connections 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5c1c50 0x56066d5b43d0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5b0440 0x56066d5b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1340848977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 -- 172.21.15.38:0/1340848977 >> 172.21.15.38:0/1340848977 conn(0x56066d5aa750 msgr2=0x56066d5bdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 -- 172.21.15.38:0/1340848977 shutdown_connections 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.540+0000 7f64bdfed740 1 -- 172.21.15.38:0/1340848977 wait complete. 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 -- start start 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 Processor -- start 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5c1c50 0x56066d5c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41add30 con 0x56066d5b0440 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ad660 con 0x56066d5c1c50 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41aa8a0 con 0x56066d5afa90 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64b88c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.541+0000 7f64bdfed740 10 client.0 _ll_register_callbacks cb 0x7f64ac009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46936/0 (socket says 172.21.15.38:46936) 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 -- 172.21.15.38:0/1604887048 learned_addr learned my addr 172.21.15.38:0/1604887048 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b90c1640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b98c2640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5c1c50 0x56066d5c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 -- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 msgr2=0x56066d5c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 -- 172.21.15.38:0/1604887048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5c1c50 msgr2=0x56066d5c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5c1c50 0x56066d5c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 -- 172.21.15.38:0/1604887048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56066d2f27d0 con 0x56066d5b0440 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64b88c0640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f64a400bc90 tx=0x7f64a400b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5316bdd027b5a9f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f649cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4015040 con 0x56066d5b0440 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f64a4005450 con 0x56066d5b0440 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.542+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ac004890 con 0x56066d5b0440 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64a4033050 con 0x56066d5b0440 2024-04-22T00:03:19.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 10 client.16248 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56066d6385b0 con 0x56066d5b0440 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ac004890 con 0x56066d5b0440 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f64a4025630 con 0x56066d5b0440 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 10 client.16248 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 10 client.16248 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 20 client.16248 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d638a40 0x56066d63ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 4 client.16248 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.543+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56066d51bdc0 con 0x56066d638a40 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.544+0000 7f64b90c1640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d638a40 0x56066d63ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.544+0000 7f64b90c1640 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d638a40 0x56066d63ae30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2132d05a9988e1f server_cookie=87acc9f1a34b4b05 in_seq=0 out_seq=0 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.544+0000 7f649cff9640 10 client.16248 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:19.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f649cff9640 1 -- 172.21.15.38:0/1604887048 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56066d51bdc0 con 0x56066d638a40 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f649cff9640 10 client.16248 handle_command_reply: tid=0 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 client.16248 shutdown 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 msgr2=0x56066d5c3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f64a400bc90 tx=0x7f64a400b7a0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 shutdown_connections 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x56066d638a40 0x56066d63ae30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56066d5c1c50 0x56066d5c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f649cff9640 0 client.16248 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56066d5b0440 0x56066d5c3020 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 --2- 172.21.15.38:0/1604887048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56066d5afa90 0x56066d5c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 >> 172.21.15.38:0/1604887048 conn(0x56066d5aa750 msgr2=0x56066d5abba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 shutdown_connections 2024-04-22T00:03:19.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 1 -- 172.21.15.38:0/1604887048 wait complete. 2024-04-22T00:03:19.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.545+0000 7f64bdfed740 20 client.16248 trim_cache size 0 max 16384 2024-04-22T00:03:19.544 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:19.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 msgr2=0x7f64a00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 0x7f64a00a53f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64ac005fd0 tx=0x7f64ac009500 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 msgr2=0x7f64b41980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009a20 tx=0x7f64b002dd30 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 shutdown_connections 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f64a00a2f30 0x7f64a00a53f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b410bd00 0x7f64b419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64b41032e0 0x7f64b41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 --2- 172.21.15.38:0/3952841783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64b41029c0 0x7f64b41980b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 >> 172.21.15.38:0/3952841783 conn(0x7f64b4077740 msgr2=0x7f64b40fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 shutdown_connections 2024-04-22T00:03:19.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.546+0000 7f64bb325640 1 -- 172.21.15.38:0/3952841783 wait complete. 2024-04-22T00:03:19.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:19 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3181199114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:19.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 -- 172.21.15.38:0/2300643914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100e60 msgr2=0x7f032010dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 --2- 172.21.15.38:0/2300643914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100e60 0x7f032010dbb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0310009860 tx=0x7f031002bb70 comp rx=0 tx=0).stop 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 -- 172.21.15.38:0/2300643914 shutdown_connections 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 --2- 172.21.15.38:0/2300643914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f03201104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 --2- 172.21.15.38:0/2300643914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100e60 0x7f032010dbb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 --2- 172.21.15.38:0/2300643914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100540 0x7f0320100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 -- 172.21.15.38:0/2300643914 >> 172.21.15.38:0/2300643914 conn(0x7f0320077740 msgr2=0x7f0320077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 -- 172.21.15.38:0/2300643914 shutdown_connections 2024-04-22T00:03:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.665+0000 7f0327639640 1 -- 172.21.15.38:0/2300643914 wait complete. 2024-04-22T00:03:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 Processor -- start 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 -- start start 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100540 0x7f032019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0324bd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0324bd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46946/0 (socket says 172.21.15.38:46946) 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f032019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0324bd4640 1 -- 172.21.15.38:0/1250595210 learned_addr learned my addr 172.21.15.38:0/1250595210 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113cf0 con 0x7f0320100e60 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113b70 con 0x7f032010e0f0 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0320113e70 con 0x7f0320100540 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.666+0000 7f03253d5640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100540 0x7f032019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 -- 172.21.15.38:0/1250595210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100540 msgr2=0x7f032019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100540 0x7f032019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 -- 172.21.15.38:0/1250595210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 msgr2=0x7f032019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0325bd6640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f032019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f032019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032019f1e0 con 0x7f0320100e60 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0325bd6640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f032019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0310005e00 tx=0x7f03100323d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f03167fc640 1 -- 172.21.15.38:0/1250595210 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310041020 con 0x7f0320100e60 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032019f470 con 0x7f0320100e60 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03201a6d60 con 0x7f0320100e60 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f03167fc640 1 -- 172.21.15.38:0/1250595210 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031002c9f0 con 0x7f0320100e60 2024-04-22T00:03:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.667+0000 7f03167fc640 1 -- 172.21.15.38:0/1250595210 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310039040 con 0x7f0320100e60 2024-04-22T00:03:19.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 Processor -- start 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 -- start start 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f03167fc640 1 -- 172.21.15.38:0/1250595210 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f0310044020 con 0x7f0320100e60 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc31bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 0x5625cc31e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5625cc277530 con 0x5625cc313020 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5625cc21f830 con 0x5625cc31c520 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f032a225740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625cc2734f0 con 0x5625cc313940 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f0325bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f0325bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46960/0 (socket says 172.21.15.38:46960) 2024-04-22T00:03:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f0325bd6640 1 -- 172.21.15.38:0/4127093269 learned_addr learned my addr 172.21.15.38:0/4127093269 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f03253d5640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc31bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.671+0000 7f0324bd4640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 0x5625cc31e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 -- 172.21.15.38:0/4127093269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 msgr2=0x5625cc31bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc31bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 -- 172.21.15.38:0/4127093269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 msgr2=0x5625cc31e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 0x5625cc31e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 -- 172.21.15.38:0/4127093269 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625cc2432c0 con 0x5625cc313020 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0324bd4640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 0x5625cc31e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f03167fc640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 0x7f030c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f0325bd6640 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0318009e10 tx=0x7f031800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297cfabb726f1a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f03167fc640 1 -- 172.21.15.38:0/1250595210 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f03100f1240 con 0x7f0320100e60 2024-04-22T00:03:19.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f03253d5640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 0x7f030c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f030b7fe640 1 -- 172.21.15.38:0/4127093269 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318004d20 con 0x5625cc313020 2024-04-22T00:03:19.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f030b7fe640 1 -- 172.21.15.38:0/4127093269 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0318005500 con 0x5625cc313020 2024-04-22T00:03:19.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.672+0000 7f030b7fe640 1 -- 172.21.15.38:0/4127093269 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318011860 con 0x5625cc313020 2024-04-22T00:03:19.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 -- 172.21.15.38:0/4127093269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 msgr2=0x5625cc313400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0318009e10 tx=0x7f031800f730 comp rx=0 tx=0).stop 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f03253d5640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 0x7f030c0a5370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5625cc312af0 tx=0x7f031c009260 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 -- 172.21.15.38:0/4127093269 shutdown_connections 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc31c520 0x5625cc31e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc31bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 --2- 172.21.15.38:0/4127093269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc313400 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 -- 172.21.15.38:0/4127093269 >> 172.21.15.38:0/4127093269 conn(0x5625cc30d750 msgr2=0x5625cc30db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 -- 172.21.15.38:0/4127093269 shutdown_connections 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.673+0000 7f032a225740 1 -- 172.21.15.38:0/4127093269 wait complete. 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 -- start start 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 Processor -- start 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 0x5625cc319220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc319780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5625cc31f310 con 0x5625cc313020 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5625cc232f10 con 0x5625cc313940 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625cc31f490 con 0x5625cc31c520 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f032a225740 10 client.0 _ll_register_callbacks cb 0x5625cbf8abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f03253d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f03253d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46968/0 (socket says 172.21.15.38:46968) 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.674+0000 7f03253d5640 1 -- 172.21.15.38:0/1303627723 learned_addr learned my addr 172.21.15.38:0/1303627723 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0325bd6640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc319780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 0x5625cc319220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 -- 172.21.15.38:0/1303627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 msgr2=0x5625cc319780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc319780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 -- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 msgr2=0x5625cc319220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 0x5625cc319220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 -- 172.21.15.38:0/1303627723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625cc204c80 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 0x5625cc319220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f03253d5640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f031c013010 tx=0x7f031c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602d1064dc6bd572 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0308ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c02e060 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f031c018030 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.675+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0318005500 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f031c004020 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 10 client.16260 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5625cbf8b3f0 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0318005500 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f031c029420 con 0x5625cc313020 2024-04-22T00:03:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 10 client.16260 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 10 client.16260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 20 client.16260 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc32c600 0x5625cc32ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 4 client.16260 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625cc284a70 con 0x5625cc32c600 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.676+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc32c600 0x5625cc32ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.677+0000 7f0308ff9640 10 client.16260 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.677+0000 7f0324bd4640 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc32c600 0x5625cc32ea70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=822bead519df31a3 server_cookie=a202f5e8d8b3c214 in_seq=0 out_seq=0 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f0308ff9640 1 -- 172.21.15.38:0/1303627723 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625cc284a70 con 0x5625cc32c600 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f0308ff9640 10 client.16260 handle_command_reply: tid=0 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 client.16260 shutdown 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 msgr2=0x5625cc31be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f031c013010 tx=0x7f031c004730 comp rx=0 tx=0).stop 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 shutdown_connections 2024-04-22T00:03:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc319780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313940 0x5625cc319220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc32c600 0x5625cc32ea70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 --2- 172.21.15.38:0/1303627723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313020 0x5625cc31be20 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 >> 172.21.15.38:0/1303627723 conn(0x5625cc30d750 msgr2=0x5625cc30f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f0308ff9640 0 client.16260 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 shutdown_connections 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 1 -- 172.21.15.38:0/1303627723 wait complete. 2024-04-22T00:03:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.678+0000 7f032a225740 20 client.16260 trim_cache size 0 max 16384 2024-04-22T00:03:19.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.704+0000 7f032a225740 1 Processor -- start 2024-04-22T00:03:19.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.704+0000 7f032a225740 1 -- start start 2024-04-22T00:03:19.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc328200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc31b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b0cf0 con 0x5625cc313940 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b2830 con 0x5625cc313020 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f032a225740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b13c0 con 0x5625cc31c520 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46982/0 (socket says 172.21.15.38:46982) 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 -- 172.21.15.38:0/2322112767 learned_addr learned my addr 172.21.15.38:0/2322112767 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f03253d5640 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc328200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0325bd6640 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc31b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 -- 172.21.15.38:0/2322112767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 msgr2=0x5625cc31b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc31b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 -- 172.21.15.38:0/2322112767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 msgr2=0x5625cc328200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc328200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 -- 172.21.15.38:0/2322112767 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625cc0557d0 con 0x5625cc313940 2024-04-22T00:03:19.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.705+0000 7f0324bd4640 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f03100b9cc0 tx=0x7f03100f36d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99dd8dcfd99ed77f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f030affd640 1 -- 172.21.15.38:0/2322112767 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100b7e60 con 0x5625cc313940 2024-04-22T00:03:19.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f030affd640 1 -- 172.21.15.38:0/2322112767 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03100bd9f0 con 0x5625cc313940 2024-04-22T00:03:19.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f030affd640 1 -- 172.21.15.38:0/2322112767 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100bb5f0 con 0x5625cc313940 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 -- 172.21.15.38:0/2322112767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 msgr2=0x5625cc318760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f03100b9cc0 tx=0x7f03100f36d0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 -- 172.21.15.38:0/2322112767 shutdown_connections 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc31c520 0x5625cc31b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc313940 0x5625cc318760 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 --2- 172.21.15.38:0/2322112767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc328200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.706+0000 7f032a225740 1 -- 172.21.15.38:0/2322112767 >> 172.21.15.38:0/2322112767 conn(0x5625cc30d750 msgr2=0x5625cc30c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- 172.21.15.38:0/2322112767 shutdown_connections 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- 172.21.15.38:0/2322112767 wait complete. 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- start start 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 Processor -- start 2024-04-22T00:03:19.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc3201d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc320710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b2160 con 0x5625cc31c520 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b1a90 con 0x5625cc313020 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.707+0000 7f032a225740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03201aecd0 con 0x5625cc313940 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f032a225740 10 client.0 _ll_register_callbacks cb 0x7f031c009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0325bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0325bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46996/0 (socket says 172.21.15.38:46996) 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0325bd6640 1 -- 172.21.15.38:0/939293348 learned_addr learned my addr 172.21.15.38:0/939293348 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0324bd4640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc320710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc3201d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 -- 172.21.15.38:0/939293348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 msgr2=0x5625cc320710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc320710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 -- 172.21.15.38:0/939293348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 msgr2=0x5625cc320c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 -- 172.21.15.38:0/939293348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625cc391c00 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0325bd6640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f03253d5640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc3201d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f031c0298c0 tx=0x7f031c0043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee83409c1c08a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.708+0000 7f0308ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c03e070 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03100bd9f0 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f031c029cc0 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f031c047050 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5625cc392f00 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03100bd9f0 con 0x5625cc313020 2024-04-22T00:03:19.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f031c041020 con 0x5625cc313020 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 20 client.25763 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc393670 0x5625cc395a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 4 client.25763 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.709+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625cc27edc0 con 0x5625cc393670 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0324bd4640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc393670 0x5625cc395a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0308ff9640 10 client.25763 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0324bd4640 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc393670 0x5625cc395a90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6829c9b97e8a671 server_cookie=88378ee8ce329ff9 in_seq=0 out_seq=0 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0308ff9640 1 -- 172.21.15.38:0/939293348 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625cc27edc0 con 0x5625cc393670 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0308ff9640 10 client.25763 handle_command_reply: tid=0 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 client.25763 shutdown 2024-04-22T00:03:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 msgr2=0x5625cc3201d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc3201d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f031c0298c0 tx=0x7f031c0043f0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 shutdown_connections 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5625cc393670 0x5625cc395a90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f0308ff9640 0 client.25763 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625cc31c520 0x5625cc320c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625cc313940 0x5625cc320710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 --2- 172.21.15.38:0/939293348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5625cc313020 0x5625cc3201d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.710+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 >> 172.21.15.38:0/939293348 conn(0x5625cc30d750 msgr2=0x5625cc313e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 shutdown_connections 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f032a225740 1 -- 172.21.15.38:0/939293348 wait complete. 2024-04-22T00:03:19.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f032a225740 20 client.25763 trim_cache size 0 max 16384 2024-04-22T00:03:19.709 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:19.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 msgr2=0x7f030c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 0x7f030c0a5370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5625cc312af0 tx=0x7f031c009260 comp rx=0 tx=0).stop 2024-04-22T00:03:19.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 msgr2=0x7f032019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0310005e00 tx=0x7f03100323d0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 shutdown_connections 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f030c0a2eb0 0x7f030c0a5370 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f032010e0f0 0x7f032019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0320100e60 0x7f032019a730 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 --2- 172.21.15.38:0/1250595210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0320100540 0x7f032019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 >> 172.21.15.38:0/1250595210 conn(0x7f0320077740 msgr2=0x7f032010f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 shutdown_connections 2024-04-22T00:03:19.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:19.711+0000 7f0327639640 1 -- 172.21.15.38:0/1250595210 wait complete. 2024-04-22T00:03:19.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:19 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3181199114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:19.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:19 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3181199114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 -- 172.21.15.38:0/2361054325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201032f0 msgr2=0x7fa82010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 --2- 172.21.15.38:0/2361054325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201032f0 0x7fa82010b7d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009900 tx=0x7fa80c02bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 -- 172.21.15.38:0/2361054325 shutdown_connections 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 --2- 172.21.15.38:0/2361054325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82010bd10 0x7fa82010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 --2- 172.21.15.38:0/2361054325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201032f0 0x7fa82010b7d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 --2- 172.21.15.38:0/2361054325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8201029d0 0x7fa820102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 -- 172.21.15.38:0/2361054325 >> 172.21.15.38:0/2361054325 conn(0x7fa8200777c0 msgr2=0x7fa820077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 -- 172.21.15.38:0/2361054325 shutdown_connections 2024-04-22T00:03:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.493+0000 7fa826b02640 1 -- 172.21.15.38:0/2361054325 wait complete. 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.494+0000 7fa826b02640 1 Processor -- start 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.494+0000 7fa826b02640 1 -- start start 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.494+0000 7fa826b02640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.494+0000 7fa826b02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa82010bd10 0x7fa820198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.494+0000 7fa826b02640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82019cab0 0x7fa82019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47020/0 (socket says 172.21.15.38:47020) 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 -- 172.21.15.38:0/3639524639 learned_addr learned my addr 172.21.15.38:0/3639524639 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa820111ba0 con 0x7fa8201029d0 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa820111a20 con 0x7fa82010bd10 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa82010bd10 0x7fa820198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82509f640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82019cab0 0x7fa82019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa820111d20 con 0x7fa82019cab0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 -- 172.21.15.38:0/3639524639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82019cab0 msgr2=0x7fa82019cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82019cab0 0x7fa82019cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 -- 172.21.15.38:0/3639524639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa82010bd10 msgr2=0x7fa820198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa82010bd10 0x7fa820198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.495+0000 7fa82489e640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82019d5a0 con 0x7fa8201029d0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa82489e640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b4a0 tx=0x7fa81400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa81dffb640 1 -- 172.21.15.38:0/3639524639 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814014070 con 0x7fa8201029d0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa81dffb640 1 -- 172.21.15.38:0/3639524639 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa814010af0 con 0x7fa8201029d0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa81dffb640 1 -- 172.21.15.38:0/3639524639 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814017d10 con 0x7fa8201029d0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82019d890 con 0x7fa8201029d0 2024-04-22T00:03:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.496+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8201a5000 con 0x7fa8201029d0 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.499+0000 7fa81dffb640 1 -- 172.21.15.38:0/3639524639 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7fa814017070 con 0x7fa8201029d0 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.499+0000 7fa8297c6740 1 Processor -- start 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 -- start start 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d90f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d9a210 0x564e95d9c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564e95cf4b70 con 0x564e95d91440 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564e95c9d470 con 0x564e95d9a210 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa8297c6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e95cf0b30 con 0x564e95d90b20 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa82489e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa82489e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47024/0 (socket says 172.21.15.38:47024) 2024-04-22T00:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa82489e640 1 -- 172.21.15.38:0/2801260428 learned_addr learned my addr 172.21.15.38:0/2801260428 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa81ffff640 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d9a210 0x564e95d9c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa81dffb640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 0x7fa8100a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.500+0000 7fa82509f640 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d90f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa81dffb640 1 -- 172.21.15.38:0/3639524639 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa8140ceba0 con 0x7fa8201029d0 2024-04-22T00:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 -- 172.21.15.38:0/2801260428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 msgr2=0x564e95d90f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d90f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 -- 172.21.15.38:0/2801260428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d9a210 msgr2=0x564e95d9c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d9a210 0x564e95d9c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 -- 172.21.15.38:0/2801260428 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e95a09210 con 0x564e95d91440 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82509f640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 0x7fa8100a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa82489e640 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa81409aa00 tx=0x7fa8140047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7176c23211b57515 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa80affd640 1 -- 172.21.15.38:0/2801260428 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81401e050 con 0x564e95d91440 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa80affd640 1 -- 172.21.15.38:0/2801260428 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa81409f000 con 0x564e95d91440 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.501+0000 7fa80affd640 1 -- 172.21.15.38:0/2801260428 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814010e20 con 0x564e95d91440 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.502+0000 7fa8297c6740 1 -- 172.21.15.38:0/2801260428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 msgr2=0x564e95d99cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.502+0000 7fa82509f640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 0x7fa8100a5490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa8180098b0 tx=0x7fa818004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.502+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa81409aa00 tx=0x7fa8140047d0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 -- 172.21.15.38:0/2801260428 shutdown_connections 2024-04-22T00:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d9a210 0x564e95d9c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d99cd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2801260428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d90f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 -- 172.21.15.38:0/2801260428 >> 172.21.15.38:0/2801260428 conn(0x564e95d8b440 msgr2=0x564e95d8b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 -- 172.21.15.38:0/2801260428 shutdown_connections 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 -- 172.21.15.38:0/2801260428 wait complete. 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 -- start start 2024-04-22T00:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.503+0000 7fa8297c6740 1 Processor -- start 2024-04-22T00:03:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d91440 0x564e95d977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564e95d9cde0 con 0x564e95d9a210 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564e95cb0a10 con 0x564e95d91440 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e95d9cf60 con 0x564e95d90b20 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa8297c6740 10 client.0 _ll_register_callbacks cb 0x564e95cebb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82509f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82509f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47028/0 (socket says 172.21.15.38:47028) 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82509f640 1 -- 172.21.15.38:0/3736462714 learned_addr learned my addr 172.21.15.38:0/3736462714 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d91440 0x564e95d977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 -- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d91440 msgr2=0x564e95d977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d91440 0x564e95d977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 -- 172.21.15.38:0/3736462714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 msgr2=0x564e95d97d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82489e640 1 -- 172.21.15.38:0/3736462714 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e95ad5620 con 0x564e95d90b20 2024-04-22T00:03:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.504+0000 7fa82509f640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa82489e640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d972b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa81409ac90 tx=0x7fa814010f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa3b7bfb25a6035c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140b0030 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa81409f000 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa81409a150 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa81409a390 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa8297c6740 10 client.25654 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.505+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e95d9cde0 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8140b3e50 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa8140b7020 con 0x564e95d90b20 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 10 client.25654 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 10 client.25654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 20 client.25654 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95dae420 0x564e95db0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 4 client.25654 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e95d020b0 con 0x564e95dae420 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95dae420 0x564e95db0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa7fffff640 10 client.25654 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.506+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95dae420 0x564e95db0810 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bafeb9d866c1214 server_cookie=9cbfd898fb4d538e in_seq=0 out_seq=0 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa7fffff640 1 -- 172.21.15.38:0/3736462714 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e95d020b0 con 0x564e95dae420 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa7fffff640 10 client.25654 handle_command_reply: tid=0 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 client.25654 shutdown 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 msgr2=0x564e95d972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d972b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa81409ac90 tx=0x7fa814010f50 comp rx=0 tx=0).stop 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 shutdown_connections 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95dae420 0x564e95db0810 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d9a210 0x564e95d97d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d91440 0x564e95d977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa7fffff640 0 client.25654 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3736462714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d90b20 0x564e95d972b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.507+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 >> 172.21.15.38:0/3736462714 conn(0x564e95d8b440 msgr2=0x564e95d8d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.508+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 shutdown_connections 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.508+0000 7fa8297c6740 1 -- 172.21.15.38:0/3736462714 wait complete. 2024-04-22T00:03:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.508+0000 7fa8297c6740 20 client.25654 trim_cache size 0 max 16384 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 Processor -- start 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 -- start start 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95db5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201acf00 con 0x564e95d91440 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201aea40 con 0x564e95d90b20 2024-04-22T00:03:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa8297c6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201ad5d0 con 0x564e95d9a210 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47030/0 (socket says 172.21.15.38:47030) 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa81ffff640 1 -- 172.21.15.38:0/3371634689 learned_addr learned my addr 172.21.15.38:0/3371634689 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa82489e640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.538+0000 7fa82509f640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95db5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 -- 172.21.15.38:0/3371634689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 msgr2=0x564e95db5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95db5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 -- 172.21.15.38:0/3371634689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 msgr2=0x564e95da5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 -- 172.21.15.38:0/3371634689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e95d95820 con 0x564e95d91440 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa82509f640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95db5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa82489e640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa81ffff640 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005940 tx=0x7fa80c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d89685f1a56e1d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa80a7fc640 1 -- 172.21.15.38:0/3371634689 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c04c070 con 0x564e95d91440 2024-04-22T00:03:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa80a7fc640 1 -- 172.21.15.38:0/3371634689 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c0041e0 con 0x564e95d91440 2024-04-22T00:03:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.539+0000 7fa80a7fc640 1 -- 172.21.15.38:0/3371634689 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c004920 con 0x564e95d91440 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 -- 172.21.15.38:0/3371634689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 msgr2=0x564e95d96180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005940 tx=0x7fa80c02e040 comp rx=0 tx=0).stop 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 -- 172.21.15.38:0/3371634689 shutdown_connections 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95db5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d96180 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 --2- 172.21.15.38:0/3371634689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.540+0000 7fa8297c6740 1 -- 172.21.15.38:0/3371634689 >> 172.21.15.38:0/3371634689 conn(0x564e95d8b440 msgr2=0x564e95d95a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.541+0000 7fa8297c6740 1 -- 172.21.15.38:0/3371634689 shutdown_connections 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.541+0000 7fa8297c6740 1 -- 172.21.15.38:0/3371634689 wait complete. 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.541+0000 7fa8297c6740 1 -- start start 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.541+0000 7fa8297c6740 1 Processor -- start 2024-04-22T00:03:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95d94080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47034/0 (socket says 172.21.15.38:47034) 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa81ffff640 1 -- 172.21.15.38:0/2564389927 learned_addr learned my addr 172.21.15.38:0/2564389927 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201ae370 con 0x564e95d91440 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201adca0 con 0x564e95d90b20 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201aaee0 con 0x564e95d9a210 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.542+0000 7fa82489e640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa82509f640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95d94080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 -- 172.21.15.38:0/2564389927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 msgr2=0x564e95d94080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95d94080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 -- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 msgr2=0x564e95da23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e95d945f0 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa82489e640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa82509f640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95d94080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa81ffff640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005ec0 tx=0x7fa80c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78bd865de2c2d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa7fffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa8297c6740 10 client.0 _ll_register_callbacks cb 0x7fa80c005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c0041e0 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa7fffff640 1 -- 172.21.15.38:0/2564389927 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c045070 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.543+0000 7fa7fffff640 1 -- 172.21.15.38:0/2564389927 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa80c0046c0 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa7fffff640 1 -- 172.21.15.38:0/2564389927 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa80c04c040 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 10 client.16284 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=28}) -- 0x564e95e14340 con 0x564e95d91440 2024-04-22T00:03:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa7fffff640 1 -- 172.21.15.38:0/2564389927 <== mon.0 v2:172.21.15.38:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa80c0048c0 con 0x564e95d91440 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 10 client.16284 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 20 client.16284 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95e14950 0x564e95e16d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 4 client.16284 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa80c0041e0 con 0x564e95e14950 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.544+0000 7fa82489e640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95e14950 0x564e95e16d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.545+0000 7fa82489e640 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95e14950 0x564e95e16d70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a05a7d54a84ce80 server_cookie=58bb2c346dbb064a in_seq=0 out_seq=0 2024-04-22T00:03:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.545+0000 7fa7fffff640 10 client.16284 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.548+0000 7fa7fffff640 1 -- 172.21.15.38:0/2564389927 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa80c0041e0 con 0x564e95e14950 2024-04-22T00:03:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.548+0000 7fa7fffff640 10 client.16284 handle_command_reply: tid=0 2024-04-22T00:03:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.548+0000 7fa8297c6740 1 client.16284 shutdown 2024-04-22T00:03:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.548+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 msgr2=0x564e95d93b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005ec0 tx=0x7fa80c005c00 comp rx=0 tx=0).stop 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 shutdown_connections 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e95d9a210 0x564e95d94080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564e95e14950 0x564e95e16d70 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e95d91440 0x564e95d93b40 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 --2- 172.21.15.38:0/2564389927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e95d90b20 0x564e95da23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 >> 172.21.15.38:0/2564389927 conn(0x564e95d8b440 msgr2=0x564e95d96300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa7fffff640 0 client.16284 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 shutdown_connections 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 1 -- 172.21.15.38:0/2564389927 wait complete. 2024-04-22T00:03:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.549+0000 7fa8297c6740 20 client.16284 trim_cache size 0 max 16384 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 msgr2=0x7fa8100a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 0x7fa8100a5490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa8180098b0 tx=0x7fa818004ab0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 msgr2=0x7fa8201981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b4a0 tx=0x7fa81400b960 comp rx=0 tx=0).stop 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 shutdown_connections 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8100a2fd0 0x7fa8100a5490 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82019cab0 0x7fa82019cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa82010bd10 0x7fa820198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 --2- 172.21.15.38:0/3639524639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8201029d0 0x7fa8201981f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 >> 172.21.15.38:0/3639524639 conn(0x7fa8200777c0 msgr2=0x7fa8200fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 shutdown_connections 2024-04-22T00:03:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:20.550+0000 7fa826b02640 1 -- 172.21.15.38:0/3639524639 wait complete. 2024-04-22T00:03:20.718 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:03:20.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:03:20.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:03:20.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:20 smithi038 ceph-mon[27972]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T00:03:20.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:20 smithi203 ceph-mon[31431]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T00:03:20.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:20 smithi111 ceph-mon[31543]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T00:03:20.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 -- 172.21.15.38:0/2203204915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc10bd00 msgr2=0x7f41fc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 --2- 172.21.15.38:0/2203204915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc10bd00 0x7f41fc10e1a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009900 tx=0x7f41ec02bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:20.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 -- 172.21.15.38:0/2203204915 shutdown_connections 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 --2- 172.21.15.38:0/2203204915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc10bd00 0x7f41fc10e1a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 --2- 172.21.15.38:0/2203204915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc1032e0 0x7f41fc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 --2- 172.21.15.38:0/2203204915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 0x7f41fc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 -- 172.21.15.38:0/2203204915 >> 172.21.15.38:0/2203204915 conn(0x7f41fc077740 msgr2=0x7f41fc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 -- 172.21.15.38:0/2203204915 shutdown_connections 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.840+0000 7f4203fbe640 1 -- 172.21.15.38:0/2203204915 wait complete. 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 Processor -- start 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 -- start start 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 0x7f41fc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc10bd00 0x7f41fc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc111b50 con 0x7f41fc1032e0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1119d0 con 0x7f41fc1029c0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4203fbe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc111cd0 con 0x7f41fc10bd00 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4201559640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4201559640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47066/0 (socket says 172.21.15.38:47066) 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4201559640 1 -- 172.21.15.38:0/844905611 learned_addr learned my addr 172.21.15.38:0/844905611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f4201d5a640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 0x7f41fc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.841+0000 7f420255b640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc10bd00 0x7f41fc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 -- 172.21.15.38:0/844905611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc10bd00 msgr2=0x7f41fc101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc10bd00 0x7f41fc101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 -- 172.21.15.38:0/844905611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 msgr2=0x7f41fc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 0x7f41fc100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 -- 172.21.15.38:0/844905611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41fc1a6470 con 0x7f41fc1032e0 2024-04-22T00:03:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4201559640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e9f0 tx=0x7f41f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f41f2ffd640 1 -- 172.21.15.38:0/844905611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800cd80 con 0x7f41fc1032e0 2024-04-22T00:03:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41fc1a6760 con 0x7f41fc1032e0 2024-04-22T00:03:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f41f2ffd640 1 -- 172.21.15.38:0/844905611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f8005500 con 0x7f41fc1032e0 2024-04-22T00:03:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f41f2ffd640 1 -- 172.21.15.38:0/844905611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8010690 con 0x7f41fc1032e0 2024-04-22T00:03:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.842+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41fc1a6c60 con 0x7f41fc1032e0 2024-04-22T00:03:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.845+0000 7f4206ba9740 1 Processor -- start 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 -- start start 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f41f2ffd640 1 -- 172.21.15.38:0/844905611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f41f80046c0 con 0x7f41fc1032e0 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213752c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x56321375e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5632136ae170 con 0x563213753250 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5632136567c0 con 0x563213753f30 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4206ba9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5632136aa130 con 0x56321375c5c0 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47076/0 (socket says 172.21.15.38:47076) 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 -- 172.21.15.38:0/3422319735 learned_addr learned my addr 172.21.15.38:0/3422319735 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f420255b640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213752c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201559640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x56321375e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 -- 172.21.15.38:0/3422319735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 msgr2=0x56321375e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x56321375e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 -- 172.21.15.38:0/3422319735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 msgr2=0x563213752c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213752c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.846+0000 7f4201d5a640 1 -- 172.21.15.38:0/3422319735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563213667e60 con 0x563213753250 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f420255b640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213752c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f41f2ffd640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 0x7f41e80a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f4201d5a640 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f41f40114f0 tx=0x7f41f40119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c40a49bdeec19c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f420255b640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 0x7f41e80a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f41f2ffd640 1 -- 172.21.15.38:0/844905611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41f80cfdb0 con 0x7f41fc1032e0 2024-04-22T00:03:20.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f41e7fff640 1 -- 172.21.15.38:0/3422319735 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f401e070 con 0x563213753250 2024-04-22T00:03:20.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.847+0000 7f41e7fff640 1 -- 172.21.15.38:0/3422319735 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f400fb00 con 0x563213753250 2024-04-22T00:03:20.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f41e7fff640 1 -- 172.21.15.38:0/3422319735 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4019bf0 con 0x563213753250 2024-04-22T00:03:20.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 -- 172.21.15.38:0/3422319735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 msgr2=0x5632137536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f420255b640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 0x7f41e80a53a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41ec006010 tx=0x7f41ec02e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f41f40114f0 tx=0x7f41f40119b0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 -- 172.21.15.38:0/3422319735 shutdown_connections 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x56321375e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x5632137536d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 --2- 172.21.15.38:0/3422319735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213752c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 -- 172.21.15.38:0/3422319735 >> 172.21.15.38:0/3422319735 conn(0x5632137114f0 msgr2=0x56321374dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 -- 172.21.15.38:0/3422319735 shutdown_connections 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.848+0000 7f4206ba9740 1 -- 172.21.15.38:0/3422319735 wait complete. 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.849+0000 7f4206ba9740 1 -- start start 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.849+0000 7f4206ba9740 1 Processor -- start 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.849+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.849+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563213753f30 0x563213759a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56321375c5c0 0x563213759f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4206ba9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56321375f4d0 con 0x56321375c5c0 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4206ba9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563213669d40 con 0x563213753250 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4206ba9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56321375f650 con 0x563213753f30 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4206ba9740 10 client.0 _ll_register_callbacks cb 0x5632136a5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:38546/0 (socket says 172.21.15.38:38546) 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f420255b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56321375c5c0 0x563213759f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 -- 172.21.15.38:0/1925557206 learned_addr learned my addr 172.21.15.38:0/1925557206 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201559640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563213753f30 0x563213759a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 -- 172.21.15.38:0/1925557206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563213753f30 msgr2=0x563213759a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563213753f30 0x563213759a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 -- 172.21.15.38:0/1925557206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56321375c5c0 msgr2=0x563213759f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56321375c5c0 0x563213759f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 -- 172.21.15.38:0/1925557206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321348a7e0 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41f4011eb0 tx=0x7f41f4009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a687677f6d2f99c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.850+0000 7f41e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f401e070 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f400fb00 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41f4004c50 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41f4004e50 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f4206ba9740 10 client.25790 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632134eaf70 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.851+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f400fb00 con 0x563213753250 2024-04-22T00:03:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f41f4005410 con 0x563213753250 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 10 client.25790 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 20 client.25790 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56321376c570 0x56321376e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 4 client.25790 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563213682450 con 0x56321376c570 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4201559640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56321376c570 0x56321376e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f4201559640 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56321376c570 0x56321376e960 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be360e66a505463 server_cookie=465e4cc4988c0d14 in_seq=0 out_seq=0 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.852+0000 7f41e57fa640 10 client.25790 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.853+0000 7f41e57fa640 1 -- 172.21.15.38:0/1925557206 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563213682450 con 0x56321376c570 2024-04-22T00:03:20.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.853+0000 7f41e57fa640 10 client.25790 handle_command_reply: tid=0 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.853+0000 7f4206ba9740 1 client.25790 shutdown 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.853+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 msgr2=0x563213759500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.853+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41f4011eb0 tx=0x7f41f4009330 comp rx=0 tx=0).stop 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 shutdown_connections 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56321376c570 0x56321376e960 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56321375c5c0 0x563213759f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563213753f30 0x563213759a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1925557206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213759500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f41e57fa640 0 client.25790 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 >> 172.21.15.38:0/1925557206 conn(0x5632137114f0 msgr2=0x56321374f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 shutdown_connections 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 1 -- 172.21.15.38:0/1925557206 wait complete. 2024-04-22T00:03:20.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.854+0000 7f4206ba9740 20 client.25790 trim_cache size 0 max 16384 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 Processor -- start 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 -- start start 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213758e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137444f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1ac850 con 0x563213753250 2024-04-22T00:03:20.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1ae390 con 0x563213753f30 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4206ba9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1acf20 con 0x56321375c5c0 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47102/0 (socket says 172.21.15.38:47102) 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 -- 172.21.15.38:0/1000992508 learned_addr learned my addr 172.21.15.38:0/1000992508 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201559640 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213758e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 -- 172.21.15.38:0/1000992508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 msgr2=0x5632137444f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 -- 172.21.15.38:0/1000992508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 msgr2=0x563213758e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213758e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.888+0000 7f4201d5a640 1 -- 172.21.15.38:0/1000992508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563213382170 con 0x563213753250 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.889+0000 7f4201d5a640 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f41f40119f0 tx=0x7f41f4012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a749483cde620b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.889+0000 7f41e77fe640 1 -- 172.21.15.38:0/1000992508 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f400fac0 con 0x563213753250 2024-04-22T00:03:20.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.889+0000 7f41e77fe640 1 -- 172.21.15.38:0/1000992508 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f4003e50 con 0x563213753250 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.889+0000 7f41e77fe640 1 -- 172.21.15.38:0/1000992508 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f402ec80 con 0x563213753250 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 -- 172.21.15.38:0/1000992508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 msgr2=0x563213768400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f41f40119f0 tx=0x7f41f4012f70 comp rx=0 tx=0).stop 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 -- 172.21.15.38:0/1000992508 shutdown_connections 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137444f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753f30 0x563213758e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 --2- 172.21.15.38:0/1000992508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753250 0x563213768400 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 -- 172.21.15.38:0/1000992508 >> 172.21.15.38:0/1000992508 conn(0x5632137114f0 msgr2=0x56321374f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 -- 172.21.15.38:0/1000992508 shutdown_connections 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.890+0000 7f4206ba9740 1 -- 172.21.15.38:0/1000992508 wait complete. 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 -- start start 2024-04-22T00:03:20.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 Processor -- start 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213764cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137563c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1adcc0 con 0x563213753f30 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1ad5f0 con 0x563213753250 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc1aa890 con 0x56321375c5c0 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.891+0000 7f4206ba9740 10 client.0 _ll_register_callbacks cb 0x7f41f401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:20.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201559640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201559640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47112/0 (socket says 172.21.15.38:47112) 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201559640 1 -- 172.21.15.38:0/4210861700 learned_addr learned my addr 172.21.15.38:0/4210861700 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213764cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f420255b640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 -- 172.21.15.38:0/4210861700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 msgr2=0x5632137563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 -- 172.21.15.38:0/4210861700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 msgr2=0x563213755e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 -- 172.21.15.38:0/4210861700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563213756bf0 con 0x563213753250 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201559640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f420255b640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:20.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f4201d5a640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213764cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41f4014010 tx=0x7f41f400f8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3203daf26fe795 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f41e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.892+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4009b50 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41f402ea00 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f4003e50 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41f4045050 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 10 client.25796 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632137d2970 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f4003e50 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f41f4004000 con 0x563213753250 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 10 client.25796 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 20 client.25796 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:20.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5632137d2fd0 0x5632137d53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 4 client.25796 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5632136bb6b0 con 0x5632137d2fd0 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.893+0000 7f4201559640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5632137d2fd0 0x5632137d53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.894+0000 7f4201559640 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5632137d2fd0 0x5632137d53f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9995a010762b1f5a server_cookie=7640d00ed93592e1 in_seq=0 out_seq=0 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.894+0000 7f41e57fa640 10 client.25796 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f41e57fa640 1 -- 172.21.15.38:0/4210861700 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5632136bb6b0 con 0x5632137d2fd0 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f41e57fa640 10 client.25796 handle_command_reply: tid=0 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 client.25796 shutdown 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 msgr2=0x563213764cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213764cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41f4014010 tx=0x7f41f400f8b0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 shutdown_connections 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56321375c5c0 0x5632137563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.895+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5632137d2fd0 0x5632137d53f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563213753f30 0x563213755e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 1 --2- 172.21.15.38:0/4210861700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563213753250 0x563213764cb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f41e57fa640 0 client.25796 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 >> 172.21.15.38:0/4210861700 conn(0x5632137114f0 msgr2=0x5632137444f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 shutdown_connections 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 1 -- 172.21.15.38:0/4210861700 wait complete. 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4206ba9740 20 client.25796 trim_cache size 0 max 16384 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:20.894 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:03:20.895 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "40c8b962-7ed8-44ed-8320-0978d83114a5", 2024-04-22T00:03:20.895 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:03:20.895 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 msgr2=0x7f41e80a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.896+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 0x7f41e80a53a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41ec006010 tx=0x7f41ec02e040 comp rx=0 tx=0).stop 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 msgr2=0x7f41fc100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e9f0 tx=0x7f41f800edf0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 shutdown_connections 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41e80a2ee0 0x7f41e80a53a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41fc10bd00 0x7f41fc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41fc1032e0 0x7f41fc100b50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 --2- 172.21.15.38:0/844905611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41fc1029c0 0x7f41fc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 >> 172.21.15.38:0/844905611 conn(0x7f41fc077740 msgr2=0x7f41fc0fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 shutdown_connections 2024-04-22T00:03:20.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:20.897+0000 7f4203fbe640 1 -- 172.21.15.38:0/844905611 wait complete. 2024-04-22T00:03:20.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:03:20.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:03:21.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 -- 172.21.15.38:0/64180437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c106810 msgr2=0x7f119c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 --2- 172.21.15.38:0/64180437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c106810 0x7f119c106bf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1198009930 tx=0x7f119802bc30 comp rx=0 tx=0).stop 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 -- 172.21.15.38:0/64180437 shutdown_connections 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 --2- 172.21.15.38:0/64180437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c1012e0 0x7f119c10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 --2- 172.21.15.38:0/64180437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c100920 0x7f119c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 --2- 172.21.15.38:0/64180437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c106810 0x7f119c106bf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 -- 172.21.15.38:0/64180437 >> 172.21.15.38:0/64180437 conn(0x7f119c077620 msgr2=0x7f119c077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 -- 172.21.15.38:0/64180437 shutdown_connections 2024-04-22T00:03:21.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.060+0000 7f11a39a0640 1 -- 172.21.15.38:0/64180437 wait complete. 2024-04-22T00:03:21.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 Processor -- start 2024-04-22T00:03:21.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 -- start start 2024-04-22T00:03:21.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c1012e0 0x7f119c196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 0x7f119c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10f9e0 con 0x7f119c100920 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10f860 con 0x7f119c1012e0 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a39a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10fb60 con 0x7f119c106810 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47122/0 (socket says 172.21.15.38:47122) 2024-04-22T00:03:21.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a173c640 1 -- 172.21.15.38:0/428484242 learned_addr learned my addr 172.21.15.38:0/428484242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c1012e0 0x7f119c196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.061+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 0x7f119c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 -- 172.21.15.38:0/428484242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 msgr2=0x7f119c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 0x7f119c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 -- 172.21.15.38:0/428484242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c1012e0 msgr2=0x7f119c196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c1012e0 0x7f119c196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 -- 172.21.15.38:0/428484242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c19ae30 con 0x7f119c100920 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 0x7f119c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a173c640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1198009230 tx=0x7f119802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11927fc640 1 -- 172.21.15.38:0/428484242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198041070 con 0x7f119c100920 2024-04-22T00:03:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11927fc640 1 -- 172.21.15.38:0/428484242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1198004450 con 0x7f119c100920 2024-04-22T00:03:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c19b0c0 con 0x7f119c100920 2024-04-22T00:03:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1a29b0 con 0x7f119c100920 2024-04-22T00:03:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.062+0000 7f11927fc640 1 -- 172.21.15.38:0/428484242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198003e50 con 0x7f119c100920 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11927fc640 1 -- 172.21.15.38:0/428484242 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f1198044070 con 0x7f119c100920 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 Processor -- start 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 -- start start 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3c2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5578ed32c560 con 0x5578ed3c2730 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5578ed2d4730 con 0x5578ed3d5f80 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.066+0000 7f11a6582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5578ed328520 con 0x5578ed3d55d0 2024-04-22T00:03:21.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47130/0 (socket says 172.21.15.38:47130) 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 -- 172.21.15.38:0/4208203240 learned_addr learned my addr 172.21.15.38:0/4208203240 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a173c640 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3c2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11927fc640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 0x7f11880a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 -- 172.21.15.38:0/4208203240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 msgr2=0x5578ed3d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 -- 172.21.15.38:0/4208203240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 msgr2=0x5578ed3c2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3c2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 -- 172.21.15.38:0/4208203240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ed10a7d0 con 0x5578ed3c2730 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11927fc640 1 -- 172.21.15.38:0/428484242 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f11980f1160 con 0x7f119c100920 2024-04-22T00:03:21.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 0x7f11880a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.067+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f118c009c90 tx=0x7f118c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7afbd8546cb86d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.068+0000 7f11877fe640 1 -- 172.21.15.38:0/4208203240 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c005220 con 0x5578ed3c2730 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.068+0000 7f11877fe640 1 -- 172.21.15.38:0/4208203240 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118c004c80 con 0x5578ed3c2730 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.068+0000 7f11877fe640 1 -- 172.21.15.38:0/4208203240 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c019930 con 0x5578ed3c2730 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- 172.21.15.38:0/4208203240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 msgr2=0x5578ed3c2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f118c009c90 tx=0x7f118c011810 comp rx=0 tx=0).stop 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 0x7f11880a5610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1194005fd0 tx=0x7f119400be50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- 172.21.15.38:0/4208203240 shutdown_connections 2024-04-22T00:03:21.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3c2b90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3c2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 --2- 172.21.15.38:0/4208203240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- 172.21.15.38:0/4208203240 >> 172.21.15.38:0/4208203240 conn(0x5578ed3c4bd0 msgr2=0x5578ed3c4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- 172.21.15.38:0/4208203240 shutdown_connections 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- 172.21.15.38:0/4208203240 wait complete. 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 -- start start 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.069+0000 7f11a6582740 1 Processor -- start 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a6582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a6582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a6582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47132/0 (socket says 172.21.15.38:47132) 2024-04-22T00:03:21.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a173c640 1 -- 172.21.15.38:0/3721901594 learned_addr learned my addr 172.21.15.38:0/3721901594 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.070+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5578ed3dca10 con 0x5578ed3c2730 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5578ed2e7e00 con 0x5578ed3d5f80 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5578ed3dcb90 con 0x5578ed3d55d0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a6582740 10 client.0 _ll_register_callbacks cb 0x5578ed322eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 -- 172.21.15.38:0/3721901594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 msgr2=0x5578ed3d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 -- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 msgr2=0x5578ed3d9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ed0403f0 con 0x5578ed3c2730 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a173c640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f11980b9670 tx=0x7f1198000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fabcc5ff148c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f1184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:21.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119804b070 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f11980dd0c0 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.071+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f118c004c80 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11980bcb70 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 10 client.16314 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5578ed0000f0 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118c004c80 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11980c59f0 con 0x5578ed3c2730 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 10 client.16314 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 20 client.16314 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed3e9f30 0x5578ed3ec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 4 client.16314 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:21.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.072+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578ed339aa0 con 0x5578ed3e9f30 2024-04-22T00:03:21.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.073+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed3e9f30 0x5578ed3ec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.073+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed3e9f30 0x5578ed3ec3d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef2d0388abd5a66d server_cookie=6e98ed39399a772a in_seq=0 out_seq=0 2024-04-22T00:03:21.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.073+0000 7f1184ff9640 10 client.16314 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:21.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f1184ff9640 1 -- 172.21.15.38:0/3721901594 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578ed339aa0 con 0x5578ed3e9f30 2024-04-22T00:03:21.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f1184ff9640 10 client.16314 handle_command_reply: tid=0 2024-04-22T00:03:21.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 client.16314 shutdown 2024-04-22T00:03:21.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 msgr2=0x5578ed3d94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f11980b9670 tx=0x7f1198000bc0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 shutdown_connections 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed3e9f30 0x5578ed3ec3d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 --2- 172.21.15.38:0/3721901594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3d94f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 >> 172.21.15.38:0/3721901594 conn(0x5578ed3c4bd0 msgr2=0x5578ed3c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f1184ff9640 0 client.16314 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 shutdown_connections 2024-04-22T00:03:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 1 -- 172.21.15.38:0/3721901594 wait complete. 2024-04-22T00:03:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.074+0000 7f11a6582740 20 client.16314 trim_cache size 0 max 16384 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 Processor -- start 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 -- start start 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3c2730 0x5578ed3dbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f118000fd70 con 0x5578ed3d55d0 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f11800118b0 con 0x5578ed3d5f80 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a6582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1180010440 con 0x5578ed3c2730 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a0f3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a0f3b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47142/0 (socket says 172.21.15.38:47142) 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a0f3b640 1 -- 172.21.15.38:0/3867444541 learned_addr learned my addr 172.21.15.38:0/3867444541 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a173c640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3c2730 0x5578ed3dbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.109+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 -- 172.21.15.38:0/3867444541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3c2730 msgr2=0x5578ed3dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3c2730 0x5578ed3dbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 -- 172.21.15.38:0/3867444541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 msgr2=0x5578ed3d4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 -- 172.21.15.38:0/3867444541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ed3c2480 con 0x5578ed3d55d0 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f118c019540 tx=0x7f118c019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e45aa1e6631ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f1186ffd640 1 -- 172.21.15.38:0/3867444541 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c031070 con 0x5578ed3d55d0 2024-04-22T00:03:21.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f1186ffd640 1 -- 172.21.15.38:0/3867444541 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118c02cbd0 con 0x5578ed3d55d0 2024-04-22T00:03:21.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.110+0000 7f1186ffd640 1 -- 172.21.15.38:0/3867444541 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c02c740 con 0x5578ed3d55d0 2024-04-22T00:03:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 -- 172.21.15.38:0/3867444541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 msgr2=0x5578ed3e5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f118c019540 tx=0x7f118c019570 comp rx=0 tx=0).stop 2024-04-22T00:03:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 -- 172.21.15.38:0/3867444541 shutdown_connections 2024-04-22T00:03:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3d55d0 0x5578ed3e5940 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 --2- 172.21.15.38:0/3867444541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3c2730 0x5578ed3dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 -- 172.21.15.38:0/3867444541 >> 172.21.15.38:0/3867444541 conn(0x5578ed3c4bd0 msgr2=0x5578ed3d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 -- 172.21.15.38:0/3867444541 shutdown_connections 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.111+0000 7f11a6582740 1 -- 172.21.15.38:0/3867444541 wait complete. 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 -- start start 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 Processor -- start 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f11800111e0 con 0x5578ed3c2730 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1180010b10 con 0x5578ed3d5f80 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f118000dd50 con 0x5578ed3d55d0 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a6582740 10 client.0 _ll_register_callbacks cb 0x7f1198009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.112+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47150/0 (socket says 172.21.15.38:47150) 2024-04-22T00:03:21.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 -- 172.21.15.38:0/444338911 learned_addr learned my addr 172.21.15.38:0/444338911 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a1f3d640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 -- 172.21.15.38:0/444338911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 msgr2=0x5578ed3d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 -- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 msgr2=0x5578ed3d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 -- 172.21.15.38:0/444338911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ed3f1160 con 0x5578ed3c2730 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a173c640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f11980dd3b0 tx=0x7f11980dd430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4ba991d1a61d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f1184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119804b040 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f11980bc660 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f118c02cbd0 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11980037f0 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a6582740 10 client.16320 fetch_fsmap learned FSMap version 28 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.113+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5578ed450890 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118c02cbd0 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11980c1170 con 0x5578ed3c2730 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 10 client.16320 fetch_fsmap finished waiting for FSMap version 28 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:21.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 20 client.16320 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed450e20 0x5578ed453240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 4 client.16320 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5578ed333df0 con 0x5578ed450e20 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.114+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed450e20 0x5578ed453240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f1184ff9640 10 client.16320 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a0f3b640 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed450e20 0x5578ed453240 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47cad051093afaa7 server_cookie=b81ef753918602b0 in_seq=0 out_seq=0 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f1184ff9640 1 -- 172.21.15.38:0/444338911 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5578ed333df0 con 0x5578ed450e20 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f1184ff9640 10 client.16320 handle_command_reply: tid=0 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 client.16320 shutdown 2024-04-22T00:03:21.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 msgr2=0x5578ed3e2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f11980dd3b0 tx=0x7f11980dd430 comp rx=0 tx=0).stop 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 shutdown_connections 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578ed3d5f80 0x5578ed3d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578ed3d55d0 0x5578ed3d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578ed450e20 0x5578ed453240 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f1184ff9640 0 client.16320 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 --2- 172.21.15.38:0/444338911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578ed3c2730 0x5578ed3e2330 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 >> 172.21.15.38:0/444338911 conn(0x5578ed3c4bd0 msgr2=0x5578ed3c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 shutdown_connections 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 1 -- 172.21.15.38:0/444338911 wait complete. 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.115+0000 7f11a6582740 20 client.16320 trim_cache size 0 max 16384 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stdout: "40c8b962-7ed8-44ed-8320-0978d83114a5": { 2024-04-22T00:03:21.119 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:03:21.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "40c8b962-7ed8-44ed-8320-0978d83114a5", 2024-04-22T00:03:21.120 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:03:21.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:03:21.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:03:21.120 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:21.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 msgr2=0x7f11880a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 0x7f11880a5610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1194005fd0 tx=0x7f119400be50 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 msgr2=0x7f119c195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1198009230 tx=0x7f119802cb30 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 shutdown_connections 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11880a3150 0x7f11880a5610 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f119c106810 0x7f119c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f119c1012e0 0x7f119c196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 --2- 172.21.15.38:0/428484242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f119c100920 0x7f119c195e40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 >> 172.21.15.38:0/428484242 conn(0x7f119c077620 msgr2=0x7f119c0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:21.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 shutdown_connections 2024-04-22T00:03:21.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:21.116+0000 7f11a39a0640 1 -- 172.21.15.38:0/428484242 wait complete. 2024-04-22T00:03:21.123 INFO:tasks.cephfs.filesystem:scrub status for tag:40c8b962-7ed8-44ed-8320-0978d83114a5 - {'path': '/', 'tag': '40c8b962-7ed8-44ed-8320-0978d83114a5', 'options': 'recursive,force'} 2024-04-22T00:03:22.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:22 smithi203 ceph-mon[31431]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-22T00:03:22.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:22 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:03:22.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:22 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:22.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:22 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:22 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:22.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:22 smithi111 ceph-mon[31543]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-22T00:03:22.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:22 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:03:22.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:22 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:22.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:22 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:22.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:22 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:22.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:22 smithi038 ceph-mon[27972]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-22T00:03:22.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:22 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:03:22.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:22 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:22.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:22 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:22.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:22 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:24.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:24 smithi203 ceph-mon[31431]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:24.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:24 smithi111 ceph-mon[31543]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:24.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:24 smithi038 ceph-mon[27972]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:26.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:26 smithi203 ceph-mon[31431]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:26.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:26 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:03:26.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:26 smithi111 ceph-mon[31543]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:26.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:26 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:03:26.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:26 smithi038 ceph-mon[27972]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T00:03:26.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:26 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:03:27.779 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:27 smithi203 ceph-mon[31431]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:03:27.826 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:27 smithi111 ceph-mon[31543]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:03:27.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:27 smithi038 ceph-mon[27972]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-22T00:03:29.289 INFO:tasks.workunit.client.0.smithi038.stderr:Updating files: 93% (11653/12485) Updating files: 94% (11736/12485) Updating files: 95% (11861/12485) Updating files: 96% (11986/12485) Updating files: 97% (12111/12485) Updating files: 98% (12236/12485) Updating files: 99% (12361/12485) Updating files: 100% (12485/12485) Updating files: 100% (12485/12485), done. 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr:Note: switching to '430e09df97c8fc7dc2b2ae424f68ed11366c540f'. 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T00:03:29.729 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: git switch -c 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr:Or undo this operation with: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: git switch - 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-04-22T00:03:29.730 INFO:tasks.workunit.client.0.smithi038.stderr:HEAD is now at 430e09df97c Merge pull request #57017 from zdover23/wip-doc-2024-04-21-backport-56904-to-squid 2024-04-22T00:03:29.733 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T00:03:29.758 INFO:tasks.workunit.client.0.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T00:03:29.760 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T00:03:29.760 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T00:03:30.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:29 smithi038 ceph-mon[27972]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:03:30.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:29 smithi203 ceph-mon[31431]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:03:30.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:29 smithi111 ceph-mon[31543]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T00:03:30.433 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T00:03:30.486 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T00:03:30.528 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T00:03:30.529 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T00:03:30.529 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T00:03:30.570 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T00:03:30.574 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:03:30.574 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T00:03:30.630 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-22T00:03:30.631 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-22T00:03:30.632 DEBUG:teuthology.orchestra.run.smithi038:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-22T00:03:30.693 INFO:tasks.workunit.client.0.smithi038.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-22T00:03:30.694 INFO:tasks.workunit.client.0.smithi038.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-22T00:03:30.694 INFO:tasks.workunit.client.0.smithi038.stderr:+ trap -- retry EXIT 2024-04-22T00:03:30.694 INFO:tasks.workunit.client.0.smithi038.stderr:+ rm -rf ffsb 2024-04-22T00:03:30.694 INFO:tasks.workunit.client.0.smithi038.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-22T00:03:30.697 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into 'ffsb'... 2024-04-22T00:03:31.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ trap - EXIT 2024-04-22T00:03:31.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd ffsb 2024-04-22T00:03:31.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ ./configure 2024-04-22T00:03:31.819 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-22T00:03:32.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:31 smithi038 ceph-mon[27972]: pgmap v46: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-22T00:03:32.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:31 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:32.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:31 smithi203 ceph-mon[31431]: pgmap v46: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-22T00:03:32.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:31 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:32.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:31 smithi111 ceph-mon[31543]: pgmap v46: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-22T00:03:32.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:31 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:32.826 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether build environment is sane... yes 2024-04-22T00:03:32.838 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-22T00:03:32.838 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gawk... gawk 2024-04-22T00:03:32.852 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether make sets $(MAKE)... yes 2024-04-22T00:03:32.886 INFO:tasks.workunit.client.0.smithi038.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-22T00:03:32.886 INFO:tasks.workunit.client.0.smithi038.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-22T00:03:32.898 INFO:tasks.workunit.client.0.smithi038.stdout:checking for style of include used by make... GNU 2024-04-22T00:03:32.903 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... gcc 2024-04-22T00:03:32.960 INFO:tasks.workunit.client.0.smithi038.stdout:checking for C compiler default output file name... a.out 2024-04-22T00:03:32.962 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether the C compiler works... yes 2024-04-22T00:03:32.966 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are cross compiling... no 2024-04-22T00:03:33.006 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of executables... 2024-04-22T00:03:33.044 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of object files... o 2024-04-22T00:03:33.080 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... yes 2024-04-22T00:03:33.117 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... yes 2024-04-22T00:03:33.198 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-22T00:03:33.329 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... gcc3 2024-04-22T00:03:33.392 INFO:tasks.workunit.client.0.smithi038.stdout:checking how to run the C preprocessor... gcc -E 2024-04-22T00:03:33.442 INFO:tasks.workunit.client.0.smithi038.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-22T00:03:33.443 INFO:tasks.workunit.client.0.smithi038.stdout:checking for egrep... /usr/bin/grep -E 2024-04-22T00:03:33.453 INFO:tasks.workunit.client.0.smithi038.stdout:checking for AIX... no 2024-04-22T00:03:33.474 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... (cached) gcc 2024-04-22T00:03:33.482 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-22T00:03:33.483 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-22T00:03:33.483 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-22T00:03:33.483 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-22T00:03:33.532 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lm... yes 2024-04-22T00:03:33.582 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lpthread... yes 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 -- 172.21.15.38:0/3987244490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8101020 msgr2=0x7f6ae810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/3987244490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8101020 0x7f6ae810d530 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009cd0 tx=0x7f6ae002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 -- 172.21.15.38:0/3987244490 shutdown_connections 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/3987244490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8101020 0x7f6ae810d530 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/3987244490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8100660 0x7f6ae8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/3987244490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8106670 0x7f6ae8106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 -- 172.21.15.38:0/3987244490 >> 172.21.15.38:0/3987244490 conn(0x7f6ae80769c0 msgr2=0x7f6ae8076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 -- 172.21.15.38:0/3987244490 shutdown_connections 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.641+0000 7f6aeea0e640 1 -- 172.21.15.38:0/3987244490 wait complete. 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 Processor -- start 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 -- start start 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8100660 0x7f6ae8198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8101020 0x7f6ae8199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8110a80 con 0x7f6ae8106670 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8110900 con 0x7f6ae8101020 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.642+0000 7f6aeea0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8110c00 con 0x7f6ae8100660 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60140/0 (socket says 172.21.15.38:60140) 2024-04-22T00:03:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8100660 0x7f6ae8198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 -- 172.21.15.38:0/483532566 learned_addr learned my addr 172.21.15.38:0/483532566 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8101020 0x7f6ae8199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 -- 172.21.15.38:0/483532566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8100660 msgr2=0x7f6ae8198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8100660 0x7f6ae8198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 -- 172.21.15.38:0/483532566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8101020 msgr2=0x7f6ae8199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8101020 0x7f6ae8199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 -- 172.21.15.38:0/483532566 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae81a4400 con 0x7f6ae8106670 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aecfab640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00348f0 tx=0x7f6ae0032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6ae57fa640 1 -- 172.21.15.38:0/483532566 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0041020 con 0x7f6ae8106670 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae81a45d0 con 0x7f6ae8106670 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6ae57fa640 1 -- 172.21.15.38:0/483532566 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae0032e80 con 0x7f6ae8106670 2024-04-22T00:03:33.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6ae57fa640 1 -- 172.21.15.38:0/483532566 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0039040 con 0x7f6ae8106670 2024-04-22T00:03:33.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.643+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81a49d0 con 0x7f6ae8106670 2024-04-22T00:03:33.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.645+0000 7f6ae57fa640 1 -- 172.21.15.38:0/483532566 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) ==== 138117+0+0 (secure 0 0 0) 0x7f6ae0044020 con 0x7f6ae8106670 2024-04-22T00:03:33.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 Processor -- start 2024-04-22T00:03:33.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6ae57fa640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 0x7f6ad00a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 -- start start 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 0x7f6ad00a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6ae57fa640 1 -- 172.21.15.38:0/483532566 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6ae00f1540 con 0x7f6ae8106670 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c36fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fc440 0x55a0c3704cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3707600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a0c365fb70 con 0x55a0c36fbb20 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a0c3608470 con 0x55a0c3705210 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6af15f6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a0c365bb30 con 0x55a0c36fc440 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c36fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3707600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.646+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c36fbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60152/0 (socket says 172.21.15.38:60152) 2024-04-22T00:03:33.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ae7fff640 1 -- 172.21.15.38:0/2829274804 learned_addr learned my addr 172.21.15.38:0/2829274804 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fc440 0x55a0c3704cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 -- 172.21.15.38:0/2829274804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fc440 msgr2=0x55a0c3704cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fc440 0x55a0c3704cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 -- 172.21.15.38:0/2829274804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 msgr2=0x55a0c36fbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c36fbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 -- 172.21.15.38:0/2829274804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0c3374210 con 0x55a0c3705210 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 0x7f6ad00a5410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80134b0 tx=0x7f6ad8013970 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6aecfab640 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3707600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00ee820 tx=0x7f6ae00b9890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a9cd031cfded5f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ace7fc640 1 -- 172.21.15.38:0/2829274804 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00bcf50 con 0x55a0c3705210 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ace7fc640 1 -- 172.21.15.38:0/2829274804 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae0034920 con 0x55a0c3705210 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.647+0000 7f6ace7fc640 1 -- 172.21.15.38:0/2829274804 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00b9c60 con 0x55a0c3705210 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- 172.21.15.38:0/2829274804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 msgr2=0x55a0c3707600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3707600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00ee820 tx=0x7f6ae00b9890 comp rx=0 tx=0).stop 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- 172.21.15.38:0/2829274804 shutdown_connections 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3707600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fc440 0x55a0c3704cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 --2- 172.21.15.38:0/2829274804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c36fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- 172.21.15.38:0/2829274804 >> 172.21.15.38:0/2829274804 conn(0x55a0c36f6440 msgr2=0x55a0c36f6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- 172.21.15.38:0/2829274804 shutdown_connections 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- 172.21.15.38:0/2829274804 wait complete. 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.648+0000 7f6af15f6740 1 -- start start 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 Processor -- start 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 0x55a0c37022a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3702d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a0c3707c00 con 0x55a0c36fc440 2024-04-22T00:03:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a0c361ba10 con 0x55a0c3705210 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a0c3707d80 con 0x55a0c36fbb20 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6af15f6740 10 client.0 _ll_register_callbacks cb 0x55a0c3656b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60156/0 (socket says 172.21.15.38:60156) 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 -- 172.21.15.38:0/641359443 learned_addr learned my addr 172.21.15.38:0/641359443 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6aecfab640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3702d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 0x55a0c37022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 -- 172.21.15.38:0/641359443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 msgr2=0x55a0c37022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 0x55a0c37022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 -- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 msgr2=0x55a0c3702d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3702d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 -- 172.21.15.38:0/641359443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0c3440620 con 0x55a0c36fc440 2024-04-22T00:03:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 0x55a0c37022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6adc012720 tx=0x7f6adc012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2807638bda13d52e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.649+0000 7f6ac77fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00f920 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6adc00fac0 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae0034920 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6adc00fd00 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 10 client.16332 fetch_fsmap learned FSMap version 29 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0c3713e90 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae0034920 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f6adc010530 con 0x55a0c36fc440 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 10 client.16332 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 20 client.16332 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c37153a0 0x55a0c37177f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 4 client.16332 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0c366d0b0 con 0x55a0c37153a0 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.650+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c37153a0 0x55a0c37177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.651+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c37153a0 0x55a0c37177f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab148e1178a6c23c server_cookie=cb1a618f358fc128 in_seq=0 out_seq=0 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.651+0000 7f6ac77fe640 10 client.16332 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.652+0000 7f6ac77fe640 1 -- 172.21.15.38:0/641359443 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0c366d0b0 con 0x55a0c37153a0 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.652+0000 7f6ac77fe640 10 client.16332 handle_command_reply: tid=0 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.652+0000 7f6af15f6740 1 client.16332 shutdown 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 msgr2=0x55a0c37027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6adc012720 tx=0x7f6adc012be0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 shutdown_connections 2024-04-22T00:03:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c3705210 0x55a0c3702d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c37153a0 0x55a0c37177f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c37027e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6ac77fe640 0 client.16332 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 --2- 172.21.15.38:0/641359443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c36fbb20 0x55a0c37022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 >> 172.21.15.38:0/641359443 conn(0x55a0c36f6440 msgr2=0x55a0c36f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 shutdown_connections 2024-04-22T00:03:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 1 -- 172.21.15.38:0/641359443 wait complete. 2024-04-22T00:03:33.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.653+0000 7f6af15f6740 20 client.16332 trim_cache size 0 max 16384 2024-04-22T00:03:33.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.668+0000 7f6af15f6740 1 Processor -- start 2024-04-22T00:03:33.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.668+0000 7f6af15f6740 1 -- start start 2024-04-22T00:03:33.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.668+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.668+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fc440 0x55a0c3710a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c371d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6af15f6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81aea10 con 0x55a0c36fbb20 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6af15f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81b0580 con 0x55a0c36fc440 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6af15f6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81af0e0 con 0x55a0c3705210 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60158/0 (socket says 172.21.15.38:60158) 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 -- 172.21.15.38:0/1702558188 learned_addr learned my addr 172.21.15.38:0/1702558188 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fc440 0x55a0c3710a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6aecfab640 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c371d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 -- 172.21.15.38:0/1702558188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 msgr2=0x55a0c371d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c371d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.666 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 -- 172.21.15.38:0/1702558188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fc440 msgr2=0x55a0c3710a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fc440 0x55a0c3710a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 -- 172.21.15.38:0/1702558188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0c36f5f70 con 0x55a0c36fbb20 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d03f83e2b2f860c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6acdffb640 1 -- 172.21.15.38:0/1702558188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8029bc0 con 0x55a0c36fbb20 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6acdffb640 1 -- 172.21.15.38:0/1702558188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ad8029890 con 0x55a0c36fbb20 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.669+0000 7f6acdffb640 1 -- 172.21.15.38:0/1702558188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8003ac0 con 0x55a0c36fbb20 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- 172.21.15.38:0/1702558188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 msgr2=0x55a0c37049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8012040 comp rx=0 tx=0).stop 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- 172.21.15.38:0/1702558188 shutdown_connections 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c371d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fc440 0x55a0c3710a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1702558188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fbb20 0x55a0c37049d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- 172.21.15.38:0/1702558188 >> 172.21.15.38:0/1702558188 conn(0x55a0c36f6440 msgr2=0x55a0c36fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- 172.21.15.38:0/1702558188 shutdown_connections 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- 172.21.15.38:0/1702558188 wait complete. 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 -- start start 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.670+0000 7f6af15f6740 1 Processor -- start 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c36feb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c36ff0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81afe80 con 0x55a0c36fc440 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81af7b0 con 0x55a0c36fbb20 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81ac9f0 con 0x55a0c3705210 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6af15f6740 10 client.0 _ll_register_callbacks cb 0x7f6adc009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.668 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c36feb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43070/0 (socket says 172.21.15.38:43070) 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae7fff640 1 -- 172.21.15.38:0/1478158762 learned_addr learned my addr 172.21.15.38:0/1478158762 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6aecfab640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c36ff0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 -- 172.21.15.38:0/1478158762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 msgr2=0x55a0c36ff0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c36ff0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 -- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 msgr2=0x55a0c370d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 -- 172.21.15.38:0/1478158762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0c36ff770 con 0x55a0c36fc440 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ac77fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ae77fe640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c36feb70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc00fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c45cf3be55256a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:33.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.671+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc02e060 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6adc010640 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ad8029890 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6adc00fea0 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 10 client.16338 fetch_fsmap learned FSMap version 29 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0c377b7d0 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ad8029890 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f6adc0039d0 con 0x55a0c36fc440 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 10 client.16338 fetch_fsmap finished waiting for FSMap version 29 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 20 client.16338 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c377bf80 0x55a0c377e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 4 client.16338 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0c3667400 con 0x55a0c377bf80 2024-04-22T00:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.672+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c377bf80 0x55a0c377e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.673+0000 7f6ae7fff640 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c377bf80 0x55a0c377e3a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aadf7235bc2506f server_cookie=bd2f6199eea6d3c3 in_seq=0 out_seq=0 2024-04-22T00:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.673+0000 7f6ac77fe640 10 client.16338 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6ac77fe640 1 -- 172.21.15.38:0/1478158762 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0c3667400 con 0x55a0c377bf80 2024-04-22T00:03:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6ac77fe640 10 client.16338 handle_command_reply: tid=0 2024-04-22T00:03:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 client.16338 shutdown 2024-04-22T00:03:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 msgr2=0x55a0c36feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c36feb70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc00fb50 comp rx=0 tx=0).stop 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 shutdown_connections 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0c3705210 0x55a0c36ff0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a0c377bf80 0x55a0c377e3a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a0c36fc440 0x55a0c36feb70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 --2- 172.21.15.38:0/1478158762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a0c36fbb20 0x55a0c370d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6ac77fe640 0 client.16338 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 >> 172.21.15.38:0/1478158762 conn(0x55a0c36f6440 msgr2=0x55a0c36f7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 shutdown_connections 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 1 -- 172.21.15.38:0/1478158762 wait complete. 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.799+0000 7f6af15f6740 20 client.16338 trim_cache size 0 max 16384 2024-04-22T00:03:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:03:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:03:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:03:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 msgr2=0x7f6ad00a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 0x7f6ad00a5410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80134b0 tx=0x7f6ad8013970 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 msgr2=0x7f6ae81a3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00348f0 tx=0x7f6ae0032c90 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 shutdown_connections 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6ad00a2f50 0x7f6ad00a5410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ae8106670 0x7f6ae81a3ec0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae8101020 0x7f6ae8199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 --2- 172.21.15.38:0/483532566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ae8100660 0x7f6ae8198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 >> 172.21.15.38:0/483532566 conn(0x7f6ae80769c0 msgr2=0x7f6ae80fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 shutdown_connections 2024-04-22T00:03:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:33.800+0000 7f6aeea0e640 1 -- 172.21.15.38:0/483532566 wait complete. 2024-04-22T00:03:33.812 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... yes 2024-04-22T00:03:33.868 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-22T00:03:33.918 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/types.h... yes 2024-04-22T00:03:33.965 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/stat.h... yes 2024-04-22T00:03:34.012 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdlib.h... yes 2024-04-22T00:03:34.059 INFO:tasks.workunit.client.0.smithi038.stdout:checking for string.h... yes 2024-04-22T00:03:34.107 INFO:tasks.workunit.client.0.smithi038.stdout:checking for memory.h... yes 2024-04-22T00:03:34.154 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strings.h... yes 2024-04-22T00:03:34.202 INFO:tasks.workunit.client.0.smithi038.stdout:checking for inttypes.h... yes 2024-04-22T00:03:34.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:33 smithi038 ceph-mon[27972]: pgmap v47: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:34.249 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... yes 2024-04-22T00:03:34.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:33 smithi203 ceph-mon[31431]: pgmap v47: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:34.298 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... yes 2024-04-22T00:03:34.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:33 smithi111 ceph-mon[31543]: pgmap v47: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:34.360 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h usability... yes 2024-04-22T00:03:34.381 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h presence... yes 2024-04-22T00:03:34.382 INFO:tasks.workunit.client.0.smithi038.stdout:checking for pthread.h... yes 2024-04-22T00:03:34.435 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h usability... yes 2024-04-22T00:03:34.456 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h presence... yes 2024-04-22T00:03:34.456 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fcntl.h... yes 2024-04-22T00:03:34.509 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h usability... yes 2024-04-22T00:03:34.530 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h presence... yes 2024-04-22T00:03:34.530 INFO:tasks.workunit.client.0.smithi038.stdout:checking for limits.h... yes 2024-04-22T00:03:34.545 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... (cached) yes 2024-04-22T00:03:34.588 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h usability... yes 2024-04-22T00:03:34.608 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h presence... yes 2024-04-22T00:03:34.609 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/time.h... yes 2024-04-22T00:03:34.624 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... (cached) yes 2024-04-22T00:03:34.667 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h usability... yes 2024-04-22T00:03:34.687 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h presence... yes 2024-04-22T00:03:34.688 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/vfs.h... yes 2024-04-22T00:03:34.735 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h usability... no 2024-04-22T00:03:34.756 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h presence... no 2024-04-22T00:03:34.756 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/limits.h... no 2024-04-22T00:03:34.791 INFO:tasks.workunit.client.0.smithi038.stdout:checking for an ANSI C-conforming const... yes 2024-04-22T00:03:34.826 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-22T00:03:34.827 INFO:tasks.workunit.client.0.smithi038.stdout:checking for function prototypes... yes 2024-04-22T00:03:34.857 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether setvbuf arguments are reversed... no 2024-04-22T00:03:34.933 INFO:tasks.workunit.client.0.smithi038.stdout:checking for system... yes 2024-04-22T00:03:34.995 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gettimeofday... yes 2024-04-22T00:03:35.059 INFO:tasks.workunit.client.0.smithi038.stdout:checking for mkdir... yes 2024-04-22T00:03:35.121 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strerror... yes 2024-04-22T00:03:35.183 INFO:tasks.workunit.client.0.smithi038.stdout:checking for open64... yes 2024-04-22T00:03:35.246 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stat64... yes 2024-04-22T00:03:35.309 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fseeko64... yes 2024-04-22T00:03:35.372 INFO:tasks.workunit.client.0.smithi038.stdout:checking for lrand48_r... yes 2024-04-22T00:03:35.435 INFO:tasks.workunit.client.0.smithi038.stdout:checking for srand48_r... yes 2024-04-22T00:03:35.458 INFO:tasks.workunit.client.0.smithi038.stdout:configure: creating ./config.status 2024-04-22T00:03:35.556 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating Makefile 2024-04-22T00:03:35.569 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-04-22T00:03:35.587 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing depfiles commands 2024-04-22T00:03:35.639 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing default commands 2024-04-22T00:03:35.694 INFO:tasks.workunit.client.0.smithi038.stderr:+ make 2024-04-22T00:03:35.702 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-22T00:03:35.704 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-22T00:03:35.705 INFO:tasks.workunit.client.0.smithi038.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-22T00:03:35.705 INFO:tasks.workunit.client.0.smithi038.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-22T00:03:35.705 INFO:tasks.workunit.client.0.smithi038.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-22T00:03:35.705 INFO:tasks.workunit.client.0.smithi038.stderr: any GNU archive site. 2024-04-22T00:03:35.707 INFO:tasks.workunit.client.0.smithi038.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-22T00:03:35.708 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-22T00:03:35.709 INFO:tasks.workunit.client.0.smithi038.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-22T00:03:35.709 INFO:tasks.workunit.client.0.smithi038.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-22T00:03:35.709 INFO:tasks.workunit.client.0.smithi038.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-22T00:03:35.709 INFO:tasks.workunit.client.0.smithi038.stderr: Grab them from any GNU archive site. 2024-04-22T00:03:35.780 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-22T00:03:36.118 INFO:tasks.workunit.client.0.smithi038.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-22T00:03:36.118 INFO:tasks.workunit.client.0.smithi038.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-22T00:03:36.118 INFO:tasks.workunit.client.0.smithi038.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-22T00:03:36.118 INFO:tasks.workunit.client.0.smithi038.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-22T00:03:36.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:35 smithi038 ceph-mon[27972]: pgmap v48: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:36.244 INFO:tasks.workunit.client.0.smithi038.stdout:/bin/sh ./config.status --recheck 2024-04-22T00:03:36.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:35 smithi203 ceph-mon[31431]: pgmap v48: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:36.297 INFO:tasks.workunit.client.0.smithi038.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-22T00:03:36.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:35 smithi111 ceph-mon[31543]: pgmap v48: 129 pgs: 129 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-22T00:03:36.407 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-22T00:03:37.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:36 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:37.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:36 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:37.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:36 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:37.415 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether build environment is sane... yes 2024-04-22T00:03:37.420 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-22T00:03:37.420 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gawk... gawk 2024-04-22T00:03:37.436 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether make sets $(MAKE)... yes 2024-04-22T00:03:37.468 INFO:tasks.workunit.client.0.smithi038.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-22T00:03:37.468 INFO:tasks.workunit.client.0.smithi038.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-22T00:03:37.480 INFO:tasks.workunit.client.0.smithi038.stdout:checking for style of include used by make... GNU 2024-04-22T00:03:37.485 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... gcc 2024-04-22T00:03:37.579 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether the C compiler works... yes 2024-04-22T00:03:37.579 INFO:tasks.workunit.client.0.smithi038.stdout:checking for C compiler default output file name... a.out 2024-04-22T00:03:37.619 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of executables... 2024-04-22T00:03:37.658 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are cross compiling... no 2024-04-22T00:03:37.699 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of object files... o 2024-04-22T00:03:37.723 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... yes 2024-04-22T00:03:37.748 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... yes 2024-04-22T00:03:37.788 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-22T00:03:37.896 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... gcc3 2024-04-22T00:03:37.942 INFO:tasks.workunit.client.0.smithi038.stdout:checking how to run the C preprocessor... gcc -E 2024-04-22T00:03:37.993 INFO:tasks.workunit.client.0.smithi038.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-22T00:03:37.994 INFO:tasks.workunit.client.0.smithi038.stdout:checking for egrep... /usr/bin/grep -E 2024-04-22T00:03:38.118 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... yes 2024-04-22T00:03:38.152 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/types.h... yes 2024-04-22T00:03:38.191 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/stat.h... yes 2024-04-22T00:03:38.200 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:37 smithi203 ceph-mon[31431]: pgmap v49: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 403 KiB/s wr, 138 op/s 2024-04-22T00:03:38.200 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:37 smithi203 ceph-mon[31431]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T00:03:38.200 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:37 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:38.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:37 smithi038 ceph-mon[27972]: pgmap v49: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 403 KiB/s wr, 138 op/s 2024-04-22T00:03:38.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:37 smithi038 ceph-mon[27972]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T00:03:38.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:37 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:38.230 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdlib.h... yes 2024-04-22T00:03:38.270 INFO:tasks.workunit.client.0.smithi038.stdout:checking for string.h... yes 2024-04-22T00:03:38.312 INFO:tasks.workunit.client.0.smithi038.stdout:checking for memory.h... yes 2024-04-22T00:03:38.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:37 smithi111 ceph-mon[31543]: pgmap v49: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 403 KiB/s wr, 138 op/s 2024-04-22T00:03:38.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:37 smithi111 ceph-mon[31543]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-22T00:03:38.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:37 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:38.353 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strings.h... yes 2024-04-22T00:03:38.393 INFO:tasks.workunit.client.0.smithi038.stdout:checking for inttypes.h... yes 2024-04-22T00:03:38.434 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... yes 2024-04-22T00:03:38.476 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... yes 2024-04-22T00:03:38.521 INFO:tasks.workunit.client.0.smithi038.stdout:checking minix/config.h usability... no 2024-04-22T00:03:38.549 INFO:tasks.workunit.client.0.smithi038.stdout:checking minix/config.h presence... no 2024-04-22T00:03:38.549 INFO:tasks.workunit.client.0.smithi038.stdout:checking for minix/config.h... no 2024-04-22T00:03:38.586 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-22T00:03:38.586 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... (cached) gcc 2024-04-22T00:03:38.622 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-22T00:03:38.622 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-22T00:03:38.622 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-22T00:03:38.622 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-22T00:03:38.664 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lm... yes 2024-04-22T00:03:38.708 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lpthread... yes 2024-04-22T00:03:38.709 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... (cached) yes 2024-04-22T00:03:38.741 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-22T00:03:38.808 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h usability... yes 2024-04-22T00:03:38.827 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h presence... yes 2024-04-22T00:03:38.828 INFO:tasks.workunit.client.0.smithi038.stdout:checking for pthread.h... yes 2024-04-22T00:03:38.874 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h usability... yes 2024-04-22T00:03:38.891 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h presence... yes 2024-04-22T00:03:38.892 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fcntl.h... yes 2024-04-22T00:03:38.939 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h usability... yes 2024-04-22T00:03:38.956 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h presence... yes 2024-04-22T00:03:38.957 INFO:tasks.workunit.client.0.smithi038.stdout:checking for limits.h... yes 2024-04-22T00:03:38.971 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... (cached) yes 2024-04-22T00:03:39.008 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h usability... yes 2024-04-22T00:03:39.026 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h presence... yes 2024-04-22T00:03:39.026 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/time.h... yes 2024-04-22T00:03:39.040 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... (cached) yes 2024-04-22T00:03:39.077 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h usability... yes 2024-04-22T00:03:39.094 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h presence... yes 2024-04-22T00:03:39.094 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/vfs.h... yes 2024-04-22T00:03:39.145 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h usability... no 2024-04-22T00:03:39.174 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h presence... no 2024-04-22T00:03:39.174 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/limits.h... no 2024-04-22T00:03:39.199 INFO:tasks.workunit.client.0.smithi038.stdout:checking for an ANSI C-conforming const... yes 2024-04-22T00:03:39.228 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-22T00:03:39.287 INFO:tasks.workunit.client.0.smithi038.stdout:checking for system... yes 2024-04-22T00:03:39.342 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gettimeofday... yes 2024-04-22T00:03:39.397 INFO:tasks.workunit.client.0.smithi038.stdout:checking for mkdir... yes 2024-04-22T00:03:39.454 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strerror... yes 2024-04-22T00:03:39.510 INFO:tasks.workunit.client.0.smithi038.stdout:checking for open64... yes 2024-04-22T00:03:39.566 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stat64... yes 2024-04-22T00:03:39.621 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fseeko64... yes 2024-04-22T00:03:39.678 INFO:tasks.workunit.client.0.smithi038.stdout:checking for lrand48_r... yes 2024-04-22T00:03:39.733 INFO:tasks.workunit.client.0.smithi038.stdout:checking for srand48_r... yes 2024-04-22T00:03:39.755 INFO:tasks.workunit.client.0.smithi038.stdout:configure: creating ./config.status 2024-04-22T00:03:39.815 INFO:tasks.workunit.client.0.smithi038.stdout: /bin/sh ./config.status 2024-04-22T00:03:39.860 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating Makefile 2024-04-22T00:03:39.873 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-04-22T00:03:39.893 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing depfiles commands 2024-04-22T00:03:39.904 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing default commands 2024-04-22T00:03:39.932 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-22T00:03:40.086 INFO:tasks.workunit.client.0.smithi038.stdout:rm -f stamp-h1 2024-04-22T00:03:40.089 INFO:tasks.workunit.client.0.smithi038.stdout:touch config.h.in 2024-04-22T00:03:40.090 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh ./config.status config.h 2024-04-22T00:03:40.130 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-04-22T00:03:40.159 INFO:tasks.workunit.client.0.smithi038.stdout:make all-am 2024-04-22T00:03:40.164 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-22T00:03:40.164 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-22T00:03:40.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:39 smithi038 ceph-mon[27972]: pgmap v50: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 403 KiB/s wr, 133 op/s 2024-04-22T00:03:40.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:39 smithi203 ceph-mon[31431]: pgmap v50: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 403 KiB/s wr, 133 op/s 2024-04-22T00:03:40.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:39 smithi111 ceph-mon[31543]: pgmap v50: 129 pgs: 129 active+clean; 5.6 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 403 KiB/s wr, 133 op/s 2024-04-22T00:03:40.327 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-22T00:03:40.330 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-22T00:03:40.419 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-22T00:03:40.422 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-22T00:03:40.505 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-22T00:03:40.508 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-22T00:03:40.596 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-22T00:03:40.599 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-22T00:03:40.714 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-22T00:03:40.717 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-22T00:03:40.765 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-22T00:03:40.768 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-22T00:03:40.812 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-22T00:03:40.817 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-22T00:03:40.865 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-22T00:03:40.868 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-22T00:03:41.078 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-22T00:03:41.081 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-22T00:03:41.221 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-22T00:03:41.224 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-22T00:03:41.370 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-22T00:03:41.375 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-22T00:03:41.447 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-22T00:03:41.451 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-22T00:03:41.524 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-22T00:03:41.528 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-22T00:03:41.638 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-22T00:03:41.643 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-22T00:03:41.671 INFO:tasks.workunit.client.0.smithi038.stderr:In file included from parser.h:22, 2024-04-22T00:03:41.671 INFO:tasks.workunit.client.0.smithi038.stderr: from parser.c:25: 2024-04-22T00:03:41.671 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘init_filesys’: 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.672 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.673 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.674 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.674 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.674 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.674 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.674 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.675 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.676 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.677 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.678 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.679 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-22T00:03:41.680 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-22T00:03:41.681 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.724 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘get_optstr’: 2024-04-22T00:03:41.724 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-22T00:03:41.725 INFO:tasks.workunit.client.0.smithi038.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-22T00:03:41.725 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:41.725 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:200:14: note: source object declared here 2024-04-22T00:03:41.725 INFO:tasks.workunit.client.0.smithi038.stderr: 200 | char temp[BUFSIZE]; 2024-04-22T00:03:41.725 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~ 2024-04-22T00:03:41.735 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘size64_convert’: 2024-04-22T00:03:41.735 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-22T00:03:41.735 INFO:tasks.workunit.client.0.smithi038.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-22T00:03:41.736 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-22T00:03:42.042 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-22T00:03:42.045 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-22T00:03:42.102 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-22T00:03:42.106 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-22T00:03:42.195 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-22T00:03:42.198 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-22T00:03:42.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:41 smithi038 ceph-mon[27972]: pgmap v51: 129 pgs: 129 active+clean; 7.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 613 KiB/s wr, 166 op/s 2024-04-22T00:03:42.229 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-22T00:03:42.233 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-22T00:03:42.269 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-22T00:03:42.269 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd .. 2024-04-22T00:03:42.270 INFO:tasks.workunit.client.0.smithi038.stderr:+ mkdir tmp 2024-04-22T00:03:42.271 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd tmp 2024-04-22T00:03:42.271 INFO:tasks.workunit.client.0.smithi038.stderr:+ for f in $mydir/*.ffsb 2024-04-22T00:03:42.271 INFO:tasks.workunit.client.0.smithi038.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-22T00:03:42.272 INFO:tasks.workunit.client.0.smithi038.stdout:FFSB version 6.0-RC2 started 2024-04-22T00:03:42.272 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout:benchmark time = 300 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout:ThreadGroup 0 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout:================ 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: num_threads = 32 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: read_random = off 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: read_size = 0 (0B) 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: read_blocksize = 0 (0B) 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: read_skip = off 2024-04-22T00:03:42.275 INFO:tasks.workunit.client.0.smithi038.stdout: read_skipsize = 0 (0B) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: write_random = on 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: write_size = 5242880 (5MB) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: fsync_file = 0 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: write_blocksize = 4096 (4KB) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: wait time = 0 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: op weights 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: read = 0 (0.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: readall = 0 (0.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: write = 1 (100.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: create = 0 (0.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: append = 0 (0.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: delete = 0 (0.00%) 2024-04-22T00:03:42.276 INFO:tasks.workunit.client.0.smithi038.stdout: metaop = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: createdir = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: stat = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: writeall = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: writeall_fsync = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: open_close = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: write_fsync = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: create_fsync = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: append_fsync = 0 (0.00%) 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout:FileSystem . 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout:========== 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: num_dirs = 0 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: starting files = 128 2024-04-22T00:03:42.277 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: min file size = 104857600 (100MB) 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: max file size = 104857600 (100MB) 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: directio = off 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: alignedio = on 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: bufferedio = off 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: aging is off 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: current utilization = 0.04% 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:03:42.278 INFO:tasks.workunit.client.0.smithi038.stdout:checking existing fs: . 2024-04-22T00:03:42.279 INFO:tasks.workunit.client.0.smithi038.stdout:recreating new fileset 2024-04-22T00:03:42.279 INFO:tasks.workunit.client.0.smithi038.stderr:opendir: No such file or directory 2024-04-22T00:03:42.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:41 smithi203 ceph-mon[31431]: pgmap v51: 129 pgs: 129 active+clean; 7.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 613 KiB/s wr, 166 op/s 2024-04-22T00:03:42.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:41 smithi111 ceph-mon[31543]: pgmap v51: 129 pgs: 129 active+clean; 7.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 613 KiB/s wr, 166 op/s 2024-04-22T00:03:43.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:42 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:43.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:42 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:43.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:42 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:03:44.226 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:43 smithi111 ceph-mon[31543]: pgmap v52: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:44.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:43 smithi203 ceph-mon[31431]: pgmap v52: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:44.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:43 smithi038 ceph-mon[27972]: pgmap v52: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:46.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:45 smithi203 ceph-mon[31431]: pgmap v53: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:46.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:45 smithi111 ceph-mon[31543]: pgmap v53: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:45 smithi038 ceph-mon[27972]: pgmap v53: 129 pgs: 129 active+clean; 7.4 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 609 KiB/s wr, 169 op/s 2024-04-22T00:03:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.937+0000 7f2a051be640 1 -- 172.21.15.38:0/2507337409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 msgr2=0x7f2a00108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.937+0000 7f2a051be640 1 --2- 172.21.15.38:0/2507337409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a00108d90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009860 tx=0x7f29f402bb70 comp rx=0 tx=0).stop 2024-04-22T00:03:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.938+0000 7f2a051be640 1 -- 172.21.15.38:0/2507337409 shutdown_connections 2024-04-22T00:03:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.938+0000 7f2a051be640 1 --2- 172.21.15.38:0/2507337409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a0010d2f0 0x7f2a00114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.938+0000 7f2a051be640 1 --2- 172.21.15.38:0/2507337409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a00109360 0x7f2a0010cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.938+0000 7f2a051be640 1 --2- 172.21.15.38:0/2507337409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a00108d90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.938+0000 7f2a051be640 1 -- 172.21.15.38:0/2507337409 >> 172.21.15.38:0/2507337409 conn(0x7f2a0006cad0 msgr2=0x7f2a0006cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.942+0000 7f2a051be640 1 -- 172.21.15.38:0/2507337409 shutdown_connections 2024-04-22T00:03:46.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.942+0000 7f2a051be640 1 -- 172.21.15.38:0/2507337409 wait complete. 2024-04-22T00:03:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.943+0000 7f2a051be640 1 Processor -- start 2024-04-22T00:03:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.944+0000 7f2a051be640 1 -- start start 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.945+0000 7f2a051be640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.945+0000 7f2a051be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a0010d2f0 0x7f2a0019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.945+0000 7f2a051be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a001a2d50 0x7f2a001a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f2a051be640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00117f50 con 0x7f2a001089b0 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f2a051be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00117dd0 con 0x7f2a001a2d50 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f2a051be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001180d0 con 0x7f2a0010d2f0 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f2a049bd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a001a2d50 0x7f2a001a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35444/0 (socket says 172.21.15.38:35444) 2024-04-22T00:03:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.946+0000 7f29fffff640 1 -- 172.21.15.38:0/3247615557 learned_addr learned my addr 172.21.15.38:0/3247615557 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.947+0000 7f29fffff640 1 -- 172.21.15.38:0/3247615557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a0010d2f0 msgr2=0x7f2a0019e9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.947+0000 7f29fffff640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a0010d2f0 0x7f2a0019e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.947+0000 7f29fffff640 1 -- 172.21.15.38:0/3247615557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a001a2d50 msgr2=0x7f2a001a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.947+0000 7f29fffff640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a001a2d50 0x7f2a001a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.947+0000 7f29fffff640 1 -- 172.21.15.38:0/3247615557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a001a3840 con 0x7f2a001089b0 2024-04-22T00:03:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.948+0000 7f29fffff640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f29f4037040 tx=0x7f29f40323b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.950+0000 7f29fd7fa640 1 -- 172.21.15.38:0/3247615557 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f4041020 con 0x7f2a001089b0 2024-04-22T00:03:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.950+0000 7f29fd7fa640 1 -- 172.21.15.38:0/3247615557 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f402c9f0 con 0x7f2a001089b0 2024-04-22T00:03:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.950+0000 7f29fd7fa640 1 -- 172.21.15.38:0/3247615557 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f4039040 con 0x7f2a001089b0 2024-04-22T00:03:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.951+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a001a3ad0 con 0x7f2a001089b0 2024-04-22T00:03:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.951+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a001ab240 con 0x7f2a001089b0 2024-04-22T00:03:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.967+0000 7f29fd7fa640 1 -- 172.21.15.38:0/3247615557 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f29f4044020 con 0x7f2a001089b0 2024-04-22T00:03:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.967+0000 7f2a0900a740 1 Processor -- start 2024-04-22T00:03:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.968+0000 7f29fd7fa640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 0x7f29e80a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.968+0000 7f29fd7fa640 1 -- 172.21.15.38:0/3247615557 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f29f40f13f0 con 0x7f2a001089b0 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 -- start start 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678f3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678f3660 0x55d4678fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a049bd640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 0x7f29e80a7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d467856b70 con 0x55d4678fc430 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d4677ff470 con 0x55d4678f3660 2024-04-22T00:03:46.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.971+0000 7f2a0900a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d467852b30 con 0x55d4678f2d40 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.972+0000 7f2a049bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.972+0000 7f2a049bd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35454/0 (socket says 172.21.15.38:35454) 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.972+0000 7f2a049bd640 1 -- 172.21.15.38:0/729473658 learned_addr learned my addr 172.21.15.38:0/729473658 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f29fffff640 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f2a049bd640 1 -- 172.21.15.38:0/729473658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 msgr2=0x55d4678f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f2a049bd640 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f2a049bd640 1 -- 172.21.15.38:0/729473658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678f3660 msgr2=0x55d4678fbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f2a049bd640 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678f3660 0x55d4678fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.973+0000 7f2a049bd640 1 -- 172.21.15.38:0/729473658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d46756b210 con 0x55d4678fc430 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.974+0000 7f2a049bd640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 0x7f29e80a7590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e7a0 tx=0x7f29f800eba0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.974+0000 7f2a049bd640 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f29f800eea0 tx=0x7f29f801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7390aade6f78d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:46.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.976+0000 7f29e67fc640 1 -- 172.21.15.38:0/729473658 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f801ead0 con 0x55d4678fc430 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.976+0000 7f29e67fc640 1 -- 172.21.15.38:0/729473658 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f8003e50 con 0x55d4678fc430 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 -- 172.21.15.38:0/729473658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 msgr2=0x55d4678fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f29f800eea0 tx=0x7f29f801b000 comp rx=0 tx=0).stop 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 -- 172.21.15.38:0/729473658 shutdown_connections 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678fc430 0x55d4678fe820 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678f3660 0x55d4678fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 --2- 172.21.15.38:0/729473658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.978+0000 7f2a0900a740 1 -- 172.21.15.38:0/729473658 >> 172.21.15.38:0/729473658 conn(0x55d4678ed660 msgr2=0x55d4678eda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.983+0000 7f2a0900a740 1 -- 172.21.15.38:0/729473658 shutdown_connections 2024-04-22T00:03:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.984+0000 7f2a0900a740 1 -- 172.21.15.38:0/729473658 wait complete. 2024-04-22T00:03:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.985+0000 7f2a0900a740 1 -- start start 2024-04-22T00:03:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.985+0000 7f2a0900a740 1 Processor -- start 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d4678fefa0 con 0x55d4678f3660 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d467812a10 con 0x55d4678fc430 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d4678ff120 con 0x55d4678f2d40 2024-04-22T00:03:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.986+0000 7f2a0900a740 10 client.0 _ll_register_callbacks cb 0x55d46784db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:46.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.988+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.988+0000 7f29ff7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35466/0 (socket says 172.21.15.38:35466) 2024-04-22T00:03:46.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.988+0000 7f29ff7fe640 1 -- 172.21.15.38:0/1733888969 learned_addr learned my addr 172.21.15.38:0/1733888969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:46.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.988+0000 7f2a049bd640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.989+0000 7f29ff7fe640 1 -- 172.21.15.38:0/1733888969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 msgr2=0x55d4678fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.989+0000 7f29ff7fe640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.989+0000 7f29ff7fe640 1 -- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 msgr2=0x55d4678f99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.990+0000 7f29ff7fe640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.990+0000 7f29ff7fe640 1 -- 172.21.15.38:0/1733888969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d467637620 con 0x55d4678f3660 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.990+0000 7f2a049bd640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.991+0000 7f29ff7fe640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f29f00058c0 tx=0x7f29f00076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32f1055c0fa3462 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.991+0000 7f29df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.991+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0005230 con 0x55d4678f3660 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.991+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f29f00053d0 con 0x55d4678f3660 2024-04-22T00:03:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.991+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f8003e50 con 0x55d4678f3660 2024-04-22T00:03:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.992+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f29f0007e40 con 0x55d4678f3660 2024-04-22T00:03:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.992+0000 7f2a0900a740 10 client.16350 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.992+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d467905360 con 0x55d4678f3660 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.993+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f8003e50 con 0x55d4678f3660 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.993+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f29f001b030 con 0x55d4678f3660 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 10 client.16350 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 20 client.16350 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467906910 0x55d467908d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 4 client.16350 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4678640b0 con 0x55d467906910 2024-04-22T00:03:46.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.995+0000 7f29fffff640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467906910 0x55d467908d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:46.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.997+0000 7f29df7fe640 10 client.16350 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:46.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.997+0000 7f29fffff640 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467906910 0x55d467908d50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf58418579168f9 server_cookie=426af0e36c2f1557 in_seq=0 out_seq=0 2024-04-22T00:03:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f29df7fe640 1 -- 172.21.15.38:0/1733888969 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4678640b0 con 0x55d467906910 2024-04-22T00:03:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f29df7fe640 10 client.16350 handle_command_reply: tid=0 2024-04-22T00:03:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 client.16350 shutdown 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 msgr2=0x55d4678f9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f29f00058c0 tx=0x7f29f00076d0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 shutdown_connections 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467906910 0x55d467908d50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f3660 0x55d4678f9460 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 --2- 172.21.15.38:0/1733888969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f2d40 0x55d4678fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:46.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f29df7fe640 0 client.16350 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:46.999+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 >> 172.21.15.38:0/1733888969 conn(0x55d4678ed660 msgr2=0x55d4678ef3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.000+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 shutdown_connections 2024-04-22T00:03:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.000+0000 7f2a0900a740 1 -- 172.21.15.38:0/1733888969 wait complete. 2024-04-22T00:03:46.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.000+0000 7f2a0900a740 20 client.16350 trim_cache size 0 max 16384 2024-04-22T00:03:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.026+0000 7f2a0900a740 1 Processor -- start 2024-04-22T00:03:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.026+0000 7f2a0900a740 1 -- start start 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.026+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.026+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4679280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f2a0900a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b4f40 con 0x55d4678f2d40 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f2a0900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b6c00 con 0x55d4678fc430 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f2a0900a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b5610 con 0x55d4678f3660 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35474/0 (socket says 172.21.15.38:35474) 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 -- 172.21.15.38:0/3302887938 learned_addr learned my addr 172.21.15.38:0/3302887938 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f2a049bd640 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4679280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 -- 172.21.15.38:0/3302887938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 msgr2=0x55d4678ef1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 -- 172.21.15.38:0/3302887938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 msgr2=0x55d4679280c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.027+0000 7f29fffff640 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4679280c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.028+0000 7f29fffff640 1 -- 172.21.15.38:0/3302887938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4678f9d00 con 0x55d4678f2d40 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.028+0000 7f29fffff640 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29f40bab40 tx=0x7f29f4048040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d5b67f8e2dca94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.028+0000 7f29e5ffb640 1 -- 172.21.15.38:0/3302887938 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f40c6de0 con 0x55d4678f2d40 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.028+0000 7f29e5ffb640 1 -- 172.21.15.38:0/3302887938 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f4032e40 con 0x55d4678f2d40 2024-04-22T00:03:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.028+0000 7f29e5ffb640 1 -- 172.21.15.38:0/3302887938 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f40ba7e0 con 0x55d4678f2d40 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 -- 172.21.15.38:0/3302887938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 msgr2=0x55d4678fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29f40bab40 tx=0x7f29f4048040 comp rx=0 tx=0).stop 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 -- 172.21.15.38:0/3302887938 shutdown_connections 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4679280c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678ef1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3302887938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d4678fbb90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 -- 172.21.15.38:0/3302887938 >> 172.21.15.38:0/3302887938 conn(0x55d4678ed660 msgr2=0x55d4678eeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 -- 172.21.15.38:0/3302887938 shutdown_connections 2024-04-22T00:03:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.030+0000 7f2a0900a740 1 -- 172.21.15.38:0/3302887938 wait complete. 2024-04-22T00:03:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.031+0000 7f2a0900a740 1 -- start start 2024-04-22T00:03:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.031+0000 7f2a0900a740 1 Processor -- start 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.031+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.031+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a0900a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a0900a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b6470 con 0x55d4678f2d40 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a0900a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b5da0 con 0x55d4678fc430 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a0900a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b2f20 con 0x55d4678f3660 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a0900a740 10 client.0 _ll_register_callbacks cb 0x7f29f0005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35488/0 (socket says 172.21.15.38:35488) 2024-04-22T00:03:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 -- 172.21.15.38:0/3267903935 learned_addr learned my addr 172.21.15.38:0/3267903935 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f2a049bd640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 -- 172.21.15.38:0/3267903935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 msgr2=0x55d4678f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 -- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 msgr2=0x55d4678f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.032+0000 7f29fffff640 1 -- 172.21.15.38:0/3267903935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4678f6a20 con 0x55d4678f2d40 2024-04-22T00:03:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.033+0000 7f29fffff640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f29f40be100 tx=0x7f29f402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcad9ae3e07440a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.033+0000 7f29df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f40d6070 con 0x55d4678f2d40 2024-04-22T00:03:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f29f402cba0 con 0x55d4678f2d40 2024-04-22T00:03:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f4032e40 con 0x55d4678f2d40 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f29f40ba7e0 con 0x55d4678f2d40 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f2a0900a740 10 client.16356 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.034+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d467984870 con 0x55d4678f2d40 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f40c6de0 con 0x55d4678f2d40 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f29f40b8ce0 con 0x55d4678f2d40 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 10 client.16356 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 20 client.16356 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467984ed0 0x55d467987310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 4 client.16356 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:03:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f29f4032e40 con 0x55d467984ed0 2024-04-22T00:03:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.035+0000 7f29ff7fe640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467984ed0 0x55d467987310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.036+0000 7f29ff7fe640 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467984ed0 0x55d467987310 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e984d78e7ac0312 server_cookie=8b42054309c9edc in_seq=0 out_seq=0 2024-04-22T00:03:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.036+0000 7f29df7fe640 10 client.16356 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.242+0000 7f29df7fe640 1 -- 172.21.15.38:0/3267903935 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f29f4032e40 con 0x55d467984ed0 2024-04-22T00:03:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.242+0000 7f29df7fe640 10 client.16356 handle_command_reply: tid=0 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.242+0000 7f2a0900a740 1 client.16356 shutdown 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.242+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 msgr2=0x55d46790e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.242+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f29f40be100 tx=0x7f29f402bb70 comp rx=0 tx=0).stop 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 shutdown_connections 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4678fc430 0x55d4678f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d4678f3660 0x55d4678f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d467984ed0 0x55d467987310 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 --2- 172.21.15.38:0/3267903935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d4678f2d40 0x55d46790e8f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f29df7fe640 0 client.16356 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 >> 172.21.15.38:0/3267903935 conn(0x55d4678ed660 msgr2=0x55d4678f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 shutdown_connections 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 1 -- 172.21.15.38:0/3267903935 wait complete. 2024-04-22T00:03:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.243+0000 7f2a0900a740 20 client.16356 trim_cache size 0 max 16384 2024-04-22T00:03:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:03:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:03:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:03:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:03:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 msgr2=0x7f29e80a7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 0x7f29e80a7590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e7a0 tx=0x7f29f800eba0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 msgr2=0x7f2a0019e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f29f4037040 tx=0x7f29f40323b0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 shutdown_connections 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f29e80a50d0 0x7f29e80a7590 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a001a2d50 0x7f2a001a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a0010d2f0 0x7f2a0019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 --2- 172.21.15.38:0/3247615557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a001089b0 0x7f2a0019e490 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 >> 172.21.15.38:0/3247615557 conn(0x7f2a0006cad0 msgr2=0x7f2a00071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 shutdown_connections 2024-04-22T00:03:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:03:47.244+0000 7f2a051be640 1 -- 172.21.15.38:0/3247615557 wait complete. 2024-04-22T00:03:48.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:47 smithi203 ceph-mon[31431]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 126 MiB/s wr, 272 op/s 2024-04-22T00:03:48.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:47 smithi111 ceph-mon[31543]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 126 MiB/s wr, 272 op/s 2024-04-22T00:03:48.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:47 smithi038 ceph-mon[27972]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 126 MiB/s wr, 272 op/s 2024-04-22T00:03:50.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:50 smithi203 ceph-mon[31431]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 125 MiB/s wr, 143 op/s 2024-04-22T00:03:50.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:50 smithi111 ceph-mon[31543]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 125 MiB/s wr, 143 op/s 2024-04-22T00:03:50.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:50 smithi038 ceph-mon[27972]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 125 MiB/s wr, 143 op/s 2024-04-22T00:03:51.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:03:51.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 -- 172.21.15.38:0/1348764679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc100620 msgr2=0x7f16bc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 --2- 172.21.15.38:0/1348764679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc100620 0x7f16bc100aa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009860 tx=0x7f16b002bb70 comp rx=0 tx=0).stop 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 -- 172.21.15.38:0/1348764679 shutdown_connections 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 --2- 172.21.15.38:0/1348764679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16bc100fe0 0x7f16bc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 --2- 172.21.15.38:0/1348764679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc100620 0x7f16bc100aa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 --2- 172.21.15.38:0/1348764679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc106630 0x7f16bc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 -- 172.21.15.38:0/1348764679 >> 172.21.15.38:0/1348764679 conn(0x7f16bc0769c0 msgr2=0x7f16bc076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 -- 172.21.15.38:0/1348764679 shutdown_connections 2024-04-22T00:03:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.304+0000 7f16c11e5640 1 -- 172.21.15.38:0/1348764679 wait complete. 2024-04-22T00:03:51.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.305+0000 7f16c11e5640 1 Processor -- start 2024-04-22T00:03:51.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 -- start start 2024-04-22T00:03:51.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16bc100620 0x7f16bc198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc106630 0x7f16bc1a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc110a40 con 0x7f16bc100fe0 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1108c0 con 0x7f16bc106630 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16c11e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc110bc0 con 0x7f16bc100620 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35502/0 (socket says 172.21.15.38:35502) 2024-04-22T00:03:51.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.306+0000 7f16ba59c640 1 -- 172.21.15.38:0/3332547411 learned_addr learned my addr 172.21.15.38:0/3332547411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 -- 172.21.15.38:0/3332547411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16bc100620 msgr2=0x7f16bc198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16bc100620 0x7f16bc198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 -- 172.21.15.38:0/3332547411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc106630 msgr2=0x7f16bc1a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc106630 0x7f16bc1a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 -- 172.21.15.38:0/3332547411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc1a44a0 con 0x7f16bc100fe0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f16b0037040 tx=0x7f16b0032480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16a3fff640 1 -- 172.21.15.38:0/3332547411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0041020 con 0x7f16bc100fe0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.307+0000 7f16a3fff640 1 -- 172.21.15.38:0/3332547411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b002c9f0 con 0x7f16bc100fe0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.308+0000 7f16a3fff640 1 -- 172.21.15.38:0/3332547411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0039040 con 0x7f16bc100fe0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.308+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc1a4670 con 0x7f16bc100fe0 2024-04-22T00:03:51.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.308+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16bc1a4a20 con 0x7f16bc100fe0 2024-04-22T00:03:51.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.311+0000 7f16a3fff640 1 -- 172.21.15.38:0/3332547411 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f16b0044020 con 0x7f16bc100fe0 2024-04-22T00:03:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.312+0000 7f16a3fff640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 0x7f16a40a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.312+0000 7f16a3fff640 1 -- 172.21.15.38:0/3332547411 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f16b00f1150 con 0x7f16bc100fe0 2024-04-22T00:03:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.312+0000 7f16bad9d640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 0x7f16a40a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.312+0000 7f16c3dd2740 1 Processor -- start 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 -- start start 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16bad9d640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 0x7f16a40a52c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005fd0 tx=0x7f16ac005a00 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d788ff30 0x5606d7890310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7899430 0x5606d789b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5606d77f4560 con 0x5606d7890850 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5606d779c730 con 0x5606d7899430 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606d77f0520 con 0x5606d788ff30 2024-04-22T00:03:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35518/0 (socket says 172.21.15.38:35518) 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 -- 172.21.15.38:0/3007054380 learned_addr learned my addr 172.21.15.38:0/3007054380 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 -- 172.21.15.38:0/3007054380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d788ff30 msgr2=0x5606d7890310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d788ff30 0x5606d7890310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 -- 172.21.15.38:0/3007054380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7899430 msgr2=0x5606d789b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7899430 0x5606d789b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.313+0000 7f16ba59c640 1 -- 172.21.15.38:0/3007054380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606d75d27d0 con 0x5606d7890850 2024-04-22T00:03:51.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.314+0000 7f16ba59c640 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f16b00c5b20 tx=0x7f16b002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22c6f18fdb5504d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.314+0000 7f16a17fa640 1 -- 172.21.15.38:0/3007054380 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b00bd3e0 con 0x5606d7890850 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.314+0000 7f16a17fa640 1 -- 172.21.15.38:0/3007054380 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b00c1790 con 0x5606d7890850 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.314+0000 7f16a17fa640 1 -- 172.21.15.38:0/3007054380 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b004d300 con 0x5606d7890850 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 -- 172.21.15.38:0/3007054380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 msgr2=0x5606d7898ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f16b00c5b20 tx=0x7f16b002bb70 comp rx=0 tx=0).stop 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 -- 172.21.15.38:0/3007054380 shutdown_connections 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7899430 0x5606d789b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d7898ef0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/3007054380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d788ff30 0x5606d7890310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 -- 172.21.15.38:0/3007054380 >> 172.21.15.38:0/3007054380 conn(0x5606d788a660 msgr2=0x5606d788aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 -- 172.21.15.38:0/3007054380 shutdown_connections 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.315+0000 7f16c3dd2740 1 -- 172.21.15.38:0/3007054380 wait complete. 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 -- start start 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 Processor -- start 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d7896580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7890850 0x5606d7896ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5606d789bfe0 con 0x5606d7899430 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5606d77afe00 con 0x5606d788ff30 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606d789c160 con 0x5606d7890850 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16c3dd2740 10 client.0 _ll_register_callbacks cb 0x5606d77eaeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16bb59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16bb59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35526/0 (socket says 172.21.15.38:35526) 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.316+0000 7f16bb59e640 1 -- 172.21.15.38:0/2414316973 learned_addr learned my addr 172.21.15.38:0/2414316973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 -- 172.21.15.38:0/2414316973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7890850 msgr2=0x5606d7896ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7890850 0x5606d7896ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 -- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 msgr2=0x5606d7896580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d7896580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 -- 172.21.15.38:0/2414316973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606d75083f0 con 0x5606d7899430 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16bb59e640 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f16b400b390 tx=0x7f16b400b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb00c17ef13fbf50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f169eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b40115a0 con 0x5606d7899430 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f16b4011740 con 0x5606d7899430 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b00c1790 con 0x5606d7899430 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.317+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16b4020050 con 0x5606d7899430 2024-04-22T00:03:51.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.318+0000 7f16c3dd2740 10 client.16368 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:51.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.318+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5606d74c80f0 con 0x5606d7899430 2024-04-22T00:03:51.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.318+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b00c1790 con 0x5606d7899430 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.318+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f16b4011070 con 0x5606d7899430 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 10 client.16368 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 20 client.16368 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d78a9700 0x5606d78abb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 4 client.16368 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.319+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606d7801aa0 con 0x5606d78a9700 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.320+0000 7f16bad9d640 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d78a9700 0x5606d78abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.322+0000 7f16bad9d640 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d78a9700 0x5606d78abb90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c41fa916e4d4e62 server_cookie=79634323620e5785 in_seq=0 out_seq=0 2024-04-22T00:03:51.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.322+0000 7f169eff5640 10 client.16368 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f169eff5640 1 -- 172.21.15.38:0/2414316973 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606d7801aa0 con 0x5606d78a9700 2024-04-22T00:03:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f169eff5640 10 client.16368 handle_command_reply: tid=0 2024-04-22T00:03:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f16c3dd2740 1 client.16368 shutdown 2024-04-22T00:03:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 msgr2=0x5606d7897000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f16b400b390 tx=0x7f16b400b850 comp rx=0 tx=0).stop 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.323+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 shutdown_connections 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d78a9700 0x5606d78abb90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7899430 0x5606d7897000 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7890850 0x5606d7896ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f169eff5640 0 client.16368 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2414316973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d7896580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 >> 172.21.15.38:0/2414316973 conn(0x5606d788a660 msgr2=0x5606d788c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 shutdown_connections 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2414316973 wait complete. 2024-04-22T00:03:51.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.324+0000 7f16c3dd2740 20 client.16368 trim_cache size 0 max 16384 2024-04-22T00:03:51.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 Processor -- start 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 -- start start 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7890850 0x5606d7895720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7889560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1aeaa0 con 0x5606d788ff30 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1b0760 con 0x5606d7890850 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.358+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1af170 con 0x5606d7899430 2024-04-22T00:03:51.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35540/0 (socket says 172.21.15.38:35540) 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 -- 172.21.15.38:0/2094580565 learned_addr learned my addr 172.21.15.38:0/2094580565 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 -- 172.21.15.38:0/2094580565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 msgr2=0x5606d7889560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7889560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 -- 172.21.15.38:0/2094580565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7890850 msgr2=0x5606d7895720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7890850 0x5606d7895720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 -- 172.21.15.38:0/2094580565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606d789bb90 con 0x5606d788ff30 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16bad9d640 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f16ac012040 tx=0x7f16ac005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a37360734c1b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.359+0000 7f16a0ff9640 1 -- 172.21.15.38:0/2094580565 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0231e0 con 0x5606d788ff30 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.360+0000 7f16a0ff9640 1 -- 172.21.15.38:0/2094580565 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ac027030 con 0x5606d788ff30 2024-04-22T00:03:51.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.360+0000 7f16a0ff9640 1 -- 172.21.15.38:0/2094580565 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac01fc70 con 0x5606d788ff30 2024-04-22T00:03:51.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.360+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2094580565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 msgr2=0x5606d7898cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.360+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f16ac012040 tx=0x7f16ac005d50 comp rx=0 tx=0).stop 2024-04-22T00:03:51.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2094580565 shutdown_connections 2024-04-22T00:03:51.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7889560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d7890850 0x5606d7895720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/2094580565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d788ff30 0x5606d7898cb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2094580565 >> 172.21.15.38:0/2094580565 conn(0x5606d788a660 msgr2=0x5606d7888e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2094580565 shutdown_connections 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 -- 172.21.15.38:0/2094580565 wait complete. 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 -- start start 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.361+0000 7f16c3dd2740 1 Processor -- start 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d78931a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7893c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1affd0 con 0x5606d7890850 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1af900 con 0x5606d788ff30 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1aca80 con 0x5606d7899430 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16c3dd2740 10 client.0 _ll_register_callbacks cb 0x7f16b40094e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:51.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16bad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d78931a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16bad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d78931a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:34836/0 (socket says 172.21.15.38:34836) 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35554/0 (socket says 172.21.15.38:35554) 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16bad9d640 1 -- 172.21.15.38:0/887906695 learned_addr learned my addr 172.21.15.38:0/887906695 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 -- 172.21.15.38:0/887906695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 msgr2=0x5606d7893c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7893c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 -- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 msgr2=0x5606d78931a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d78931a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.362+0000 7f16ba59c640 1 -- 172.21.15.38:0/887906695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606d7913120 con 0x5606d7890850 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.363+0000 7f16ba59c640 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x5606d788fa00 tx=0x7f16b00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0f550960f5e169 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.363+0000 7f169eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.363+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b00da020 con 0x5606d7890850 2024-04-22T00:03:51.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.363+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f16b00d6230 con 0x5606d7890850 2024-04-22T00:03:51.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.364+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ac027030 con 0x5606d7890850 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.364+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16b00bd3e0 con 0x5606d7890850 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.364+0000 7f16c3dd2740 10 client.16374 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.364+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5606d7914330 con 0x5606d7890850 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.365+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ac027030 con 0x5606d7890850 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f16b003c030 con 0x5606d7890850 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 10 client.16374 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 20 client.16374 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d7914940 0x5606d7916d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 4 client.16374 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:03:51.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606d77fbdf0 con 0x5606d7914940 2024-04-22T00:03:51.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.366+0000 7f16bad9d640 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d7914940 0x5606d7916d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.368+0000 7f169eff5640 10 client.16374 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:51.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.368+0000 7f16bad9d640 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d7914940 0x5606d7916d60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73dfa2736ec7cefd server_cookie=72241564f5629b85 in_seq=0 out_seq=0 2024-04-22T00:03:51.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.369+0000 7f169eff5640 1 -- 172.21.15.38:0/887906695 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5606d77fbdf0 con 0x5606d7914940 2024-04-22T00:03:51.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.369+0000 7f169eff5640 10 client.16374 handle_command_reply: tid=0 2024-04-22T00:03:51.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.369+0000 7f16c3dd2740 1 client.16374 shutdown 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 msgr2=0x5606d78936e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x5606d788fa00 tx=0x7f16b00057c0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 shutdown_connections 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606d7899430 0x5606d7893c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5606d7914940 0x5606d7916d60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606d7890850 0x5606d78936e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f169eff5640 0 client.16374 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.371+0000 7f16c3dd2740 1 --2- 172.21.15.38:0/887906695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5606d788ff30 0x5606d78931a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.372+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 >> 172.21.15.38:0/887906695 conn(0x5606d788a660 msgr2=0x5606d78896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.372+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 shutdown_connections 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.372+0000 7f16c3dd2740 1 -- 172.21.15.38:0/887906695 wait complete. 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.372+0000 7f16c3dd2740 20 client.16374 trim_cache size 0 max 16384 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:51.365 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:03:51.366 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:03:51.366 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 msgr2=0x7f16a40a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 0x7f16a40a52c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005fd0 tx=0x7f16ac005a00 comp rx=0 tx=0).stop 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 msgr2=0x7f16bc199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f16b0037040 tx=0x7f16b0032480 comp rx=0 tx=0).stop 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 shutdown_connections 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.373+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f16a40a2e00 0x7f16a40a52c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16bc106630 0x7f16bc1a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16bc100fe0 0x7f16bc199420 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 --2- 172.21.15.38:0/3332547411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16bc100620 0x7f16bc198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 >> 172.21.15.38:0/3332547411 conn(0x7f16bc0769c0 msgr2=0x7f16bc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 shutdown_connections 2024-04-22T00:03:51.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.374+0000 7f16c11e5640 1 -- 172.21.15.38:0/3332547411 wait complete. 2024-04-22T00:03:51.404 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:03:51.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 -- 172.21.15.38:0/3893956154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 msgr2=0x7effc4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 --2- 172.21.15.38:0/3893956154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc4106a10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7effc0009930 tx=0x7effc002bc30 comp rx=0 tx=0).stop 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 -- 172.21.15.38:0/3893956154 shutdown_connections 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 --2- 172.21.15.38:0/3893956154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100fe0 0x7effc410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 --2- 172.21.15.38:0/3893956154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100620 0x7effc4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 --2- 172.21.15.38:0/3893956154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc4106a10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 -- 172.21.15.38:0/3893956154 >> 172.21.15.38:0/3893956154 conn(0x7effc4077740 msgr2=0x7effc4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 -- 172.21.15.38:0/3893956154 shutdown_connections 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.562+0000 7effcc0e2640 1 -- 172.21.15.38:0/3893956154 wait complete. 2024-04-22T00:03:51.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 Processor -- start 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 -- start start 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 0x7effc4102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100fe0 0x7effc4103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7effc4110a00 con 0x7effc4106630 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7effc4110880 con 0x7effc4100620 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effcc0e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7effc4110b80 con 0x7effc4100fe0 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effca67f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effca67f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35568/0 (socket says 172.21.15.38:35568) 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effca67f640 1 -- 172.21.15.38:0/2816768080 learned_addr learned my addr 172.21.15.38:0/2816768080 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effc9e7e640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 0x7effc4102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.563+0000 7effc967d640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100fe0 0x7effc4103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 -- 172.21.15.38:0/2816768080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100fe0 msgr2=0x7effc4103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100fe0 0x7effc4103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 -- 172.21.15.38:0/2816768080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 msgr2=0x7effc4102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 0x7effc4102c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 -- 172.21.15.38:0/2816768080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc41a6530 con 0x7effc4106630 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effc9e7e640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 0x7effc4102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effca67f640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7effbc00eb10 tx=0x7effbc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc00ce20 con 0x7effc4106630 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effbc005020 con 0x7effc4106630 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc41a6820 con 0x7effc4106630 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc010690 con 0x7effc4106630 2024-04-22T00:03:51.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.564+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc41a6d70 con 0x7effc4106630 2024-04-22T00:03:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.566+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc4068560 con 0x7effc4106630 2024-04-22T00:03:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.567+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7effbc005540 con 0x7effc4106630 2024-04-22T00:03:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.568+0000 7effbaffd640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 0x7effb00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.568+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7effbc0cfc30 con 0x7effc4106630 2024-04-22T00:03:51.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.568+0000 7effc9e7e640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 0x7effb00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:51.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.568+0000 7effc9e7e640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 0x7effb00a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7effc0005ab0 tx=0x7effc0005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:51.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.573+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7effbc09c510 con 0x7effc4106630 2024-04-22T00:03:51.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.921+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effc4109ba0 con 0x7effc4106630 2024-04-22T00:03:51.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.922+0000 7effbaffd640 1 -- 172.21.15.38:0/2816768080 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11949 (secure 0 0 0) 0x7effbc09be10 con 0x7effc4106630 2024-04-22T00:03:51.916 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:03:51.917 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:03:41.555018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:03:51.918 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 32 2024-04-22T00:03:51.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 msgr2=0x7effb00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 0x7effb00a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7effc0005ab0 tx=0x7effc0005fd0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 msgr2=0x7effc41036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:51.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7effbc00eb10 tx=0x7effbc00ef10 comp rx=0 tx=0).stop 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 shutdown_connections 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7effb00a2d80 0x7effb00a5240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.927+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effc4106630 0x7effc41036c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.928+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc4100fe0 0x7effc4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.928+0000 7effcc0e2640 1 --2- 172.21.15.38:0/2816768080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effc4100620 0x7effc4102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.928+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 >> 172.21.15.38:0/2816768080 conn(0x7effc4077740 msgr2=0x7effc4068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.928+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 shutdown_connections 2024-04-22T00:03:51.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:51.928+0000 7effcc0e2640 1 -- 172.21.15.38:0/2816768080 wait complete. 2024-04-22T00:03:51.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:03:52.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.170+0000 7f2ae4566640 1 -- 172.21.15.38:0/178535981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 msgr2=0x7f2adc107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.170+0000 7f2ae1300640 1 -- 172.21.15.38:0/178535981 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc032990 con 0x7f2adc107710 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.170+0000 7f2ae4566640 1 --2- 172.21.15.38:0/178535981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc107af0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009cd0 tx=0x7f2acc02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 -- 172.21.15.38:0/178535981 shutdown_connections 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 --2- 172.21.15.38:0/178535981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc1005f0 0x7f2adc10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 --2- 172.21.15.38:0/178535981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc1000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 --2- 172.21.15.38:0/178535981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc107af0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 -- 172.21.15.38:0/178535981 >> 172.21.15.38:0/178535981 conn(0x7f2adc076be0 msgr2=0x7f2adc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 -- 172.21.15.38:0/178535981 shutdown_connections 2024-04-22T00:03:52.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.171+0000 7f2ae4566640 1 -- 172.21.15.38:0/178535981 wait complete. 2024-04-22T00:03:52.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.172+0000 7f2ae4566640 1 Processor -- start 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.172+0000 7f2ae4566640 1 -- start start 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc1005f0 0x7f2adc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc111b60 con 0x7f2adc107710 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1119e0 con 0x7f2adc1005f0 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae4566640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc111ce0 con 0x7f2adc0ffc50 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae2302640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae2302640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:35510/0 (socket says 172.21.15.38:35510) 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae2302640 1 -- 172.21.15.38:0/1484568357 learned_addr learned my addr 172.21.15.38:0/1484568357 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.173+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 -- 172.21.15.38:0/1484568357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc1005f0 msgr2=0x7f2adc198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc1005f0 0x7f2adc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 -- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 msgr2=0x7f2adc19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 -- 172.21.15.38:0/1484568357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc19cfd0 con 0x7f2adc0ffc50 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.174+0000 7f2ae2302640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005e30 tx=0x7f2acc032d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.175+0000 7f2adb7fe640 1 -- 172.21.15.38:0/1484568357 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc041020 con 0x7f2adc0ffc50 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.175+0000 7f2adb7fe640 1 -- 172.21.15.38:0/1484568357 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2acc034cd0 con 0x7f2adc0ffc50 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.175+0000 7f2adb7fe640 1 -- 172.21.15.38:0/1484568357 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc039040 con 0x7f2adc0ffc50 2024-04-22T00:03:52.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.175+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc19d260 con 0x7f2adc0ffc50 2024-04-22T00:03:52.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.175+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc19d640 con 0x7f2adc0ffc50 2024-04-22T00:03:52.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.180+0000 7f2ae7220740 1 Processor -- start 2024-04-22T00:03:52.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.180+0000 7f2adb7fe640 1 -- 172.21.15.38:0/1484568357 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2acc044020 con 0x7f2adc0ffc50 2024-04-22T00:03:52.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 -- start start 2024-04-22T00:03:52.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db25aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2adb7fe640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 0x7f2ac40a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db12c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d14da7c530 con 0x55d14db12820 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d14da24830 con 0x55d14db26070 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.181+0000 7f2ae7220740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d14da784f0 con 0x55d14db256c0 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:34892/0 (socket says 172.21.15.38:34892) 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 -- 172.21.15.38:0/1715035592 learned_addr learned my addr 172.21.15.38:0/1715035592 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2adb7fe640 1 -- 172.21.15.38:0/1484568357 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2acc0f12b0 con 0x7f2adc0ffc50 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae2302640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 0x7f2ac40a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db12c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db25aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 -- 172.21.15.38:0/1715035592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 msgr2=0x55d14db25aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db25aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 -- 172.21.15.38:0/1715035592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 msgr2=0x55d14db12c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db12c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 -- 172.21.15.38:0/1715035592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14da482c0 con 0x55d14db26070 2024-04-22T00:03:52.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.182+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000bae0 tx=0x7f2ad000d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10dedbc3eae1259d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ae2302640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 0x7f2ac40a5340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0ee830 tx=0x7f2acc02e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ad8bfe640 1 -- 172.21.15.38:0/1715035592 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0016070 con 0x55d14db26070 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ad8bfe640 1 -- 172.21.15.38:0/1715035592 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad0005500 con 0x55d14db26070 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ad8bfe640 1 -- 172.21.15.38:0/1715035592 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0010460 con 0x55d14db26070 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ae7220740 1 -- 172.21.15.38:0/1715035592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 msgr2=0x55d14db12200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.183+0000 7f2ae7220740 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000bae0 tx=0x7f2ad000d2e0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.184+0000 7f2ae7220740 1 -- 172.21.15.38:0/1715035592 shutdown_connections 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.184+0000 7f2ae7220740 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db12c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.184+0000 7f2ae7220740 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db12200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.184+0000 7f2ae7220740 1 --2- 172.21.15.38:0/1715035592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db25aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.184+0000 7f2ae7220740 1 -- 172.21.15.38:0/1715035592 >> 172.21.15.38:0/1715035592 conn(0x55d14db14cc0 msgr2=0x55d14db150a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.185+0000 7f2ae7220740 1 -- 172.21.15.38:0/1715035592 shutdown_connections 2024-04-22T00:03:52.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.185+0000 7f2ae7220740 1 -- 172.21.15.38:0/1715035592 wait complete. 2024-04-22T00:03:52.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.186+0000 7f2ae7220740 1 -- start start 2024-04-22T00:03:52.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.186+0000 7f2ae7220740 1 Processor -- start 2024-04-22T00:03:52.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.187+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.187+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 0x55d14db29420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.188+0000 7f2ae2b03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.189+0000 7f2ae2b03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35592/0 (socket says 172.21.15.38:35592) 2024-04-22T00:03:52.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.189+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 0x55d14db29420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.190+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 0x55d14db29420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:34900/0 (socket says 172.21.15.38:34900) 2024-04-22T00:03:52.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.190+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3822246658 learned_addr learned my addr 172.21.15.38:0/3822246658 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.190+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db26070 0x55d14db29b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.190+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d14db2cb00 con 0x55d14db12820 2024-04-22T00:03:52.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.191+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d14da37f10 con 0x55d14db256c0 2024-04-22T00:03:52.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.191+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d14db2cc80 con 0x55d14db26070 2024-04-22T00:03:52.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.191+0000 7f2ae7220740 10 client.0 _ll_register_callbacks cb 0x55d14d78fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.191+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db26070 msgr2=0x55d14db29b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.191+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db26070 0x55d14db29b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3822246658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 msgr2=0x55d14db29420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 0x55d14db29420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d14da09c80 con 0x55d14db12820 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4010d10 tx=0x7f2ad400e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fd83106b5ac275 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ac1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad401e020 con 0x55d14db12820 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2ad4005020 con 0x55d14db12820 2024-04-22T00:03:52.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.192+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ad0005500 con 0x55d14db12820 2024-04-22T00:03:52.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.193+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2ad4026050 con 0x55d14db12820 2024-04-22T00:03:52.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.193+0000 7f2ae7220740 10 client.16389 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.193+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d14d7903f0 con 0x55d14db12820 2024-04-22T00:03:52.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.194+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad0005500 con 0x55d14db12820 2024-04-22T00:03:52.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.194+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f2ad4019070 con 0x55d14db12820 2024-04-22T00:03:52.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.195+0000 7f2ae7220740 10 client.16389 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.196+0000 7f2ae7220740 10 client.16389 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:03:52.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.196+0000 7f2ae7220740 20 client.16389 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.196+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14db39d10 0x55d14db3c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.196+0000 7f2ae7220740 4 client.16389 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:52.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.196+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d14da89a70 con 0x55d14db39d10 2024-04-22T00:03:52.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.197+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14db39d10 0x55d14db3c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.205+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14db39d10 0x55d14db3c100 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ec729c54ff49200 server_cookie=52108fe2b26bd9bf in_seq=0 out_seq=0 2024-04-22T00:03:52.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.205+0000 7f2ac1ffb640 10 client.16389 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:52.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3822246658 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d14da89a70 con 0x55d14db39d10 2024-04-22T00:03:52.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ac1ffb640 10 client.16389 handle_command_reply: tid=0 2024-04-22T00:03:52.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 client.16389 shutdown 2024-04-22T00:03:52.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 msgr2=0x55d14db2c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4010d10 tx=0x7f2ad400e780 comp rx=0 tx=0).stop 2024-04-22T00:03:52.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 shutdown_connections 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14db39d10 0x55d14db3c100 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db26070 0x55d14db29b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db256c0 0x55d14db29420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3822246658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db2c090 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 >> 172.21.15.38:0/3822246658 conn(0x55d14db14cc0 msgr2=0x55d14db14420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ac1ffb640 0 client.16389 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.206+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 shutdown_connections 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.207+0000 7f2ae7220740 1 -- 172.21.15.38:0/3822246658 wait complete. 2024-04-22T00:03:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.207+0000 7f2ae7220740 20 client.16389 trim_cache size 0 max 16384 2024-04-22T00:03:52.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:52 smithi038 ceph-mon[27972]: pgmap v56: 129 pgs: 129 active+clean; 3.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 310 MiB/s wr, 252 op/s 2024-04-22T00:03:52.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:52 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2816768080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.235+0000 7f2ae7220740 1 Processor -- start 2024-04-22T00:03:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.235+0000 7f2ae7220740 1 -- start start 2024-04-22T00:03:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db12820 0x55d14db35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db23800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1ac940 con 0x55d14db256c0 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1ae600 con 0x55d14db26070 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae7220740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1ad010 con 0x55d14db12820 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35604/0 (socket says 172.21.15.38:35604) 2024-04-22T00:03:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 -- 172.21.15.38:0/462397110 learned_addr learned my addr 172.21.15.38:0/462397110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 -- 172.21.15.38:0/462397110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db12820 msgr2=0x55d14db35a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db12820 0x55d14db35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 -- 172.21.15.38:0/462397110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 msgr2=0x55d14db23800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db23800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.236+0000 7f2ae1b01640 1 -- 172.21.15.38:0/462397110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d14d85a7d0 con 0x55d14db256c0 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.237+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000b420 tx=0x7f2ad0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c0c90de49d983f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.237+0000 7f2ac3fff640 1 -- 172.21.15.38:0/462397110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0016040 con 0x55d14db256c0 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.237+0000 7f2ac3fff640 1 -- 172.21.15.38:0/462397110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad0005790 con 0x55d14db256c0 2024-04-22T00:03:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.237+0000 7f2ac3fff640 1 -- 172.21.15.38:0/462397110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0026080 con 0x55d14db256c0 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 -- 172.21.15.38:0/462397110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 msgr2=0x55d14db24d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000b420 tx=0x7f2ad0004ad0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 -- 172.21.15.38:0/462397110 shutdown_connections 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db23800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db256c0 0x55d14db24d50 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 --2- 172.21.15.38:0/462397110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db12820 0x55d14db35a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 -- 172.21.15.38:0/462397110 >> 172.21.15.38:0/462397110 conn(0x55d14db14cc0 msgr2=0x55d14db230f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.238+0000 7f2ae7220740 1 -- 172.21.15.38:0/462397110 shutdown_connections 2024-04-22T00:03:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.239+0000 7f2ae7220740 1 -- 172.21.15.38:0/462397110 wait complete. 2024-04-22T00:03:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.239+0000 7f2ae7220740 1 -- start start 2024-04-22T00:03:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.239+0000 7f2ae7220740 1 Processor -- start 2024-04-22T00:03:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.240+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.240+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db2aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.240+0000 7f2ae7220740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db2b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae7220740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1ade70 con 0x55d14db12820 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae7220740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1ad7a0 con 0x55d14db26070 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae7220740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc1aa920 con 0x55d14db256c0 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae2b03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae7220740 10 client.0 _ll_register_callbacks cb 0x7f2ad4009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae2b03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35616/0 (socket says 172.21.15.38:35616) 2024-04-22T00:03:52.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.241+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3806181933 learned_addr learned my addr 172.21.15.38:0/3806181933 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db2aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 msgr2=0x55d14db2aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db2aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3806181933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 msgr2=0x55d14db2b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db2b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 -- 172.21.15.38:0/3806181933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d14db2b580 con 0x55d14db12820 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ae2b03640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ad401a6a0 tx=0x7f2ad4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583f3c57de29da8e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.242+0000 7f2ac1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad402e020 con 0x55d14db12820 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2ad4005240 con 0x55d14db12820 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d14da83dc0 con 0x55d14db12820 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2ad401e040 con 0x55d14db12820 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ae7220740 10 client.16395 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.243+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d14dba0020 con 0x55d14db12820 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d14da83dc0 con 0x55d14db12820 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f2ad4019070 con 0x55d14db12820 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 10 client.16395 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 10 client.16395 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 20 client.16395 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14dba0750 0x55d14dba2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 4 client.16395 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:52.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.244+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d14da016e0 con 0x55d14dba0750 2024-04-22T00:03:52.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.245+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14dba0750 0x55d14dba2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.247+0000 7f2ac1ffb640 10 client.16395 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.247+0000 7f2ae1b01640 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14dba0750 0x55d14dba2b70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a28bbde47ea3ad6 server_cookie=bb112d15dd0c176f in_seq=0 out_seq=0 2024-04-22T00:03:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.248+0000 7f2ac1ffb640 1 -- 172.21.15.38:0/3806181933 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d14da016e0 con 0x55d14dba0750 2024-04-22T00:03:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.248+0000 7f2ac1ffb640 10 client.16395 handle_command_reply: tid=0 2024-04-22T00:03:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.248+0000 7f2ae7220740 1 client.16395 shutdown 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 msgr2=0x55d14db32100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2ad401a6a0 tx=0x7f2ad4010740 comp rx=0 tx=0).stop 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 shutdown_connections 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55d14dba0750 0x55d14dba2b70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d14db26070 0x55d14db2b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d14db256c0 0x55d14db2aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 --2- 172.21.15.38:0/3806181933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d14db12820 0x55d14db32100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 >> 172.21.15.38:0/3806181933 conn(0x55d14db14cc0 msgr2=0x55d14db13af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ac1ffb640 0 client.16395 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:03:52.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.249+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 shutdown_connections 2024-04-22T00:03:52.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.250+0000 7f2ae7220740 1 -- 172.21.15.38:0/3806181933 wait complete. 2024-04-22T00:03:52.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.250+0000 7f2ae7220740 20 client.16395 trim_cache size 0 max 16384 2024-04-22T00:03:52.243 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:52.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 msgr2=0x7f2ac40a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 0x7f2ac40a5340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0ee830 tx=0x7f2acc02e040 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 msgr2=0x7f2adc197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2acc005e30 tx=0x7f2acc032d10 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 shutdown_connections 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ac40a2e80 0x7f2ac40a5340 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2adc107710 0x7f2adc19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc1005f0 0x7f2adc198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 --2- 172.21.15.38:0/1484568357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2adc0ffc50 0x7f2adc197fe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 >> 172.21.15.38:0/1484568357 conn(0x7f2adc076be0 msgr2=0x7f2adc0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 shutdown_connections 2024-04-22T00:03:52.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.252+0000 7f2ae4566640 1 -- 172.21.15.38:0/1484568357 wait complete. 2024-04-22T00:03:52.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:03:52.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:52 smithi203 ceph-mon[31431]: pgmap v56: 129 pgs: 129 active+clean; 3.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 310 MiB/s wr, 252 op/s 2024-04-22T00:03:52.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:52 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2816768080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:52.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:52 smithi111 ceph-mon[31543]: pgmap v56: 129 pgs: 129 active+clean; 3.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 310 MiB/s wr, 252 op/s 2024-04-22T00:03:52.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:52 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2816768080' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:03:52.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.423+0000 7f58561e5640 1 -- 172.21.15.38:0/2224692698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5850101c20 msgr2=0x7f585010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.423+0000 7f58561e5640 1 --2- 172.21.15.38:0/2224692698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5850101c20 0x7f585010d560 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5848009cd0 tx=0x7f584802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 -- 172.21.15.38:0/2224692698 shutdown_connections 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 --2- 172.21.15.38:0/2224692698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5850101c20 0x7f585010d560 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 --2- 172.21.15.38:0/2224692698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 0x7f58501016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 --2- 172.21.15.38:0/2224692698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58501056d0 0x7f5850105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 -- 172.21.15.38:0/2224692698 >> 172.21.15.38:0/2224692698 conn(0x7f58500767a0 msgr2=0x7f58500fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 -- 172.21.15.38:0/2224692698 shutdown_connections 2024-04-22T00:03:52.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.426+0000 7f58561e5640 1 -- 172.21.15.38:0/2224692698 wait complete. 2024-04-22T00:03:52.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.427+0000 7f58561e5640 1 Processor -- start 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.427+0000 7f58561e5640 1 -- start start 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 0x7f5850115ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5850101c20 0x7f5850116410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f585010f950 con 0x7f58501056d0 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f585010f7d0 con 0x7f5850101260 2024-04-22T00:03:52.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f58561e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f585010fad0 con 0x7f5850101c20 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f584ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f584ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35644/0 (socket says 172.21.15.38:35644) 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f584ffff640 1 -- 172.21.15.38:0/1023947935 learned_addr learned my addr 172.21.15.38:0/1023947935 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f584f7fe640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 0x7f5850115ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.428+0000 7f584effd640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5850101c20 0x7f5850116410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 -- 172.21.15.38:0/1023947935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5850101c20 msgr2=0x7f5850116410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5850101c20 0x7f5850116410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 -- 172.21.15.38:0/1023947935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 msgr2=0x7f5850115ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 0x7f5850115ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 -- 172.21.15.38:0/1023947935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58501118a0 con 0x7f58501056d0 2024-04-22T00:03:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.429+0000 7f584ffff640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5848037040 tx=0x7f5848032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.430+0000 7f584cff9640 1 -- 172.21.15.38:0/1023947935 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848042020 con 0x7f58501056d0 2024-04-22T00:03:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.430+0000 7f584cff9640 1 -- 172.21.15.38:0/1023947935 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5848034640 con 0x7f58501056d0 2024-04-22T00:03:52.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.431+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5850111b30 con 0x7f58501056d0 2024-04-22T00:03:52.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.431+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5850119db0 con 0x7f58501056d0 2024-04-22T00:03:52.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.431+0000 7f584cff9640 1 -- 172.21.15.38:0/1023947935 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848039040 con 0x7f58501056d0 2024-04-22T00:03:52.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.435+0000 7f5858dd0740 1 Processor -- start 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 -- start start 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a947837400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a94783ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a94779b530 con 0x55a947840520 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a947743830 con 0x55a947837020 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.437+0000 7f5858dd0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a9477974f0 con 0x55a947837940 2024-04-22T00:03:52.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.438+0000 7f584cff9640 1 -- 172.21.15.38:0/1023947935 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5848045020 con 0x7f58501056d0 2024-04-22T00:03:52.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.438+0000 7f584effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.438+0000 7f584effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35660/0 (socket says 172.21.15.38:35660) 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.438+0000 7f584effd640 1 -- 172.21.15.38:0/256539394 learned_addr learned my addr 172.21.15.38:0/256539394 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584f7fe640 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a94783ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584effd640 1 -- 172.21.15.38:0/256539394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 msgr2=0x55a94783ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584effd640 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a94783ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584effd640 1 -- 172.21.15.38:0/256539394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 msgr2=0x55a947837400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584effd640 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a947837400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584effd640 1 -- 172.21.15.38:0/256539394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9477672c0 con 0x55a947840520 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584f7fe640 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a94783ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:52.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584cff9640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 0x7f58380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.439+0000 7f584cff9640 1 -- 172.21.15.38:0/1023947935 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f58480f0ce0 con 0x7f58501056d0 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.440+0000 7f584effd640 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f584000cd00 tx=0x7f58400117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb2b54b7e71dfaa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.440+0000 7f584f7fe640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 0x7f58380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.440+0000 7f5835ffb640 1 -- 172.21.15.38:0/256539394 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840012450 con 0x55a947840520 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.440+0000 7f5835ffb640 1 -- 172.21.15.38:0/256539394 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5840012ad0 con 0x55a947840520 2024-04-22T00:03:52.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.440+0000 7f5835ffb640 1 -- 172.21.15.38:0/256539394 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584000fcd0 con 0x55a947840520 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.441+0000 7f5858dd0740 1 -- 172.21.15.38:0/256539394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 msgr2=0x55a947842910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.441+0000 7f5858dd0740 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f584000cd00 tx=0x7f58400117b0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.441+0000 7f584f7fe640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 0x7f58380a50e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55a947836af0 tx=0x7f584400b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 -- 172.21.15.38:0/256539394 shutdown_connections 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947840520 0x55a947842910 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a94783ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 --2- 172.21.15.38:0/256539394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a947837400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 -- 172.21.15.38:0/256539394 >> 172.21.15.38:0/256539394 conn(0x55a947831750 msgr2=0x55a947831b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 -- 172.21.15.38:0/256539394 shutdown_connections 2024-04-22T00:03:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.442+0000 7f5858dd0740 1 -- 172.21.15.38:0/256539394 wait complete. 2024-04-22T00:03:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.443+0000 7f5858dd0740 1 -- start start 2024-04-22T00:03:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.443+0000 7f5858dd0740 1 Processor -- start 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.444+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.444+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a94783d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.444+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a94783dd80 0x55a94785d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f5858dd0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a947854320 con 0x55a947837020 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f5858dd0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a947756f10 con 0x55a947837940 2024-04-22T00:03:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f5858dd0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a9478544a0 con 0x55a94783dd80 2024-04-22T00:03:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f584f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f584f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35674/0 (socket says 172.21.15.38:35674) 2024-04-22T00:03:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f584f7fe640 1 -- 172.21.15.38:0/4082580412 learned_addr learned my addr 172.21.15.38:0/4082580412 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f584ffff640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a94783dd80 0x55a94785d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.445+0000 7f5858dd0740 10 client.0 _ll_register_callbacks cb 0x55a9474aebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 -- 172.21.15.38:0/4082580412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a94783dd80 msgr2=0x55a94785d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a94783dd80 0x55a94785d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 -- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 msgr2=0x55a94783d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a94783d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 -- 172.21.15.38:0/4082580412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a947728c80 con 0x55a947837020 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f582effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.446+0000 7f584f7fe640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5844003f60 tx=0x7f58440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622de350dc9793b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.447+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584401f830 con 0x55a947837020 2024-04-22T00:03:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.447+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f584401f630 con 0x55a947837020 2024-04-22T00:03:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.448+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5840012ad0 con 0x55a947837020 2024-04-22T00:03:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.449+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f584401f830 con 0x55a947837020 2024-04-22T00:03:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.449+0000 7f5858dd0740 10 client.16407 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.449+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9474af3f0 con 0x55a947837020 2024-04-22T00:03:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.449+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5840012ad0 con 0x55a947837020 2024-04-22T00:03:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f58440319e0 con 0x55a947837020 2024-04-22T00:03:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 10 client.16407 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 10 client.16407 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:03:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 20 client.16407 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a947861a80 0x55a947863e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 4 client.16407 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.450+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9477a8a70 con 0x55a947861a80 2024-04-22T00:03:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.451+0000 7f584effd640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a947861a80 0x55a947863e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.452+0000 7f582effd640 10 client.16407 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:52.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.452+0000 7f584effd640 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a947861a80 0x55a947863e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8125c25ea762dad5 server_cookie=d18a76376c87f9f8 in_seq=0 out_seq=0 2024-04-22T00:03:52.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.455+0000 7f582effd640 1 -- 172.21.15.38:0/4082580412 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9477a8a70 con 0x55a947861a80 2024-04-22T00:03:52.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.455+0000 7f582effd640 10 client.16407 handle_command_reply: tid=0 2024-04-22T00:03:52.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.455+0000 7f5858dd0740 1 client.16407 shutdown 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.456+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 msgr2=0x55a94783d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.456+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5844003f60 tx=0x7f58440046c0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 shutdown_connections 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a94783dd80 0x55a94785d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a94783d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a947861a80 0x55a947863e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 --2- 172.21.15.38:0/4082580412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783d300 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.457+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 >> 172.21.15.38:0/4082580412 conn(0x55a947831750 msgr2=0x55a9478334a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.458+0000 7f582effd640 0 client.16407 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.458+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 shutdown_connections 2024-04-22T00:03:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.458+0000 7f5858dd0740 1 -- 172.21.15.38:0/4082580412 wait complete. 2024-04-22T00:03:52.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.458+0000 7f5858dd0740 20 client.16407 trim_cache size 0 max 16384 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 Processor -- start 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 -- start start 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 0x55a947830a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a9478696a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f58501b5140 con 0x55a947831750 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58501b6e00 con 0x55a947837020 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.482+0000 7f5858dd0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f58501b5810 con 0x55a947837940 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.483+0000 7f584ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.483+0000 7f584ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:35600/0 (socket says 172.21.15.38:35600) 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.483+0000 7f584ffff640 1 -- 172.21.15.38:0/2758943018 learned_addr learned my addr 172.21.15.38:0/2758943018 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.483+0000 7f584f7fe640 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 0x55a947830a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 -- 172.21.15.38:0/2758943018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 msgr2=0x55a947869be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 -- 172.21.15.38:0/2758943018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 msgr2=0x55a9478696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a9478696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 -- 172.21.15.38:0/2758943018 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9475797d0 con 0x55a947831750 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584ffff640 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.484+0000 7f584f7fe640 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 0x55a947830a90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f584401ccf0 tx=0x7f5844005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4df9282509c3a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.485+0000 7f58357fa640 1 -- 172.21.15.38:0/2758943018 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584403f070 con 0x55a947831750 2024-04-22T00:03:52.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.485+0000 7f58357fa640 1 -- 172.21.15.38:0/2758943018 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f584400fc90 con 0x55a947831750 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.485+0000 7f58357fa640 1 -- 172.21.15.38:0/2758943018 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844036040 con 0x55a947831750 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 -- 172.21.15.38:0/2758943018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 msgr2=0x55a947830a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 0x55a947830a90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f584401ccf0 tx=0x7f5844005d40 comp rx=0 tx=0).stop 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 -- 172.21.15.38:0/2758943018 shutdown_connections 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947837940 0x55a947869be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837020 0x55a9478696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 --2- 172.21.15.38:0/2758943018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947831750 0x55a947830a90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.486+0000 7f5858dd0740 1 -- 172.21.15.38:0/2758943018 >> 172.21.15.38:0/2758943018 conn(0x55a947838cf0 msgr2=0x55a9478331a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.487+0000 7f5858dd0740 1 -- 172.21.15.38:0/2758943018 shutdown_connections 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.487+0000 7f5858dd0740 1 -- 172.21.15.38:0/2758943018 wait complete. 2024-04-22T00:03:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.487+0000 7f5858dd0740 1 -- start start 2024-04-22T00:03:52.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 Processor -- start 2024-04-22T00:03:52.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947831750 0x55a94783a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a9478c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f58501b6670 con 0x55a947837020 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f58501b5fa0 con 0x55a947837940 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5850071ec0 con 0x55a947831750 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f584effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f584effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35690/0 (socket says 172.21.15.38:35690) 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f584effd640 1 -- 172.21.15.38:0/36535255 learned_addr learned my addr 172.21.15.38:0/36535255 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f5858dd0740 10 client.0 _ll_register_callbacks cb 0x7f5844005c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.488+0000 7f584f7fe640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947831750 0x55a94783a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 -- 172.21.15.38:0/36535255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947831750 msgr2=0x55a94783a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947831750 0x55a94783a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 -- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 msgr2=0x55a9478c6f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a9478c6f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 -- 172.21.15.38:0/36535255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9478c7450 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f584effd640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5840011cb0 tx=0x7f584000f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9a7d9550ed8870 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f582effd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584002c060 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f584000fae0 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9477a2dc0 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.489+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5840033050 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 10 client.16413 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9478c86e0 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a9477a2dc0 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f58400048b0 con 0x55a947837020 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 10 client.16413 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 10 client.16413 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 20 client.16413 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a94783dd80 0x55a9478caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 4 client.16413 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:52.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.490+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9477206e0 con 0x55a94783dd80 2024-04-22T00:03:52.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.493+0000 7f584f7fe640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a94783dd80 0x55a9478caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.496+0000 7f582effd640 10 client.16413 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:52.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.496+0000 7f584f7fe640 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a94783dd80 0x55a9478caa40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3917d275b1e783 server_cookie=b962e6dbb2bc1af2 in_seq=0 out_seq=0 2024-04-22T00:03:52.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.496+0000 7f582effd640 1 -- 172.21.15.38:0/36535255 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9477206e0 con 0x55a94783dd80 2024-04-22T00:03:52.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.496+0000 7f582effd640 10 client.16413 handle_command_reply: tid=0 2024-04-22T00:03:52.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.496+0000 7f5858dd0740 1 client.16413 shutdown 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 msgr2=0x55a94783ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5840011cb0 tx=0x7f584000f440 comp rx=0 tx=0).stop 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 shutdown_connections 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a947837940 0x55a9478c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55a94783dd80 0x55a9478caa40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a947837020 0x55a94783ac00 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 --2- 172.21.15.38:0/36535255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a947831750 0x55a94783a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 >> 172.21.15.38:0/36535255 conn(0x55a947838cf0 msgr2=0x55a94783eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f582effd640 0 client.16413 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 shutdown_connections 2024-04-22T00:03:52.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 1 -- 172.21.15.38:0/36535255 wait complete. 2024-04-22T00:03:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.497+0000 7f5858dd0740 20 client.16413 trim_cache size 0 max 16384 2024-04-22T00:03:52.491 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:52.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 msgr2=0x7f58380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 0x7f58380a50e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55a947836af0 tx=0x7f584400b390 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 msgr2=0x7f5850110fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5848037040 tx=0x7f5848032ea0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 shutdown_connections 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58380a2c20 0x7f58380a50e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f58501056d0 0x7f5850110fe0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5850101c20 0x7f5850116410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 --2- 172.21.15.38:0/1023947935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5850101260 0x7f5850115ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 >> 172.21.15.38:0/1023947935 conn(0x7f58500767a0 msgr2=0x7f58500fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 shutdown_connections 2024-04-22T00:03:52.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.498+0000 7f58561e5640 1 -- 172.21.15.38:0/1023947935 wait complete. 2024-04-22T00:03:52.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:03:52.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.673+0000 7fb43b59e640 1 -- 172.21.15.38:0/1457651421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c1091a0 msgr2=0x7fb43c10c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.673+0000 7fb43b59e640 1 --2- 172.21.15.38:0/1457651421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c1091a0 0x7fb43c10c8a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb430009860 tx=0x7fb43002bb90 comp rx=0 tx=0).stop 2024-04-22T00:03:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.674+0000 7fb43b59e640 1 -- 172.21.15.38:0/1457651421 shutdown_connections 2024-04-22T00:03:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.674+0000 7fb43b59e640 1 --2- 172.21.15.38:0/1457651421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 0x7fb43c114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.674+0000 7fb43b59e640 1 --2- 172.21.15.38:0/1457651421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c1091a0 0x7fb43c10c8a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.674+0000 7fb43b59e640 1 --2- 172.21.15.38:0/1457651421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 0x7fb43c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.674+0000 7fb43b59e640 1 -- 172.21.15.38:0/1457651421 >> 172.21.15.38:0/1457651421 conn(0x7fb43c077160 msgr2=0x7fb43c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.676+0000 7fb43b59e640 1 -- 172.21.15.38:0/1457651421 shutdown_connections 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.676+0000 7fb43b59e640 1 -- 172.21.15.38:0/1457651421 wait complete. 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 Processor -- start 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 -- start start 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 0x7fb43c083de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 0x7fb43c07ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c118090 con 0x7fb43c1087f0 2024-04-22T00:03:52.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c117f10 con 0x7fb43c07d3a0 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.678+0000 7fb43b59e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c118210 con 0x7fb43c10cfd0 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.679+0000 7fb43ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.679+0000 7fb43ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:35002/0 (socket says 172.21.15.38:35002) 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.679+0000 7fb43ad9d640 1 -- 172.21.15.38:0/646595484 learned_addr learned my addr 172.21.15.38:0/646595484 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb43a59c640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 0x7fb43c083de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb439d9b640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 0x7fb43c07ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb43ad9d640 1 -- 172.21.15.38:0/646595484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 msgr2=0x7fb43c07ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb43ad9d640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 0x7fb43c07ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb43ad9d640 1 -- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 msgr2=0x7fb43c083de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.680+0000 7fb43ad9d640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 0x7fb43c083de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.681+0000 7fb43ad9d640 1 -- 172.21.15.38:0/646595484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c081900 con 0x7fb43c07d3a0 2024-04-22T00:03:52.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.689+0000 7fb43ad9d640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb43400eee0 tx=0x7fb43400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.694+0000 7fb42b7fe640 1 -- 172.21.15.38:0/646595484 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434004b20 con 0x7fb43c07d3a0 2024-04-22T00:03:52.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.694+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c081bf0 con 0x7fb43c07d3a0 2024-04-22T00:03:52.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.694+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c082110 con 0x7fb43c07d3a0 2024-04-22T00:03:52.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.699+0000 7fb4437e0740 1 Processor -- start 2024-04-22T00:03:52.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.703+0000 7fb42b7fe640 1 -- 172.21.15.38:0/646595484 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb434003e50 con 0x7fb43c07d3a0 2024-04-22T00:03:52.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.703+0000 7fb42b7fe640 1 -- 172.21.15.38:0/646595484 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb434005280 con 0x7fb43c07d3a0 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 -- start start 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05faf0 0x557d5a068190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a06aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557d59fc3530 con 0x557d5a05f1d0 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557d59f6b830 con 0x557d5a0686d0 2024-04-22T00:03:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.710+0000 7fb4437e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557d59fbf4f0 con 0x557d5a05faf0 2024-04-22T00:03:52.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.715+0000 7fb43a59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05faf0 0x557d5a068190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.715+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.715+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35704/0 (socket says 172.21.15.38:35704) 2024-04-22T00:03:52.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.715+0000 7fb439d9b640 1 -- 172.21.15.38:0/3428928725 learned_addr learned my addr 172.21.15.38:0/3428928725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.715+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a06aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.712 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.717+0000 7fb42b7fe640 1 -- 172.21.15.38:0/646595484 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb43401d020 con 0x7fb43c07d3a0 2024-04-22T00:03:52.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.719+0000 7fb42b7fe640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 0x7fb4240a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.719+0000 7fb42b7fe640 1 -- 172.21.15.38:0/646595484 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb4340cf6c0 con 0x7fb43c07d3a0 2024-04-22T00:03:52.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb43ad9d640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 0x7fb4240a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb439d9b640 1 -- 172.21.15.38:0/3428928725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05faf0 msgr2=0x557d5a068190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05faf0 0x557d5a068190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb439d9b640 1 -- 172.21.15.38:0/3428928725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 msgr2=0x557d5a06aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a06aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.723+0000 7fb439d9b640 1 -- 172.21.15.38:0/3428928725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d59f8f2c0 con 0x557d5a05f1d0 2024-04-22T00:03:52.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.725+0000 7fb43ad9d640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 0x7fb4240a5390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb43409b730 tx=0x7fb434097380 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.727+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb4300057c0 tx=0x7fb430000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7a5df80f76761e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.727+0000 7fb428ff9640 1 -- 172.21.15.38:0/3428928725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430048070 con 0x557d5a05f1d0 2024-04-22T00:03:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.728+0000 7fb428ff9640 1 -- 172.21.15.38:0/3428928725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430004240 con 0x557d5a05f1d0 2024-04-22T00:03:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.728+0000 7fb428ff9640 1 -- 172.21.15.38:0/3428928725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430048070 con 0x557d5a05f1d0 2024-04-22T00:03:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.730+0000 7fb4437e0740 1 -- 172.21.15.38:0/3428928725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 msgr2=0x557d5a05f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.730+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb4300057c0 tx=0x7fb430000bc0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.733+0000 7fb4437e0740 1 -- 172.21.15.38:0/3428928725 shutdown_connections 2024-04-22T00:03:52.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.733+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a06aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.733+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05faf0 0x557d5a068190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.733+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3428928725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a05f5b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.733+0000 7fb4437e0740 1 -- 172.21.15.38:0/3428928725 >> 172.21.15.38:0/3428928725 conn(0x557d5a059900 msgr2=0x557d5a059ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.734+0000 7fb4437e0740 1 -- 172.21.15.38:0/3428928725 shutdown_connections 2024-04-22T00:03:52.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.735+0000 7fb4437e0740 1 -- 172.21.15.38:0/3428928725 wait complete. 2024-04-22T00:03:52.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 -- start start 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 Processor -- start 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a067ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 0x557d5a065380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a0658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557d5a06b460 con 0x557d5a05faf0 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557d59f7ef10 con 0x557d5a0686d0 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557d5a06b5e0 con 0x557d5a05f1d0 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.736+0000 7fb4437e0740 10 client.0 _ll_register_callbacks cb 0x557d59cd6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.737+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a0658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.737+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a0658c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:35030/0 (socket says 172.21.15.38:35030) 2024-04-22T00:03:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.737+0000 7fb439d9b640 1 -- 172.21.15.38:0/2131564154 learned_addr learned my addr 172.21.15.38:0/2131564154 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.738+0000 7fb43a59c640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 0x557d5a065380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 -- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 msgr2=0x557d5a067ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a067ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 -- 172.21.15.38:0/2131564154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 msgr2=0x557d5a0658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a0658c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 -- 172.21.15.38:0/2131564154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d59f50c80 con 0x557d5a05faf0 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb43a59c640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 0x557d5a065380 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb42c006fd0 tx=0x7fb42c0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d5eecfeb0ee0e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.739+0000 7fb421ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.740+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c013670 con 0x557d5a05faf0 2024-04-22T00:03:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.740+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb430004240 con 0x557d5a05faf0 2024-04-22T00:03:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.740+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb42c013810 con 0x557d5a05faf0 2024-04-22T00:03:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.740+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb42c026050 con 0x557d5a05faf0 2024-04-22T00:03:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.742+0000 7fb4437e0740 10 client.16422 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.742+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557d59cd73f0 con 0x557d5a05faf0 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.742+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430004240 con 0x557d5a05faf0 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.742+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fb42c01d400 con 0x557d5a05faf0 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 10 client.16422 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 10 client.16422 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 20 client.16422 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a06d280 0x557d5a06f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 4 client.16422 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.743+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d59fd0a70 con 0x557d5a06d280 2024-04-22T00:03:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.746+0000 7fb439d9b640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a06d280 0x557d5a06f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.749+0000 7fb439d9b640 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a06d280 0x557d5a06f6c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=74b2abdca999432f server_cookie=29034b1c0a63b776 in_seq=0 out_seq=0 2024-04-22T00:03:52.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.749+0000 7fb421ffb640 10 client.16422 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:52.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.751+0000 7fb421ffb640 1 -- 172.21.15.38:0/2131564154 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d59fd0a70 con 0x557d5a06d280 2024-04-22T00:03:52.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.751+0000 7fb421ffb640 10 client.16422 handle_command_reply: tid=0 2024-04-22T00:03:52.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.751+0000 7fb4437e0740 1 client.16422 shutdown 2024-04-22T00:03:52.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 msgr2=0x557d5a065380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 0x557d5a065380 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb42c006fd0 tx=0x7fb42c0119d0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 shutdown_connections 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a06d280 0x557d5a06f6c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a0658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05faf0 0x557d5a065380 secure :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb42c006fd0 tx=0x7fb42c0119d0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 --2- 172.21.15.38:0/2131564154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a067ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb421ffb640 0 client.16422 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 >> 172.21.15.38:0/2131564154 conn(0x557d5a059900 msgr2=0x557d5a05b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.754+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 shutdown_connections 2024-04-22T00:03:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.755+0000 7fb4437e0740 1 -- 172.21.15.38:0/2131564154 wait complete. 2024-04-22T00:03:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.755+0000 7fb4437e0740 20 client.16422 trim_cache size 0 max 16384 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 Processor -- start 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 -- start start 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a058920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a067ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c00fd70 con 0x557d5a085510 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c011a30 con 0x557d5a0686d0 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.774+0000 7fb4437e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c010440 con 0x557d5a05f1d0 2024-04-22T00:03:52.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35732/0 (socket says 172.21.15.38:35732) 2024-04-22T00:03:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 -- 172.21.15.38:0/3975728346 learned_addr learned my addr 172.21.15.38:0/3975728346 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 -- 172.21.15.38:0/3975728346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 msgr2=0x557d5a058920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a058920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 -- 172.21.15.38:0/3975728346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 msgr2=0x557d5a067ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a067ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.775+0000 7fb439d9b640 1 -- 172.21.15.38:0/3975728346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d59da17d0 con 0x557d5a085510 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.776+0000 7fb439d9b640 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb43002c1a0 tx=0x7fb4300058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b981a2fef84844 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.776+0000 7fb423fff640 1 -- 172.21.15.38:0/3975728346 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430000e60 con 0x557d5a085510 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.776+0000 7fb423fff640 1 -- 172.21.15.38:0/3975728346 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430055070 con 0x557d5a085510 2024-04-22T00:03:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.776+0000 7fb423fff640 1 -- 172.21.15.38:0/3975728346 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430000e60 con 0x557d5a085510 2024-04-22T00:03:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- 172.21.15.38:0/3975728346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 msgr2=0x557d5a067ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb43002c1a0 tx=0x7fb4300058c0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- 172.21.15.38:0/3975728346 shutdown_connections 2024-04-22T00:03:52.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a085510 0x557d5a067ff0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a067ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 --2- 172.21.15.38:0/3975728346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a05f1d0 0x557d5a058920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- 172.21.15.38:0/3975728346 >> 172.21.15.38:0/3975728346 conn(0x557d5a059900 msgr2=0x557d5a058210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- 172.21.15.38:0/3975728346 shutdown_connections 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- 172.21.15.38:0/3975728346 wait complete. 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 -- start start 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.777+0000 7fb4437e0740 1 Processor -- start 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a07cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a085510 0x557d5a07d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c0112a0 con 0x557d5a05f1d0 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c010bd0 con 0x557d5a0686d0 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c00dd50 con 0x557d5a085510 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35734/0 (socket says 172.21.15.38:35734) 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 -- 172.21.15.38:0/704473205 learned_addr learned my addr 172.21.15.38:0/704473205 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb4437e0740 10 client.0 _ll_register_callbacks cb 0x7fb42c009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 -- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a085510 msgr2=0x557d5a07d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a085510 0x557d5a07d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 -- 172.21.15.38:0/704473205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 msgr2=0x557d5a07cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a07cc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.778+0000 7fb439d9b640 1 -- 172.21.15.38:0/704473205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d5a07d840 con 0x557d5a05f1d0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb439d9b640 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb430032570 tx=0x7fb430005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3cb9c1289c1300 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb421ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430060060 con 0x557d5a05f1d0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb430004040 con 0x557d5a05f1d0 2024-04-22T00:03:52.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb430055070 con 0x557d5a05f1d0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb430060060 con 0x557d5a05f1d0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb4437e0740 10 client.16428 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.779+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557d5a101460 con 0x557d5a05f1d0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430055070 con 0x557d5a05f1d0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fb43004b040 con 0x557d5a05f1d0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 10 client.16428 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 10 client.16428 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 20 client.16428 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a101780 0x557d5a103b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 4 client.16428 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.780+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d59fcadc0 con 0x557d5a101780 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.781+0000 7fb43a59c640 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a101780 0x557d5a103b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.782+0000 7fb43a59c640 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a101780 0x557d5a103b70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d608c3d05ac727a server_cookie=98c48dabbd525ff7 in_seq=0 out_seq=0 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.782+0000 7fb421ffb640 10 client.16428 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.782+0000 7fb421ffb640 1 -- 172.21.15.38:0/704473205 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d59fcadc0 con 0x557d5a101780 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.782+0000 7fb421ffb640 10 client.16428 handle_command_reply: tid=0 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.782+0000 7fb4437e0740 1 client.16428 shutdown 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 msgr2=0x557d5a080890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb430032570 tx=0x7fb430005bd0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 shutdown_connections 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557d5a101780 0x557d5a103b70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557d5a085510 0x557d5a07d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb421ffb640 0 client.16428 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d5a0686d0 0x557d5a07cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 --2- 172.21.15.38:0/704473205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d5a05f1d0 0x557d5a080890 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 >> 172.21.15.38:0/704473205 conn(0x557d5a059900 msgr2=0x557d5a058210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 shutdown_connections 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 1 -- 172.21.15.38:0/704473205 wait complete. 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.783+0000 7fb4437e0740 20 client.16428 trim_cache size 0 max 16384 2024-04-22T00:03:52.777 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 msgr2=0x7fb4240a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 0x7fb4240a5390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb43409b730 tx=0x7fb434097380 comp rx=0 tx=0).stop 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 msgr2=0x7fb43c07d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb43400eee0 tx=0x7fb43400c740 comp rx=0 tx=0).stop 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 shutdown_connections 2024-04-22T00:03:52.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb4240a2ed0 0x7fb4240a5390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb43c07d3a0 0x7fb43c07d850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb43c10cfd0 0x7fb43c07ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 --2- 172.21.15.38:0/646595484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c1087f0 0x7fb43c083de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 >> 172.21.15.38:0/646595484 conn(0x7fb43c077160 msgr2=0x7fb43c10b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 shutdown_connections 2024-04-22T00:03:52.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.784+0000 7fb43b59e640 1 -- 172.21.15.38:0/646595484 wait complete. 2024-04-22T00:03:52.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:03:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.999+0000 7f354b525640 1 -- 172.21.15.38:0/1712640411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544107720 msgr2=0x7f3544107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.999+0000 7f354b525640 1 --2- 172.21.15.38:0/1712640411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544107720 0x7f3544107b00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3540009930 tx=0x7f354002bc40 comp rx=0 tx=0).stop 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.999+0000 7f354b525640 1 -- 172.21.15.38:0/1712640411 shutdown_connections 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.999+0000 7f354b525640 1 --2- 172.21.15.38:0/1712640411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35441020d0 0x7f354410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:52.999+0000 7f354b525640 1 --2- 172.21.15.38:0/1712640411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544101710 0x7f3544101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.000+0000 7f354b525640 1 --2- 172.21.15.38:0/1712640411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3544107720 0x7f3544107b00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.000+0000 7f354b525640 1 -- 172.21.15.38:0/1712640411 >> 172.21.15.38:0/1712640411 conn(0x7f3544078b90 msgr2=0x7f3544078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.000+0000 7f354b525640 1 -- 172.21.15.38:0/1712640411 shutdown_connections 2024-04-22T00:03:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.000+0000 7f354b525640 1 -- 172.21.15.38:0/1712640411 wait complete. 2024-04-22T00:03:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.001+0000 7f354b525640 1 Processor -- start 2024-04-22T00:03:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.001+0000 7f354b525640 1 -- start start 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3544101710 0x7f3544198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 0x7f354419c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111c20 con 0x7f35441020d0 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111aa0 con 0x7f3544101710 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f354b525640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111da0 con 0x7f3544107720 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35766/0 (socket says 172.21.15.38:35766) 2024-04-22T00:03:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f3548ac0640 1 -- 172.21.15.38:0/541828704 learned_addr learned my addr 172.21.15.38:0/541828704 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.002+0000 7f35492c1640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3544101710 0x7f3544198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3548ac0640 1 -- 172.21.15.38:0/541828704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 msgr2=0x7f354419c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3549ac2640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 0x7f354419c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3548ac0640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 0x7f354419c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3548ac0640 1 -- 172.21.15.38:0/541828704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3544101710 msgr2=0x7f3544198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3548ac0640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3544101710 0x7f3544198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3548ac0640 1 -- 172.21.15.38:0/541828704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354419d110 con 0x7f35441020d0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.003+0000 7f3549ac2640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 0x7f354419c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.004+0000 7f3548ac0640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f353400b450 tx=0x7f353400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.004+0000 7f353a7fc640 1 -- 172.21.15.38:0/541828704 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534015020 con 0x7f35441020d0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.004+0000 7f353a7fc640 1 -- 172.21.15.38:0/541828704 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3534011d70 con 0x7f35441020d0 2024-04-22T00:03:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.004+0000 7f353a7fc640 1 -- 172.21.15.38:0/541828704 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353400f340 con 0x7f35441020d0 2024-04-22T00:03:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.006+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354419d400 con 0x7f35441020d0 2024-04-22T00:03:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.006+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441a4c90 con 0x7f35441020d0 2024-04-22T00:03:53.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.011+0000 7f354e1eb740 1 Processor -- start 2024-04-22T00:03:53.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.011+0000 7f354e1eb740 1 -- start start 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.011+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.011+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699428200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699428c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f354e1eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f699392530 con 0x55f69943b6c0 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f354e1eb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f69933a830 con 0x55f69943c070 2024-04-22T00:03:53.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f354e1eb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f69938e4f0 con 0x55f699428820 2024-04-22T00:03:53.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f3549ac2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f3549ac2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35778/0 (socket says 172.21.15.38:35778) 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f3549ac2640 1 -- 172.21.15.38:0/4249353659 learned_addr learned my addr 172.21.15.38:0/4249353659 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.012+0000 7f35492c1640 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699428200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 -- 172.21.15.38:0/4249353659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 msgr2=0x55f699428c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699428c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 -- 172.21.15.38:0/4249353659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 msgr2=0x55f699428200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699428200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 -- 172.21.15.38:0/4249353659 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f69935e2c0 con 0x55f69943b6c0 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f35492c1640 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699428200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f353a7fc640 1 -- 172.21.15.38:0/541828704 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3534018020 con 0x7f35441020d0 2024-04-22T00:03:53.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f3549ac2640 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f353c007f30 tx=0x7f353c00c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f515d45103f964f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.013+0000 7f352f7fe640 1 -- 172.21.15.38:0/4249353659 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c014070 con 0x55f69943b6c0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.014+0000 7f352f7fe640 1 -- 172.21.15.38:0/4249353659 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c00fd30 con 0x55f69943b6c0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.014+0000 7f352f7fe640 1 -- 172.21.15.38:0/4249353659 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c00ed90 con 0x55f69943b6c0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.014+0000 7f353a7fc640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 0x7f35300a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.014+0000 7f353a7fc640 1 -- 172.21.15.38:0/541828704 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f35340cef80 con 0x7f35441020d0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.014+0000 7f35492c1640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 0x7f35300a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 -- 172.21.15.38:0/4249353659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 msgr2=0x55f69943baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f353c007f30 tx=0x7f353c00c750 comp rx=0 tx=0).stop 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 -- 172.21.15.38:0/4249353659 shutdown_connections 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699428c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699428200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 --2- 172.21.15.38:0/4249353659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943baa0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f35492c1640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 0x7f35300a52c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3540009900 tx=0x7f354002e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 -- 172.21.15.38:0/4249353659 >> 172.21.15.38:0/4249353659 conn(0x55f69942acc0 msgr2=0x55f69942b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.015+0000 7f354e1eb740 1 -- 172.21.15.38:0/4249353659 shutdown_connections 2024-04-22T00:03:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.016+0000 7f354e1eb740 1 -- 172.21.15.38:0/4249353659 wait complete. 2024-04-22T00:03:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.016+0000 7f354e1eb740 1 -- start start 2024-04-22T00:03:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.017+0000 7f354e1eb740 1 Processor -- start 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.017+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f69943b6c0 0x55f69943f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f69943fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f699442b00 con 0x55f699428820 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35790/0 (socket says 172.21.15.38:35790) 2024-04-22T00:03:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 -- 172.21.15.38:0/2350251765 learned_addr learned my addr 172.21.15.38:0/2350251765 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f69934df10 con 0x55f69943c070 2024-04-22T00:03:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f699442c80 con 0x55f69943b6c0 2024-04-22T00:03:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f3548ac0640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f69943b6c0 0x55f69943f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f354e1eb740 10 client.0 _ll_register_callbacks cb 0x55f6990a5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 -- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f69943b6c0 msgr2=0x55f69943f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f69943b6c0 0x55f69943f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 -- 172.21.15.38:0/2350251765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 msgr2=0x55f69943fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f69943fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.018+0000 7f35492c1640 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f69931fc80 con 0x55f699428820 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f35492c1640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f354002e010 tx=0x7f3540000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baac22e40b9f8e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f352cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354004f070 con 0x55f699428820 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3540004c10 con 0x55f699428820 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f353c00fd30 con 0x55f699428820 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3540005000 con 0x55f699428820 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f354e1eb740 10 client.16440 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f6990a63f0 con 0x55f699428820 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.019+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c00fd30 con 0x55f699428820 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f3540054460 con 0x55f699428820 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 10 client.16440 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 10 client.16440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 20 client.16440 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f69944fdf0 0x55f699452220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 4 client.16440 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f69939fa70 con 0x55f69944fdf0 2024-04-22T00:03:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.020+0000 7f3548ac0640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f69944fdf0 0x55f699452220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.021+0000 7f3548ac0640 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f69944fdf0 0x55f699452220 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db5cc46ee1590696 server_cookie=c86b798eee48ae68 in_seq=0 out_seq=0 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.021+0000 7f352cff9640 10 client.16440 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.022+0000 7f352cff9640 1 -- 172.21.15.38:0/2350251765 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f69939fa70 con 0x55f69944fdf0 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.022+0000 7f352cff9640 10 client.16440 handle_command_reply: tid=0 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 client.16440 shutdown 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 msgr2=0x55f6994420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f354002e010 tx=0x7f3540000bc0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 shutdown_connections 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f69944fdf0 0x55f699452220 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f69943fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f69943b6c0 0x55f69943f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f352cff9640 0 client.16440 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 --2- 172.21.15.38:0/2350251765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f699428820 0x55f6994420b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 >> 172.21.15.38:0/2350251765 conn(0x55f69942acc0 msgr2=0x55f69942a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 shutdown_connections 2024-04-22T00:03:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 1 -- 172.21.15.38:0/2350251765 wait complete. 2024-04-22T00:03:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.023+0000 7f354e1eb740 20 client.16440 trim_cache size 0 max 16384 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f354e1eb740 1 Processor -- start 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f354e1eb740 1 -- start start 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699441b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f35492c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f35492c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:35692/0 (socket says 172.21.15.38:35692) 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35800/0 (socket says 172.21.15.38:35800) 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.053+0000 7f35492c1640 1 -- 172.21.15.38:0/1295756029 learned_addr learned my addr 172.21.15.38:0/1295756029 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35441aca30 con 0x55f69943b6c0 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3549ac2640 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699441b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ae6f0 con 0x55f69943c070 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ad100 con 0x55f699428820 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 -- 172.21.15.38:0/1295756029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 msgr2=0x55f69944ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 -- 172.21.15.38:0/1295756029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 msgr2=0x55f699441b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699441b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 -- 172.21.15.38:0/1295756029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f69943fea0 con 0x55f69943b6c0 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f35492c1640 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:53.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f3548ac0640 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f35340cc440 tx=0x7f35340bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a844ce0dd96bf7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f352effd640 1 -- 172.21.15.38:0/1295756029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35340b4070 con 0x55f69943b6c0 2024-04-22T00:03:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f352effd640 1 -- 172.21.15.38:0/1295756029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35340b9e50 con 0x55f69943b6c0 2024-04-22T00:03:53.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.054+0000 7f352effd640 1 -- 172.21.15.38:0/1295756029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534084070 con 0x55f69943b6c0 2024-04-22T00:03:53.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.055+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 msgr2=0x55f69943ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.055+0000 7f354e1eb740 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f35340cc440 tx=0x7f35340bb040 comp rx=0 tx=0).stop 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 shutdown_connections 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f699441b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f69943ac30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 --2- 172.21.15.38:0/1295756029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f69944ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 >> 172.21.15.38:0/1295756029 conn(0x55f69942acc0 msgr2=0x55f69943a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 shutdown_connections 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 -- 172.21.15.38:0/1295756029 wait complete. 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 -- start start 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.056+0000 7f354e1eb740 1 Processor -- start 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699441490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f6994b5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f3548ac0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35812/0 (socket says 172.21.15.38:35812) 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f35492c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699441490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f3548ac0640 1 -- 172.21.15.38:0/3246750154 learned_addr learned my addr 172.21.15.38:0/3246750154 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f35441adf60 con 0x55f69943b6c0 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ad890 con 0x55f69943c070 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35441aaa10 con 0x55f699428820 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.057+0000 7f3549ac2640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f6994b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 -- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 msgr2=0x55f699441490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699441490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 -- 172.21.15.38:0/3246750154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 msgr2=0x55f6994b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f6994b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6991707d0 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f35492c1640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699441490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f3548ac0640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f35340bb480 tx=0x7f35340a3f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7a1fa1f97ed2c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f352cff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f352cff9640 1 -- 172.21.15.38:0/3246750154 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534084040 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f352cff9640 1 -- 172.21.15.38:0/3246750154 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3534097880 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f354e1eb740 10 client.0 _ll_register_callbacks cb 0x7f354002cc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35340b9e50 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f352cff9640 1 -- 172.21.15.38:0/3246750154 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f35340b4040 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f354e1eb740 10 client.16446 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.058+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55f6994b6660 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f352cff9640 1 -- 172.21.15.38:0/3246750154 <== mon.0 v2:172.21.15.38:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f3534022060 con 0x55f69943b6c0 2024-04-22T00:03:53.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 10 client.16446 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 10 client.16446 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 20 client.16446 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f6994b6a50 0x55f6994b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 4 client.16446 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f35340b9e50 con 0x55f6994b6a50 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f35492c1640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f6994b6a50 0x55f6994b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f35492c1640 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f6994b6a50 0x55f6994b8e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c67c09ddd8ea8e9c server_cookie=1905b1ef25b3590d in_seq=0 out_seq=0 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.059+0000 7f352cff9640 10 client.16446 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f352cff9640 1 -- 172.21.15.38:0/3246750154 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f35340b9e50 con 0x55f6994b6a50 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f352cff9640 10 client.16446 handle_command_reply: tid=0 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 client.16446 shutdown 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 msgr2=0x55f6994419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f35340bb480 tx=0x7f35340a3f10 comp rx=0 tx=0).stop 2024-04-22T00:03:53.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 shutdown_connections 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55f6994b6a50 0x55f6994b8e70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f69943c070 0x55f6994b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f352cff9640 0 client.16446 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f69943b6c0 0x55f6994419d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 --2- 172.21.15.38:0/3246750154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f699428820 0x55f699441490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 >> 172.21.15.38:0/3246750154 conn(0x55f69942acc0 msgr2=0x55f69943a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 shutdown_connections 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.060+0000 7f354e1eb740 1 -- 172.21.15.38:0/3246750154 wait complete. 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354e1eb740 20 client.16446 trim_cache size 0 max 16384 2024-04-22T00:03:53.059 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 msgr2=0x7f35300a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 0x7f35300a52c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3540009900 tx=0x7f354002e040 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 msgr2=0x7f3544198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f353400b450 tx=0x7f353400b910 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 shutdown_connections 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f35300a2e00 0x7f35300a52c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3544107720 0x7f354419c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35441020d0 0x7f3544198660 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 --2- 172.21.15.38:0/541828704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3544101710 0x7f3544198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 >> 172.21.15.38:0/541828704 conn(0x7f3544078b90 msgr2=0x7f35440ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.061+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 shutdown_connections 2024-04-22T00:03:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.062+0000 7f354b525640 1 -- 172.21.15.38:0/541828704 wait complete. 2024-04-22T00:03:53.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:03:53.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 -- 172.21.15.38:0/1280040330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100fd0 msgr2=0x7f254410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 --2- 172.21.15.38:0/1280040330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100fd0 0x7f254410d4e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f253c009cd0 tx=0x7f253c02c100 comp rx=0 tx=0).stop 2024-04-22T00:03:53.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 -- 172.21.15.38:0/1280040330 shutdown_connections 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 --2- 172.21.15.38:0/1280040330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100fd0 0x7f254410d4e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 --2- 172.21.15.38:0/1280040330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2544100610 0x7f2544100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 --2- 172.21.15.38:0/1280040330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 0x7f2544106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 -- 172.21.15.38:0/1280040330 >> 172.21.15.38:0/1280040330 conn(0x7f2544077740 msgr2=0x7f2544077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.280+0000 7f254a2c9640 1 -- 172.21.15.38:0/1280040330 shutdown_connections 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.281+0000 7f254a2c9640 1 -- 172.21.15.38:0/1280040330 wait complete. 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.281+0000 7f254a2c9640 1 Processor -- start 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.281+0000 7f254a2c9640 1 -- start start 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2544100fd0 0x7f25441984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 0x7f254419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2544110a50 con 0x7f2544100610 2024-04-22T00:03:53.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25441108d0 con 0x7f2544100fd0 2024-04-22T00:03:53.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f254a2c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2544110bd0 con 0x7f25441066b0 2024-04-22T00:03:53.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35830/0 (socket says 172.21.15.38:35830) 2024-04-22T00:03:53.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 -- 172.21.15.38:0/2993188802 learned_addr learned my addr 172.21.15.38:0/2993188802 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2548866640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 0x7f254419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 -- 172.21.15.38:0/2993188802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 msgr2=0x7f254419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 0x7f254419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 -- 172.21.15.38:0/2993188802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2544100fd0 msgr2=0x7f25441984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2544100fd0 0x7f25441984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.282+0000 7f2543fff640 1 -- 172.21.15.38:0/2993188802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254419cfa0 con 0x7f2544100610 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f2543fff640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f253400d770 tx=0x7f253400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f25417fa640 1 -- 172.21.15.38:0/2993188802 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534017070 con 0x7f2544100610 2024-04-22T00:03:53.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f25417fa640 1 -- 172.21.15.38:0/2993188802 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2534009c30 con 0x7f2544100610 2024-04-22T00:03:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254419d290 con 0x7f2544100610 2024-04-22T00:03:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f25417fa640 1 -- 172.21.15.38:0/2993188802 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534003dd0 con 0x7f2544100610 2024-04-22T00:03:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.283+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25441a4ba0 con 0x7f2544100610 2024-04-22T00:03:53.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.286+0000 7f25417fa640 1 -- 172.21.15.38:0/2993188802 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2534010460 con 0x7f2544100610 2024-04-22T00:03:53.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 Processor -- start 2024-04-22T00:03:53.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 -- start start 2024-04-22T00:03:53.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f25417fa640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 0x7f252c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb5cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1ceab8530 con 0x55d1ceb54020 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d1cea60830 con 0x55d1ceb5d520 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.287+0000 7f254cf8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1ceab44f0 con 0x55d1ceb54940 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35844/0 (socket says 172.21.15.38:35844) 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 -- 172.21.15.38:0/3233965911 learned_addr learned my addr 172.21.15.38:0/3233965911 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f25417fa640 1 -- 172.21.15.38:0/2993188802 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f25340cf0d0 con 0x7f2544100610 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f25437fe640 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb5cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 -- 172.21.15.38:0/3233965911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 msgr2=0x55d1ceb5cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb5cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 -- 172.21.15.38:0/3233965911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 msgr2=0x55d1ceb5f910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2548866640 1 -- 172.21.15.38:0/3233965911 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1cea842c0 con 0x55d1ceb54020 2024-04-22T00:03:53.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f25437fe640 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb5cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:03:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.288+0000 7f2543fff640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 0x7f252c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.289+0000 7f2548866640 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c710 tx=0x7f253c005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc5cb8e1e58dbf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.289+0000 7f252a7fc640 1 -- 172.21.15.38:0/3233965911 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c043020 con 0x55d1ceb54020 2024-04-22T00:03:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.289+0000 7f252a7fc640 1 -- 172.21.15.38:0/3233965911 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f253c003970 con 0x55d1ceb54020 2024-04-22T00:03:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.289+0000 7f252a7fc640 1 -- 172.21.15.38:0/3233965911 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c039040 con 0x55d1ceb54020 2024-04-22T00:03:53.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 -- 172.21.15.38:0/3233965911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 msgr2=0x55d1ceb54400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f2543fff640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 0x7f252c0a5240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f25340cc590 tx=0x7f2534003870 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c710 tx=0x7f253c005f60 comp rx=0 tx=0).stop 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 -- 172.21.15.38:0/3233965911 shutdown_connections 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb5cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3233965911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb54400 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 -- 172.21.15.38:0/3233965911 >> 172.21.15.38:0/3233965911 conn(0x55d1ceb4e750 msgr2=0x55d1ceb4eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 -- 172.21.15.38:0/3233965911 shutdown_connections 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.290+0000 7f254cf8d740 1 -- 172.21.15.38:0/3233965911 wait complete. 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.291+0000 7f254cf8d740 1 -- start start 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.291+0000 7f254cf8d740 1 Processor -- start 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb54020 0x55d1ceb5ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1ceb60300 con 0x55d1ceb54940 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d1cea73f10 con 0x55d1ceb54020 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1ceb60480 con 0x55d1ceb5d520 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f254cf8d740 10 client.0 _ll_register_callbacks cb 0x55d1ce7cbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f2548866640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f25437fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f2548866640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:35734/0 (socket says 172.21.15.38:35734) 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f2548866640 1 -- 172.21.15.38:0/3993614991 learned_addr learned my addr 172.21.15.38:0/3993614991 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.292+0000 7f2548866640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb54020 0x55d1ceb5ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.293+0000 7f25437fe640 1 -- 172.21.15.38:0/3993614991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 msgr2=0x55d1ceb5a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.293+0000 7f25437fe640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.293+0000 7f25437fe640 1 -- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb54020 msgr2=0x55d1ceb5ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.293+0000 7f25437fe640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb54020 0x55d1ceb5ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.293+0000 7f25437fe640 1 -- 172.21.15.38:0/3993614991 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1cea45c80 con 0x55d1ceb54940 2024-04-22T00:03:53.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.296+0000 7f25437fe640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5a260 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2544105c80 tx=0x7f25380098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19eb6fcaf845dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.296+0000 7f25237fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:53.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.297+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253800d770 con 0x55d1ceb54940 2024-04-22T00:03:53.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.297+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c003970 con 0x55d1ceb54940 2024-04-22T00:03:53.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.297+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f253800d910 con 0x55d1ceb54940 2024-04-22T00:03:53.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.298+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2538020050 con 0x55d1ceb54940 2024-04-22T00:03:53.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.298+0000 7f254cf8d740 10 client.16458 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:53.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.298+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1ce7cc3f0 con 0x55d1ceb54940 2024-04-22T00:03:53.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.299+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f253c003970 con 0x55d1ceb54940 2024-04-22T00:03:53.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.299+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f2538015350 con 0x55d1ceb54940 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 10 client.16458 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 20 client.16458 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1ceb6dac0 0x55d1ceb6ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 4 client.16458 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1ceac5a70 con 0x55d1ceb6dac0 2024-04-22T00:03:53.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.300+0000 7f2548866640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1ceb6dac0 0x55d1ceb6ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.302+0000 7f2548866640 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1ceb6dac0 0x55d1ceb6ff40 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbeaa6ea654213d3 server_cookie=32330881201b2f14 in_seq=0 out_seq=0 2024-04-22T00:03:53.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.302+0000 7f25237fe640 10 client.16458 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:53.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.303+0000 7f25237fe640 1 -- 172.21.15.38:0/3993614991 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1ceac5a70 con 0x55d1ceb6dac0 2024-04-22T00:03:53.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.303+0000 7f25237fe640 10 client.16458 handle_command_reply: tid=0 2024-04-22T00:03:53.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.303+0000 7f254cf8d740 1 client.16458 shutdown 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 msgr2=0x55d1ceb5a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5a260 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2544105c80 tx=0x7f25380098e0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 shutdown_connections 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1ceb6dac0 0x55d1ceb6ff40 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5a260 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 --2- 172.21.15.38:0/3993614991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb54020 0x55d1ceb5ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 >> 172.21.15.38:0/3993614991 conn(0x55d1ceb4e750 msgr2=0x55d1ceb504a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f25237fe640 0 client.16458 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:53.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.306+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 shutdown_connections 2024-04-22T00:03:53.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.308+0000 7f254cf8d740 1 -- 172.21.15.38:0/3993614991 wait complete. 2024-04-22T00:03:53.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.308+0000 7f254cf8d740 20 client.16458 trim_cache size 0 max 16384 2024-04-22T00:03:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.342+0000 7f254cf8d740 1 Processor -- start 2024-04-22T00:03:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.344+0000 7f254cf8d740 1 -- start start 2024-04-22T00:03:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.346+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54020 0x55d1ceb691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f254cf8d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25441ac960 con 0x55d1ceb54940 2024-04-22T00:03:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35862/0 (socket says 172.21.15.38:35862) 2024-04-22T00:03:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f254cf8d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25441ae620 con 0x55d1ceb5d520 2024-04-22T00:03:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 -- 172.21.15.38:0/2045751225 learned_addr learned my addr 172.21.15.38:0/2045751225 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25441ad030 con 0x55d1ceb54020 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 -- 172.21.15.38:0/2045751225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54020 msgr2=0x55d1ceb691f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54020 0x55d1ceb691f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 -- 172.21.15.38:0/2045751225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 msgr2=0x55d1ceb5ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.347+0000 7f25437fe640 1 -- 172.21.15.38:0/2045751225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1ce8967d0 con 0x55d1ceb54940 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.348+0000 7f25437fe640 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2538011040 tx=0x7f2538015a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221b35e3c871a48b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.348+0000 7f2529ffb640 1 -- 172.21.15.38:0/2045751225 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538028070 con 0x55d1ceb54940 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.348+0000 7f2529ffb640 1 -- 172.21.15.38:0/2045751225 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2538009e90 con 0x55d1ceb54940 2024-04-22T00:03:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.348+0000 7f2529ffb640 1 -- 172.21.15.38:0/2045751225 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253800dcb0 con 0x55d1ceb54940 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.349+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 msgr2=0x55d1ceb5c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.349+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2538011040 tx=0x7f2538015a00 comp rx=0 tx=0).stop 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 shutdown_connections 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1ceb5ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54940 0x55d1ceb5c990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2045751225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54020 0x55d1ceb691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 >> 172.21.15.38:0/2045751225 conn(0x55d1ceb4e750 msgr2=0x55d1ceb4d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 shutdown_connections 2024-04-22T00:03:53.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.350+0000 7f254cf8d740 1 -- 172.21.15.38:0/2045751225 wait complete. 2024-04-22T00:03:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.351+0000 7f254cf8d740 1 -- start start 2024-04-22T00:03:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.351+0000 7f254cf8d740 1 Processor -- start 2024-04-22T00:03:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb60df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f254cf8d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1cebd20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f2548866640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f254cf8d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25441ade90 con 0x55d1ceb54020 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f2548866640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35870/0 (socket says 172.21.15.38:35870) 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.352+0000 7f2548866640 1 -- 172.21.15.38:0/2545520537 learned_addr learned my addr 172.21.15.38:0/2545520537 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.353+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f25441ad7c0 con 0x55d1ceb5d520 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.353+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25441aa940 con 0x55d1ceb54940 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.353+0000 7f254cf8d740 10 client.0 _ll_register_callbacks cb 0x7f2538006f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.354+0000 7f2548866640 1 -- 172.21.15.38:0/2545520537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 msgr2=0x55d1ceb60df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.354+0000 7f2548866640 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb60df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.354+0000 7f2548866640 1 -- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 msgr2=0x55d1cebd20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.354+0000 7f2548866640 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1cebd20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.354+0000 7f2548866640 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1cebd25f0 con 0x55d1ceb54020 2024-04-22T00:03:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.355+0000 7f2548866640 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c640 tx=0x7f253c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92647fbebe6e0beb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.355+0000 7f25237fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.356+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c043040 con 0x55d1ceb54020 2024-04-22T00:03:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.357+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f253c003b70 con 0x55d1ceb54020 2024-04-22T00:03:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.357+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2538009e90 con 0x55d1ceb54020 2024-04-22T00:03:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.357+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f253c039040 con 0x55d1ceb54020 2024-04-22T00:03:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.357+0000 7f254cf8d740 10 client.16464 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.358+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1cebd38c0 con 0x55d1ceb54020 2024-04-22T00:03:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.358+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f253c009b20 con 0x55d1ceb54020 2024-04-22T00:03:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f253c052c70 con 0x55d1ceb54020 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 10 client.16464 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 20 client.16464 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1cebd3d10 0x55d1cebd6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 4 client.16464 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:03:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.359+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2538009e90 con 0x55d1cebd3d10 2024-04-22T00:03:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.360+0000 7f25437fe640 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1cebd3d10 0x55d1cebd6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.362+0000 7f25437fe640 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1cebd3d10 0x55d1cebd6100 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81630e0d16d50705 server_cookie=51d5dd511813c6f4 in_seq=0 out_seq=0 2024-04-22T00:03:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.362+0000 7f25237fe640 10 client.16464 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.363+0000 7f25237fe640 1 -- 172.21.15.38:0/2545520537 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2538009e90 con 0x55d1cebd3d10 2024-04-22T00:03:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.363+0000 7f25237fe640 10 client.16464 handle_command_reply: tid=0 2024-04-22T00:03:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.363+0000 7f254cf8d740 1 client.16464 shutdown 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 msgr2=0x55d1ceb608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c640 tx=0x7f253c005fd0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 shutdown_connections 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1ceb5d520 0x55d1cebd20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ceb54940 0x55d1ceb60df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1cebd3d10 0x55d1cebd6100 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 --2- 172.21.15.38:0/2545520537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1ceb54020 0x55d1ceb608b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 >> 172.21.15.38:0/2545520537 conn(0x55d1ceb4e750 msgr2=0x55d1ceb4d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f25237fe640 0 client.16464 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 shutdown_connections 2024-04-22T00:03:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.364+0000 7f254cf8d740 1 -- 172.21.15.38:0/2545520537 wait complete. 2024-04-22T00:03:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.365+0000 7f254cf8d740 20 client.16464 trim_cache size 0 max 16384 2024-04-22T00:03:53.358 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:03:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 msgr2=0x7f252c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 0x7f252c0a5240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f25340cc590 tx=0x7f2534003870 comp rx=0 tx=0).stop 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 msgr2=0x7f2544197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f253400d770 tx=0x7f253400dc30 comp rx=0 tx=0).stop 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 shutdown_connections 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f252c0a2d80 0x7f252c0a5240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25441066b0 0x7f254419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2544100fd0 0x7f25441984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 --2- 172.21.15.38:0/2993188802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2544100610 0x7f2544197fb0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.366+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 >> 172.21.15.38:0/2993188802 conn(0x7f2544077740 msgr2=0x7f25440ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.367+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 shutdown_connections 2024-04-22T00:03:53.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:53.367+0000 7f254a2c9640 1 -- 172.21.15.38:0/2993188802 wait complete. 2024-04-22T00:03:54.279 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:54 smithi203 ceph-mon[31431]: pgmap v57: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 482 MiB/s wr, 278 op/s 2024-04-22T00:03:54.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:54 smithi111 ceph-mon[31543]: pgmap v57: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 482 MiB/s wr, 278 op/s 2024-04-22T00:03:54.381 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:03:54.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:03:54.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:03:54.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:54 smithi038 ceph-mon[27972]: pgmap v57: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 482 MiB/s wr, 278 op/s 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 -- 172.21.15.38:0/354260737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 msgr2=0x7f96f0102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 --2- 172.21.15.38:0/354260737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f0102da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009900 tx=0x7f96e002bd60 comp rx=0 tx=0).stop 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 -- 172.21.15.38:0/354260737 shutdown_connections 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 --2- 172.21.15.38:0/354260737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f010bd00 0x7f96f010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 --2- 172.21.15.38:0/354260737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f01032e0 0x7f96f010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 --2- 172.21.15.38:0/354260737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f0102da0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 -- 172.21.15.38:0/354260737 >> 172.21.15.38:0/354260737 conn(0x7f96f0077740 msgr2=0x7f96f0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 -- 172.21.15.38:0/354260737 shutdown_connections 2024-04-22T00:03:54.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.522+0000 7f96f51c8640 1 -- 172.21.15.38:0/354260737 wait complete. 2024-04-22T00:03:54.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.523+0000 7f96f51c8640 1 Processor -- start 2024-04-22T00:03:54.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 -- start start 2024-04-22T00:03:54.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f01032e0 0x7f96f01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 0x7f96f019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96f0111b50 con 0x7f96f010bd00 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01119d0 con 0x7f96f01032e0 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.524+0000 7f96f51c8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96f0111cd0 con 0x7f96f01029c0 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96ef59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 0x7f96f019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96ef59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 0x7f96f019c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35612/0 (socket says 172.21.15.38:35612) 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96ef59e640 1 -- 172.21.15.38:0/1741017145 learned_addr learned my addr 172.21.15.38:0/1741017145 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47144/0 (socket says 172.21.15.38:47144) 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 -- 172.21.15.38:0/1741017145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f01032e0 msgr2=0x7f96f01985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f01032e0 0x7f96f01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 -- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 msgr2=0x7f96f019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 0x7f96f019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 -- 172.21.15.38:0/1741017145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f019d0a0 con 0x7f96f01029c0 2024-04-22T00:03:54.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.525+0000 7f96eed9d640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c2a0 tx=0x7f96e0005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.526+0000 7f96d7fff640 1 -- 172.21.15.38:0/1741017145 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0043020 con 0x7f96f01029c0 2024-04-22T00:03:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.526+0000 7f96d7fff640 1 -- 172.21.15.38:0/1741017145 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e0005000 con 0x7f96f01029c0 2024-04-22T00:03:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.526+0000 7f96d7fff640 1 -- 172.21.15.38:0/1741017145 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0003b60 con 0x7f96f01029c0 2024-04-22T00:03:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.526+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f019d330 con 0x7f96f01029c0 2024-04-22T00:03:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.526+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f01a4c20 con 0x7f96f01029c0 2024-04-22T00:03:54.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 Processor -- start 2024-04-22T00:03:54.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 -- start start 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96d7fff640 1 -- 172.21.15.38:0/1741017145 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f96e0046020 con 0x7f96f01029c0 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fd720 0x563beb1fc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb20c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563beb160170 con 0x563beb1fd720 2024-04-22T00:03:54.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563beb1087c0 con 0x563beb1fca40 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.530+0000 7f96f7da8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563beb15c130 con 0x563beb20a1c0 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33364/0 (socket says 172.21.15.38:33364) 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 -- 172.21.15.38:0/882819527 learned_addr learned my addr 172.21.15.38:0/882819527 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96d7fff640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 0x7f96d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 -- 172.21.15.38:0/882819527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 msgr2=0x563beb20c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb20c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 -- 172.21.15.38:0/882819527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fd720 msgr2=0x563beb1fc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fd720 0x563beb1fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.531+0000 7f96eed9d640 1 -- 172.21.15.38:0/882819527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563beb119e60 con 0x563beb1fca40 2024-04-22T00:03:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.532+0000 7f96eed9d640 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96e00eea90 tx=0x7f96e00e8480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbc75924ea411536 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.533+0000 7f96d7fff640 1 -- 172.21.15.38:0/1741017145 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96e00f1600 con 0x7f96f01029c0 2024-04-22T00:03:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.533+0000 7f96ef59e640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 0x7f96d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.533+0000 7f96d57fa640 1 -- 172.21.15.38:0/882819527 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e00bd430 con 0x563beb1fca40 2024-04-22T00:03:54.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.533+0000 7f96d57fa640 1 -- 172.21.15.38:0/882819527 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e0088170 con 0x563beb1fca40 2024-04-22T00:03:54.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.533+0000 7f96d57fa640 1 -- 172.21.15.38:0/882819527 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0086b50 con 0x563beb1fca40 2024-04-22T00:03:54.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 -- 172.21.15.38:0/882819527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 msgr2=0x563beb1fcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96ef59e640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 0x7f96d80a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f96e8005c40 tx=0x7f96e8005bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96e00eea90 tx=0x7f96e00e8480 comp rx=0 tx=0).stop 2024-04-22T00:03:54.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 -- 172.21.15.38:0/882819527 shutdown_connections 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb20c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb1fcec0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 --2- 172.21.15.38:0/882819527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fd720 0x563beb1fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.534+0000 7f96f7da8740 1 -- 172.21.15.38:0/882819527 >> 172.21.15.38:0/882819527 conn(0x563beb1c34f0 msgr2=0x563beb1f7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.535+0000 7f96f7da8740 1 -- 172.21.15.38:0/882819527 shutdown_connections 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.535+0000 7f96f7da8740 1 -- 172.21.15.38:0/882819527 wait complete. 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.535+0000 7f96f7da8740 1 -- start start 2024-04-22T00:03:54.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.535+0000 7f96f7da8740 1 Processor -- start 2024-04-22T00:03:54.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb20a1c0 0x563beb203c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563beb1fde00 con 0x563beb1fca40 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563beb11bd40 con 0x563beb20a1c0 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563beb1fdf80 con 0x563beb1fd720 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96f7da8740 10 client.0 _ll_register_callbacks cb 0x563beb157820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.536+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35618/0 (socket says 172.21.15.38:35618) 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 -- 172.21.15.38:0/3923975101 learned_addr learned my addr 172.21.15.38:0/3923975101 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 -- 172.21.15.38:0/3923975101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 msgr2=0x563beb203730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 -- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb20a1c0 msgr2=0x563beb203c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb20a1c0 0x563beb203c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 -- 172.21.15.38:0/3923975101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563beaf3c7e0 con 0x563beb1fca40 2024-04-22T00:03:54.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.537+0000 7f96eed9d640 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c340 tx=0x7f96e002c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9145f1f2c909a9cd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.538+0000 7f96d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.538+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e00bd250 con 0x563beb1fca40 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.538+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96e00d1ad0 con 0x563beb1fca40 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.538+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563beb134450 con 0x563beb1fca40 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.539+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f96e0086b50 con 0x563beb1fca40 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.539+0000 7f96f7da8740 10 client.16473 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:54.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.540+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563beaf9cf70 con 0x563beb1fca40 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.540+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563beb134450 con 0x563beb1fca40 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.540+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f96e00b9f60 con 0x563beb1fca40 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 10 client.16473 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 20 client.16473 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb21e2a0 0x563beb220740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 4 client.16473 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563beb16d6b0 con 0x563beb21e2a0 2024-04-22T00:03:54.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.541+0000 7f96ee59c640 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb21e2a0 0x563beb220740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.542+0000 7f96ee59c640 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb21e2a0 0x563beb220740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a45c84bcdad33c2 server_cookie=844d1fe7b5a102ab in_seq=0 out_seq=0 2024-04-22T00:03:54.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.542+0000 7f96d2ff5640 10 client.16473 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.543+0000 7f96d2ff5640 1 -- 172.21.15.38:0/3923975101 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563beb16d6b0 con 0x563beb21e2a0 2024-04-22T00:03:54.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.543+0000 7f96d2ff5640 10 client.16473 handle_command_reply: tid=0 2024-04-22T00:03:54.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.543+0000 7f96f7da8740 1 client.16473 shutdown 2024-04-22T00:03:54.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.544+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 msgr2=0x563beb2031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.544+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c340 tx=0x7f96e002c580 comp rx=0 tx=0).stop 2024-04-22T00:03:54.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.544+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 shutdown_connections 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb20a1c0 0x563beb203c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb21e2a0 0x563beb220740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 --2- 172.21.15.38:0/3923975101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb2031f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 >> 172.21.15.38:0/3923975101 conn(0x563beb1c34f0 msgr2=0x563beb1f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96d2ff5640 0 client.16473 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.545+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 shutdown_connections 2024-04-22T00:03:54.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.546+0000 7f96f7da8740 1 -- 172.21.15.38:0/3923975101 wait complete. 2024-04-22T00:03:54.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.546+0000 7f96f7da8740 20 client.16473 trim_cache size 0 max 16384 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 Processor -- start 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 -- start start 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb202350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 0x563beb1f6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01aca30 con 0x563beb20a1c0 2024-04-22T00:03:54.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.588+0000 7f96f7da8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01ae6f0 con 0x563beb1fca40 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96f7da8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01ad100 con 0x563beb1fd720 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb202350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb202350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33380/0 (socket says 172.21.15.38:33380) 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96eed9d640 1 -- 172.21.15.38:0/4075749032 learned_addr learned my addr 172.21.15.38:0/4075749032 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ee59c640 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 0x563beb1f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 -- 172.21.15.38:0/4075749032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 msgr2=0x563beb203a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 -- 172.21.15.38:0/4075749032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 msgr2=0x563beb202350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb202350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.589+0000 7f96ef59e640 1 -- 172.21.15.38:0/4075749032 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563beae34170 con 0x563beb20a1c0 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.590+0000 7f96ef59e640 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 0x563beb1f6370 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f96e8004a80 tx=0x7f96e8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e574fb763081cf9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.590+0000 7f96d4ff9640 1 -- 172.21.15.38:0/4075749032 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8026600 con 0x563beb20a1c0 2024-04-22T00:03:54.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.590+0000 7f96d4ff9640 1 -- 172.21.15.38:0/4075749032 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e8026c80 con 0x563beb20a1c0 2024-04-22T00:03:54.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.590+0000 7f96d4ff9640 1 -- 172.21.15.38:0/4075749032 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e801ed30 con 0x563beb20a1c0 2024-04-22T00:03:54.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 -- 172.21.15.38:0/4075749032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 msgr2=0x563beb1f6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 0x563beb1f6370 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f96e8004a80 tx=0x7f96e8014000 comp rx=0 tx=0).stop 2024-04-22T00:03:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 -- 172.21.15.38:0/4075749032 shutdown_connections 2024-04-22T00:03:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb20a1c0 0x563beb1f6370 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb1fd720 0x563beb203a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 --2- 172.21.15.38:0/4075749032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fca40 0x563beb202350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 -- 172.21.15.38:0/4075749032 >> 172.21.15.38:0/4075749032 conn(0x563beb1c34f0 msgr2=0x563beb1f5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 -- 172.21.15.38:0/4075749032 shutdown_connections 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.592+0000 7f96f7da8740 1 -- 172.21.15.38:0/4075749032 wait complete. 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.593+0000 7f96f7da8740 1 -- start start 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.593+0000 7f96f7da8740 1 Processor -- start 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.593+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fd720 0x563beb21a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96eed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35632/0 (socket says 172.21.15.38:35632) 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96eed9d640 1 -- 172.21.15.38:0/933279703 learned_addr learned my addr 172.21.15.38:0/933279703 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb21aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01adf60 con 0x563beb1fca40 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01ad890 con 0x563beb1fd720 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96f01aaa10 con 0x563beb20a1c0 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96ee59c640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fd720 0x563beb21a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96f7da8740 10 client.0 _ll_register_callbacks cb 0x7f96e00a6f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.594+0000 7f96ef59e640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb21aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 -- 172.21.15.38:0/933279703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 msgr2=0x563beb21aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb21aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 -- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fd720 msgr2=0x563beb21a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fd720 0x563beb21a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563beb21b000 con 0x563beb1fca40 2024-04-22T00:03:54.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e8026c80 con 0x563beb1fca40 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96eed9d640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f96e00bd880 tx=0x7f96e0009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa97722a6f3692f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.595+0000 7f96d2ff5640 10 client.16479 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.596+0000 7f96d2ff5640 1 -- 172.21.15.38:0/933279703 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e00bd430 con 0x563beb1fca40 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.596+0000 7f96d2ff5640 1 -- 172.21.15.38:0/933279703 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96e00edf80 con 0x563beb1fca40 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.596+0000 7f96d2ff5640 1 -- 172.21.15.38:0/933279703 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f96e00bd250 con 0x563beb1fca40 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.597+0000 7f96f7da8740 10 client.16479 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.597+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=32}) -- 0x563beb21b000 con 0x563beb1fca40 2024-04-22T00:03:54.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.597+0000 7f96d2ff5640 1 -- 172.21.15.38:0/933279703 <== mon.0 v2:172.21.15.38:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f96e00f1250 con 0x563beb1fca40 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 10 client.16479 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 20 client.16479 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb284940 0x563beb286d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 4 client.16479 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563beb21b340 con 0x563beb284940 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.598+0000 7f96ee59c640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb284940 0x563beb286d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.600+0000 7f96ee59c640 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb284940 0x563beb286d30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=714c8151e64c142a server_cookie=9f728e793607cab8 in_seq=0 out_seq=0 2024-04-22T00:03:54.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.600+0000 7f96d2ff5640 10 client.16479 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.601+0000 7f96d2ff5640 1 -- 172.21.15.38:0/933279703 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563beb21b340 con 0x563beb284940 2024-04-22T00:03:54.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.601+0000 7f96d2ff5640 10 client.16479 handle_command_reply: tid=0 2024-04-22T00:03:54.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.601+0000 7f96f7da8740 1 client.16479 shutdown 2024-04-22T00:03:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.601+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 msgr2=0x563beb200b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.602+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f96e00bd880 tx=0x7f96e0009880 comp rx=0 tx=0).stop 2024-04-22T00:03:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.602+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 shutdown_connections 2024-04-22T00:03:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.602+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563beb20a1c0 0x563beb21aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.602+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563beb1fd720 0x563beb21a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.602+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563beb284940 0x563beb286d30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.603+0000 7f96d2ff5640 0 client.16479 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.603+0000 7f96f7da8740 1 --2- 172.21.15.38:0/933279703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563beb1fca40 0x563beb200b30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.603+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 >> 172.21.15.38:0/933279703 conn(0x563beb1c34f0 msgr2=0x563beb1f64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.603+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 shutdown_connections 2024-04-22T00:03:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.603+0000 7f96f7da8740 1 -- 172.21.15.38:0/933279703 wait complete. 2024-04-22T00:03:54.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.604+0000 7f96f7da8740 20 client.16479 trim_cache size 0 max 16384 2024-04-22T00:03:54.598 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:54.598 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:03:54.598 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "55e8c45f-62c0-4c88-83ae-67c569f2636a", 2024-04-22T00:03:54.598 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:03:54.598 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.606+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 msgr2=0x7f96d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.606+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 0x7f96d80a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f96e8005c40 tx=0x7f96e8005bd0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 msgr2=0x7f96f01980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c2a0 tx=0x7f96e0005e80 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 shutdown_connections 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96d80a2d00 0x7f96d80a51c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96f010bd00 0x7f96f019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96f01032e0 0x7f96f01985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 --2- 172.21.15.38:0/1741017145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96f01029c0 0x7f96f01980b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.607+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 >> 172.21.15.38:0/1741017145 conn(0x7f96f0077740 msgr2=0x7f96f00fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.608+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 shutdown_connections 2024-04-22T00:03:54.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.608+0000 7f96f51c8640 1 -- 172.21.15.38:0/1741017145 wait complete. 2024-04-22T00:03:54.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:03:54.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:03:54.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.871+0000 7fcf5f266640 1 -- 172.21.15.38:0/1880067609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5811af20 msgr2=0x7fcf5811d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.871+0000 7fcf5f266640 1 --2- 172.21.15.38:0/1880067609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5811af20 0x7fcf5811d310 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009cd0 tx=0x7fcf5002c120 comp rx=0 tx=0).stop 2024-04-22T00:03:54.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.871+0000 7fcf5f266640 1 -- 172.21.15.38:0/1880067609 shutdown_connections 2024-04-22T00:03:54.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.872+0000 7fcf5f266640 1 --2- 172.21.15.38:0/1880067609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5811af20 0x7fcf5811d310 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.872+0000 7fcf5f266640 1 --2- 172.21.15.38:0/1880067609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5810dfc0 0x7fcf5810e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.872+0000 7fcf5f266640 1 --2- 172.21.15.38:0/1880067609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf5810ff90 0x7fcf5810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.872+0000 7fcf5f266640 1 -- 172.21.15.38:0/1880067609 >> 172.21.15.38:0/1880067609 conn(0x7fcf5806c850 msgr2=0x7fcf5806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.875+0000 7fcf5f266640 1 -- 172.21.15.38:0/1880067609 shutdown_connections 2024-04-22T00:03:54.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.879+0000 7fcf5f266640 1 -- 172.21.15.38:0/1880067609 wait complete. 2024-04-22T00:03:54.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.880+0000 7fcf5f266640 1 Processor -- start 2024-04-22T00:03:54.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.881+0000 7fcf5f266640 1 -- start start 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf5810dfc0 0x7fcf5811a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5810ff90 0x7fcf581147c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58111150 con 0x7fcf5810ff90 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58110fd0 con 0x7fcf5810dfc0 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.882+0000 7fcf5f266640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf581112d0 con 0x7fcf5811af20 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.883+0000 7fcf5c801640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5810ff90 0x7fcf581147c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.883+0000 7fcf5d803640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.883+0000 7fcf5d803640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47192/0 (socket says 172.21.15.38:47192) 2024-04-22T00:03:54.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.883+0000 7fcf5d803640 1 -- 172.21.15.38:0/2437892603 learned_addr learned my addr 172.21.15.38:0/2437892603 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.884+0000 7fcf5d803640 1 -- 172.21.15.38:0/2437892603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf5810dfc0 msgr2=0x7fcf5811a700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.884+0000 7fcf5d803640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf5810dfc0 0x7fcf5811a700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.884+0000 7fcf5d803640 1 -- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5810ff90 msgr2=0x7fcf581147c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.884+0000 7fcf5d803640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5810ff90 0x7fcf581147c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.884+0000 7fcf5d803640 1 -- 172.21.15.38:0/2437892603 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf581154d0 con 0x7fcf5811af20 2024-04-22T00:03:54.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.885+0000 7fcf5d803640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf50005e30 tx=0x7fcf50034d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.889+0000 7fcf4e7fc640 1 -- 172.21.15.38:0/2437892603 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50041020 con 0x7fcf5811af20 2024-04-22T00:03:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.890+0000 7fcf4e7fc640 1 -- 172.21.15.38:0/2437892603 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf5003d3f0 con 0x7fcf5811af20 2024-04-22T00:03:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.890+0000 7fcf4e7fc640 1 -- 172.21.15.38:0/2437892603 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50039040 con 0x7fcf5811af20 2024-04-22T00:03:54.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.891+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf58125f00 con 0x7fcf5811af20 2024-04-22T00:03:54.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.891+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf581263a0 con 0x7fcf5811af20 2024-04-22T00:03:54.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.897+0000 7fcf61e49740 1 Processor -- start 2024-04-22T00:03:54.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.897+0000 7fcf61e49740 1 -- start start 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124b7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124c2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56261241b560 con 0x5626124b7850 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626123c3730 con 0x5626124c0430 2024-04-22T00:03:54.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.899+0000 7fcf61e49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562612417520 con 0x5626124b6f30 2024-04-22T00:03:54.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.902+0000 7fcf5d002640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.902+0000 7fcf5d002640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35662/0 (socket says 172.21.15.38:35662) 2024-04-22T00:03:54.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.902+0000 7fcf5d002640 1 -- 172.21.15.38:0/3484422842 learned_addr learned my addr 172.21.15.38:0/3484422842 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.902+0000 7fcf5c801640 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.903+0000 7fcf5c801640 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124c2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.903+0000 7fcf4e7fc640 1 -- 172.21.15.38:0/2437892603 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcf50044020 con 0x7fcf5811af20 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.904+0000 7fcf5d002640 1 -- 172.21.15.38:0/3484422842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 msgr2=0x5626124b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.904+0000 7fcf5d002640 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.904+0000 7fcf5d002640 1 -- 172.21.15.38:0/3484422842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 msgr2=0x5626124c2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.904+0000 7fcf5d002640 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124c2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.904+0000 7fcf5d002640 1 -- 172.21.15.38:0/3484422842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626121f97d0 con 0x5626124b7850 2024-04-22T00:03:54.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.905+0000 7fcf5d002640 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ba80 tx=0x7fcf5400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f55be4daf5db25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.906+0000 7fcf4e7fc640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 0x7fcf440a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.906+0000 7fcf5c801640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 0x7fcf440a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.906+0000 7fcf437fe640 1 -- 172.21.15.38:0/3484422842 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54016070 con 0x5626124b7850 2024-04-22T00:03:54.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.906+0000 7fcf437fe640 1 -- 172.21.15.38:0/3484422842 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf54005500 con 0x5626124b7850 2024-04-22T00:03:54.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.906+0000 7fcf437fe640 1 -- 172.21.15.38:0/3484422842 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54010460 con 0x5626124b7850 2024-04-22T00:03:54.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.907+0000 7fcf4e7fc640 1 -- 172.21.15.38:0/2437892603 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcf500f0e80 con 0x7fcf5811af20 2024-04-22T00:03:54.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf5c801640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 0x7fcf440a53c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5626124b27f0 tx=0x7fcf48008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- 172.21.15.38:0/3484422842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 msgr2=0x5626124bfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ba80 tx=0x7fcf5400bf40 comp rx=0 tx=0).stop 2024-04-22T00:03:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- 172.21.15.38:0/3484422842 shutdown_connections 2024-04-22T00:03:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124c2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bfef0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 --2- 172.21.15.38:0/3484422842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- 172.21.15.38:0/3484422842 >> 172.21.15.38:0/3484422842 conn(0x5626124b1660 msgr2=0x5626124b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- 172.21.15.38:0/3484422842 shutdown_connections 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- 172.21.15.38:0/3484422842 wait complete. 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 -- start start 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.908+0000 7fcf61e49740 1 Processor -- start 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b6f30 0x5626124bd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124bdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5626124c3220 con 0x5626124b7850 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5626123d6e00 con 0x5626124b6f30 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5626124c33a0 con 0x5626124c0430 2024-04-22T00:03:54.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf61e49740 10 client.0 _ll_register_callbacks cb 0x562612411eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf5c801640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf5c801640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35672/0 (socket says 172.21.15.38:35672) 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.909+0000 7fcf5c801640 1 -- 172.21.15.38:0/2514106508 learned_addr learned my addr 172.21.15.38:0/2514106508 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5d002640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b6f30 0x5626124bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 -- 172.21.15.38:0/2514106508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 msgr2=0x5626124bdf10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124bdf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 -- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b6f30 msgr2=0x5626124bd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b6f30 0x5626124bd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 -- 172.21.15.38:0/2514106508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56261212f3f0 con 0x5626124b7850 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf5c801640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcf48016710 tx=0x7fcf480090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c6acde53e63c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48027020 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcf48015500 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.910+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf54005500 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcf48005970 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf61e49740 10 client.16491 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5626120ef0f0 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4801c030 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcf480060b0 con 0x5626124b7850 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf61e49740 10 client.16491 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf61e49740 10 client.16491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.911+0000 7fcf61e49740 20 client.16491 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:54.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.912+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5626124d0930 0x5626124d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.912+0000 7fcf61e49740 4 client.16491 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:03:54.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.912+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcf54005500 con 0x5626124d0930 2024-04-22T00:03:54.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.914+0000 7fcf5d002640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5626124d0930 0x5626124d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.919+0000 7fcf5d002640 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5626124d0930 0x5626124d2db0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3bd3ea64bc61530 server_cookie=53ac7be8f34568c7 in_seq=0 out_seq=0 2024-04-22T00:03:54.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.919+0000 7fcf40ff9640 10 client.16491 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.921+0000 7fcf40ff9640 1 -- 172.21.15.38:0/2514106508 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fcf54005500 con 0x5626124d0930 2024-04-22T00:03:54.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.921+0000 7fcf40ff9640 10 client.16491 handle_command_reply: tid=0 2024-04-22T00:03:54.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.921+0000 7fcf61e49740 1 client.16491 shutdown 2024-04-22T00:03:54.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 msgr2=0x5626124bd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcf48016710 tx=0x7fcf480090e0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 shutdown_connections 2024-04-22T00:03:54.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124bdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5626124d0930 0x5626124d2db0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124bd9d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 --2- 172.21.15.38:0/2514106508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b6f30 0x5626124bd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 >> 172.21.15.38:0/2514106508 conn(0x5626124b1660 msgr2=0x5626124b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf40ff9640 0 client.16491 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 shutdown_connections 2024-04-22T00:03:54.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 1 -- 172.21.15.38:0/2514106508 wait complete. 2024-04-22T00:03:54.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.922+0000 7fcf61e49740 20 client.16491 trim_cache size 0 max 16384 2024-04-22T00:03:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 Processor -- start 2024-04-22T00:03:54.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 -- start start 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58130450 con 0x5626124b7850 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf581d8e40 con 0x5626124c0430 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.955+0000 7fcf61e49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58130b20 con 0x5626124b6f30 2024-04-22T00:03:54.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.956+0000 7fcf5d803640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.956+0000 7fcf5d803640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33436/0 (socket says 172.21.15.38:33436) 2024-04-22T00:03:54.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.956+0000 7fcf5d803640 1 -- 172.21.15.38:0/791077997 learned_addr learned my addr 172.21.15.38:0/791077997 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.956+0000 7fcf5d002640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 -- 172.21.15.38:0/791077997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 msgr2=0x5626124bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 -- 172.21.15.38:0/791077997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 msgr2=0x5626124b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5c801640 1 -- 172.21.15.38:0/791077997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626124b6250 con 0x5626124b7850 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.957+0000 7fcf5d002640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:03:54.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.958+0000 7fcf5c801640 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124cc110 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcf480066b0 tx=0x7fcf48009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312f28d15985b6cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.958+0000 7fcf42ffd640 1 -- 172.21.15.38:0/791077997 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48027070 con 0x5626124b7850 2024-04-22T00:03:54.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.958+0000 7fcf42ffd640 1 -- 172.21.15.38:0/791077997 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4801c040 con 0x5626124b7850 2024-04-22T00:03:54.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.958+0000 7fcf42ffd640 1 -- 172.21.15.38:0/791077997 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48023ab0 con 0x5626124b7850 2024-04-22T00:03:54.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.959+0000 7fcf61e49740 1 -- 172.21.15.38:0/791077997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 msgr2=0x5626124cc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.959+0000 7fcf61e49740 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124cc110 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcf480066b0 tx=0x7fcf48009b70 comp rx=0 tx=0).stop 2024-04-22T00:03:54.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 -- 172.21.15.38:0/791077997 shutdown_connections 2024-04-22T00:03:54.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124c0430 0x5626124b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b7850 0x5626124cc110 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 --2- 172.21.15.38:0/791077997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124b6f30 0x5626124bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 -- 172.21.15.38:0/791077997 >> 172.21.15.38:0/791077997 conn(0x5626124b1660 msgr2=0x5626124afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 -- 172.21.15.38:0/791077997 shutdown_connections 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 -- 172.21.15.38:0/791077997 wait complete. 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.960+0000 7fcf61e49740 1 -- start start 2024-04-22T00:03:54.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.961+0000 7fcf61e49740 1 Processor -- start 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.961+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b7850 0x5626124ba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf581d86b0 con 0x5626124b6f30 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcf581d7fe0 con 0x5626124b7850 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5812e430 con 0x5626124c0430 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf61e49740 10 client.0 _ll_register_callbacks cb 0x7fcf48009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:03:54.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35696/0 (socket says 172.21.15.38:35696) 2024-04-22T00:03:54.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 -- 172.21.15.38:0/1263529110 learned_addr learned my addr 172.21.15.38:0/1263529110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 -- 172.21.15.38:0/1263529110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 msgr2=0x5626124baa70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124baa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 -- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b7850 msgr2=0x5626124ba530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b7850 0x5626124ba530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 -- 172.21.15.38:0/1263529110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562612535f30 con 0x5626124b6f30 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf5d002640 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ba80 tx=0x7fcf54004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c81705bd9ebea43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.962+0000 7fcf40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54016040 con 0x5626124b6f30 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcf54004940 con 0x5626124b6f30 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf4801c040 con 0x5626124b6f30 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcf54032050 con 0x5626124b6f30 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf61e49740 10 client.16497 fetch_fsmap learned FSMap version 32 2024-04-22T00:03:54.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.963+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562612537170 con 0x5626124b6f30 2024-04-22T00:03:54.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.964+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4801c040 con 0x5626124b6f30 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.965+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcf540244c0 con 0x5626124b6f30 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 10 client.16497 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 10 client.16497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 20 client.16497 populate_metadata read hostname 'smithi038' 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562612537670 0x562612539a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 4 client.16497 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:03:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.966+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562612428aa0 con 0x562612537670 2024-04-22T00:03:54.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.968+0000 7fcf5c801640 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562612537670 0x562612539a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:03:54.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.970+0000 7fcf5c801640 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562612537670 0x562612539a90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240a7ae2c7ea153d server_cookie=c4ad5cb410904243 in_seq=0 out_seq=0 2024-04-22T00:03:54.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.970+0000 7fcf40ff9640 10 client.16497 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.970+0000 7fcf40ff9640 1 -- 172.21.15.38:0/1263529110 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562612428aa0 con 0x562612537670 2024-04-22T00:03:54.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.970+0000 7fcf40ff9640 10 client.16497 handle_command_reply: tid=0 2024-04-22T00:03:54.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.971+0000 7fcf61e49740 1 client.16497 shutdown 2024-04-22T00:03:54.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.971+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 msgr2=0x5626124b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.971+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ba80 tx=0x7fcf54004b60 comp rx=0 tx=0).stop 2024-04-22T00:03:54.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.971+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 shutdown_connections 2024-04-22T00:03:54.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.971+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5626124c0430 0x5626124baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5626124b7850 0x5626124ba530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562612537670 0x562612539a90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 --2- 172.21.15.38:0/1263529110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626124b6f30 0x5626124b9ff0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 >> 172.21.15.38:0/1263529110 conn(0x5626124b1660 msgr2=0x5626124bdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf40ff9640 0 client.16497 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 shutdown_connections 2024-04-22T00:03:54.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.972+0000 7fcf61e49740 1 -- 172.21.15.38:0/1263529110 wait complete. 2024-04-22T00:03:54.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.973+0000 7fcf61e49740 20 client.16497 trim_cache size 0 max 16384 2024-04-22T00:03:54.966 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:03:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:03:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "55e8c45f-62c0-4c88-83ae-67c569f2636a": { 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "55e8c45f-62c0-4c88-83ae-67c569f2636a", 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:03:54.967 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:03:54.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 msgr2=0x7fcf440a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 0x7fcf440a53c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5626124b27f0 tx=0x7fcf48008040 comp rx=0 tx=0).stop 2024-04-22T00:03:54.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 msgr2=0x7fcf58114d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf50005e30 tx=0x7fcf50034d00 comp rx=0 tx=0).stop 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 shutdown_connections 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcf440a2f00 0x7fcf440a53c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf5811af20 0x7fcf58114d00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf5810ff90 0x7fcf581147c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 --2- 172.21.15.38:0/2437892603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf5810dfc0 0x7fcf5811a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 >> 172.21.15.38:0/2437892603 conn(0x7fcf5806c850 msgr2=0x7fcf5811c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 shutdown_connections 2024-04-22T00:03:54.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:03:54.974+0000 7fcf5f266640 1 -- 172.21.15.38:0/2437892603 wait complete. 2024-04-22T00:03:54.981 INFO:tasks.cephfs.filesystem:scrub status for tag:55e8c45f-62c0-4c88-83ae-67c569f2636a - {'path': '/', 'tag': '55e8c45f-62c0-4c88-83ae-67c569f2636a', 'options': 'recursive,force'} 2024-04-22T00:03:55.029 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:03:54 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:03:54.718+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...3,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v356 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v2 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v356>,<0x10000000007/.git v3088>,<0x10000000006/ffsb v2523>,<0x10000000005/tmp v59>,<0x100000001fb/client.0 v53>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v67>,<0x10000000001/sv_1 v87>,<0x10000000000/qa v73>,<0x1/volumes v68>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v2 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:03:55.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:55 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:03:55.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:55 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:55 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:55.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:55 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:03:55.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:55 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:55 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:03:55.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:55 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:55 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:55.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:55 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:03:55.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:55 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:55 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:03:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:55 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:55 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:55 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:03:55.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:55 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:55.920 INFO:tasks.workunit.client.0.smithi038.stdout:fs setup took 13 secs 2024-04-22T00:03:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:56 smithi111 ceph-mon[31543]: pgmap v58: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 482 MiB/s wr, 270 op/s 2024-04-22T00:03:56.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:56 smithi038 ceph-mon[27972]: pgmap v58: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 482 MiB/s wr, 270 op/s 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:03:56.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:56 smithi203 ceph-mon[31431]: pgmap v58: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 482 MiB/s wr, 270 op/s 2024-04-22T00:03:58.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:03:58 smithi111 ceph-mon[31543]: pgmap v59: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 826 MiB/s wr, 427 op/s 2024-04-22T00:03:58.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:03:58 smithi038 ceph-mon[27972]: pgmap v59: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 826 MiB/s wr, 427 op/s 2024-04-22T00:03:58.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:03:58 smithi203 ceph-mon[31431]: pgmap v59: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 826 MiB/s wr, 427 op/s 2024-04-22T00:03:59.571 INFO:tasks.workunit.client.0.smithi038.stdout:Syncing()...3 sec 2024-04-22T00:03:59.572 INFO:tasks.workunit.client.0.smithi038.stdout:Starting Actual Benchmark At: Mon Apr 22 00:03:59 2024 2024-04-22T00:03:59.572 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:04:00.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:00 smithi111 ceph-mon[31543]: pgmap v60: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 701 MiB/s wr, 325 op/s 2024-04-22T00:04:00.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:00 smithi038 ceph-mon[27972]: pgmap v60: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 701 MiB/s wr, 325 op/s 2024-04-22T00:04:00.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:00 smithi203 ceph-mon[31431]: pgmap v60: 129 pgs: 129 active+clean; 9.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 701 MiB/s wr, 325 op/s 2024-04-22T00:04:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.689+0000 7f5f9faf4640 1 -- 172.21.15.38:0/4286602924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9810ff30 msgr2=0x7f5f98108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.689+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/4286602924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9810ff30 0x7f5f98108740 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009930 tx=0x7f5f9402bcf0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.690+0000 7f5f9faf4640 1 -- 172.21.15.38:0/4286602924 shutdown_connections 2024-04-22T00:04:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.690+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/4286602924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f9811d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.690+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/4286602924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f98110310 0x7f5f98108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.690+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/4286602924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f9810ff30 0x7f5f98108740 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.690+0000 7f5f9faf4640 1 -- 172.21.15.38:0/4286602924 >> 172.21.15.38:0/4286602924 conn(0x7f5f9806c810 msgr2=0x7f5f9806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.691+0000 7f5f9faf4640 1 -- 172.21.15.38:0/4286602924 shutdown_connections 2024-04-22T00:04:00.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.691+0000 7f5f9faf4640 1 -- 172.21.15.38:0/4286602924 wait complete. 2024-04-22T00:04:00.684 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.692+0000 7f5f9faf4640 1 Processor -- start 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.692+0000 7f5f9faf4640 1 -- start start 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 0x7f5f981a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f981a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 0x7f5f981abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98120a90 con 0x7f5f981ab790 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98120910 con 0x7f5f9810ff30 2024-04-22T00:04:00.685 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9faf4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98120c10 con 0x7f5f9811af20 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9d08f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f981a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9d08f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f981a7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47240/0 (socket says 172.21.15.38:47240) 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9d08f640 1 -- 172.21.15.38:0/2530017007 learned_addr learned my addr 172.21.15.38:0/2530017007 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.693+0000 7f5f9d890640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 0x7f5f981a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.694+0000 7f5f9d890640 1 -- 172.21.15.38:0/2530017007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 msgr2=0x7f5f981a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.686 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.694+0000 7f5f9d890640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f981a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.694+0000 7f5f9d890640 1 -- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 msgr2=0x7f5f981abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.694+0000 7f5f9e091640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 0x7f5f981abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.694+0000 7f5f9d890640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 0x7f5f981abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.695+0000 7f5f9d890640 1 -- 172.21.15.38:0/2530017007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f981ac280 con 0x7f5f9810ff30 2024-04-22T00:04:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.695+0000 7f5f9e091640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 0x7f5f981abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:04:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.695+0000 7f5f9d890640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 0x7f5f981a6ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009850 tx=0x7f5f940042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.696+0000 7f5f8effd640 1 -- 172.21.15.38:0/2530017007 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94042020 con 0x7f5f9810ff30 2024-04-22T00:04:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.696+0000 7f5f8effd640 1 -- 172.21.15.38:0/2530017007 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f940044d0 con 0x7f5f9810ff30 2024-04-22T00:04:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.697+0000 7f5f8effd640 1 -- 172.21.15.38:0/2530017007 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9403e8c0 con 0x7f5f9810ff30 2024-04-22T00:04:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.700+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f981ac510 con 0x7f5f9810ff30 2024-04-22T00:04:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.700+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f98110940 con 0x7f5f9810ff30 2024-04-22T00:04:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.707+0000 7f5fa27bb740 1 Processor -- start 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.708+0000 7f5fa27bb740 1 -- start start 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c54f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c55440 0x560235c5dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560235bb8b70 con 0x560235c5e210 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560235b61470 con 0x560235c54b20 2024-04-22T00:04:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.711+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560235bb4b30 con 0x560235c55440 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.713+0000 7f5f8effd640 1 -- 172.21.15.38:0/2530017007 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5f94045020 con 0x7f5f9810ff30 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.714+0000 7f5f8effd640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 0x7f5f840a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.714+0000 7f5f8effd640 1 -- 172.21.15.38:0/2530017007 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5f940f0af0 con 0x7f5f9810ff30 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.715+0000 7f5f9d08f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.715+0000 7f5f9d08f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35728/0 (socket says 172.21.15.38:35728) 2024-04-22T00:04:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.715+0000 7f5f9d08f640 1 -- 172.21.15.38:0/1086384758 learned_addr learned my addr 172.21.15.38:0/1086384758 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.715+0000 7f5f9e091640 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c54f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.716+0000 7f5f9d890640 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c55440 0x560235c5dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9e091640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 0x7f5f840a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9d08f640 1 -- 172.21.15.38:0/1086384758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c55440 msgr2=0x560235c5dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c55440 0x560235c5dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9d08f640 1 -- 172.21.15.38:0/1086384758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 msgr2=0x560235c54f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c54f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.717+0000 7f5f9d08f640 1 -- 172.21.15.38:0/1086384758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602358cd210 con 0x560235c5e210 2024-04-22T00:04:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.718+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009ac0 tx=0x7f5f88009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e07b31b1ee23f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.719+0000 7f5f9e091640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 0x7f5f840a52e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5f9810d9e0 tx=0x7f5f90004a60 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.719+0000 7f5f83fff640 1 -- 172.21.15.38:0/1086384758 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88017070 con 0x560235c5e210 2024-04-22T00:04:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.719+0000 7f5f83fff640 1 -- 172.21.15.38:0/1086384758 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f8800eba0 con 0x560235c5e210 2024-04-22T00:04:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.719+0000 7f5f83fff640 1 -- 172.21.15.38:0/1086384758 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88012630 con 0x560235c5e210 2024-04-22T00:04:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.722+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1086384758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 msgr2=0x560235c60600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.722+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009ac0 tx=0x7f5f88009f80 comp rx=0 tx=0).stop 2024-04-22T00:04:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.722+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1086384758 shutdown_connections 2024-04-22T00:04:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.722+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c5e210 0x560235c60600 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.722+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c55440 0x560235c5dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.723+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1086384758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c54f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.723+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1086384758 >> 172.21.15.38:0/1086384758 conn(0x560235c4f440 msgr2=0x560235c4f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.724+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1086384758 shutdown_connections 2024-04-22T00:04:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.724+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1086384758 wait complete. 2024-04-22T00:04:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.724+0000 7f5fa27bb740 1 -- start start 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.725+0000 7f5fa27bb740 1 Processor -- start 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c5b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c5b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560235c60e40 con 0x560235c55440 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560235b74a10 con 0x560235c54b20 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560235c60fc0 con 0x560235c5e210 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5f9d890640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c5b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5f9d890640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c5b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33482/0 (socket says 172.21.15.38:33482) 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5f9d890640 1 -- 172.21.15.38:0/1685140422 learned_addr learned my addr 172.21.15.38:0/1685140422 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5fa27bb740 10 client.0 _ll_register_callbacks cb 0x560235bafb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.726+0000 7f5f9e091640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c5b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9e091640 1 -- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 msgr2=0x560235c5b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9e091640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c5b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9e091640 1 -- 172.21.15.38:0/1685140422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 msgr2=0x560235c5b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9e091640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c5b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.727+0000 7f5f9e091640 1 -- 172.21.15.38:0/1685140422 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560235999620 con 0x560235c5e210 2024-04-22T00:04:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.728+0000 7f5f9e091640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f90010f40 tx=0x7f5f90004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=896b8adf412899f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.728+0000 7f5f817fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:04:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.729+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90022070 con 0x560235c5e210 2024-04-22T00:04:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.730+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f8800eba0 con 0x560235c5e210 2024-04-22T00:04:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.730+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5f90003c10 con 0x560235c5e210 2024-04-22T00:04:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.731+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5f9002d050 con 0x560235c5e210 2024-04-22T00:04:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.734+0000 7f5fa27bb740 10 client.25798 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.734+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560235c6cf60 con 0x560235c5e210 2024-04-22T00:04:00.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.736+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f8800eba0 con 0x560235c5e210 2024-04-22T00:04:00.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.736+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f5f9001d070 con 0x560235c5e210 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 10 client.25798 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 20 client.25798 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235c6e470 0x560235c708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 4 client.25798 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.738+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560235bc60b0 con 0x560235c6e470 2024-04-22T00:04:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.744+0000 7f5f9d890640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235c6e470 0x560235c708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.748+0000 7f5f9d890640 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235c6e470 0x560235c708c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b760c05385d6a4 server_cookie=ae1b23c8b72ab13c in_seq=0 out_seq=0 2024-04-22T00:04:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.748+0000 7f5f817fa640 10 client.25798 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.750+0000 7f5f817fa640 1 -- 172.21.15.38:0/1685140422 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560235bc60b0 con 0x560235c6e470 2024-04-22T00:04:00.742 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.750+0000 7f5f817fa640 10 client.25798 handle_command_reply: tid=0 2024-04-22T00:04:00.743 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.750+0000 7f5fa27bb740 1 client.25798 shutdown 2024-04-22T00:04:00.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.753+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 msgr2=0x560235c5bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.753+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f90010f40 tx=0x7f5f90004ad0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 shutdown_connections 2024-04-22T00:04:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235c6e470 0x560235c708c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c5b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/1685140422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c5b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.754+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 >> 172.21.15.38:0/1685140422 conn(0x560235c4f440 msgr2=0x560235c51190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.755+0000 7f5f817fa640 0 client.25798 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.755+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 shutdown_connections 2024-04-22T00:04:00.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.756+0000 7f5fa27bb740 1 -- 172.21.15.38:0/1685140422 wait complete. 2024-04-22T00:04:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.756+0000 7f5fa27bb740 20 client.25798 trim_cache size 0 max 16384 2024-04-22T00:04:00.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.786+0000 7f5fa27bb740 1 Processor -- start 2024-04-22T00:04:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.787+0000 7f5fa27bb740 1 -- start start 2024-04-22T00:04:00.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c69d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c59af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f981c5ea0 con 0x560235c55440 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f981c7b60 con 0x560235c54b20 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f981c6570 con 0x560235c5e210 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.788+0000 7f5f9d08f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c59af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.789+0000 7f5f9e091640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.789+0000 7f5f9e091640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47282/0 (socket says 172.21.15.38:47282) 2024-04-22T00:04:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.789+0000 7f5f9e091640 1 -- 172.21.15.38:0/520481946 learned_addr learned my addr 172.21.15.38:0/520481946 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 -- 172.21.15.38:0/520481946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 msgr2=0x560235c69d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c69d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 -- 172.21.15.38:0/520481946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 msgr2=0x560235c59af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c59af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 -- 172.21.15.38:0/520481946 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560235c5a1d0 con 0x560235c5e210 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.791+0000 7f5f9e091640 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000f80 tx=0x7f5f90010ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5af443b74979760 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.792+0000 7f5f837fe640 1 -- 172.21.15.38:0/520481946 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90036070 con 0x560235c5e210 2024-04-22T00:04:00.785 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.792+0000 7f5f837fe640 1 -- 172.21.15.38:0/520481946 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f90018070 con 0x560235c5e210 2024-04-22T00:04:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.792+0000 7f5f837fe640 1 -- 172.21.15.38:0/520481946 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90022040 con 0x560235c5e210 2024-04-22T00:04:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.793+0000 7f5fa27bb740 1 -- 172.21.15.38:0/520481946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 msgr2=0x560235c5bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:00.786 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.793+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f90000f80 tx=0x7f5f90010ba0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.787 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.794+0000 7f5fa27bb740 1 -- 172.21.15.38:0/520481946 shutdown_connections 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.794+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c5bac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.794+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c55440 0x560235c59af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.794+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/520481946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c54b20 0x560235c69d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.794+0000 7f5fa27bb740 1 -- 172.21.15.38:0/520481946 >> 172.21.15.38:0/520481946 conn(0x560235c4f440 msgr2=0x560235c51120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.795+0000 7f5fa27bb740 1 -- 172.21.15.38:0/520481946 shutdown_connections 2024-04-22T00:04:00.788 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.795+0000 7f5fa27bb740 1 -- 172.21.15.38:0/520481946 wait complete. 2024-04-22T00:04:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.796+0000 7f5fa27bb740 1 -- start start 2024-04-22T00:04:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.796+0000 7f5fa27bb740 1 Processor -- start 2024-04-22T00:04:00.789 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.797+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.797+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c55440 0x560235c58490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.797+0000 7f5fa27bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c589d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f981c73d0 con 0x560235c54b20 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f981c6d00 con 0x560235c55440 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5fa27bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9804f500 con 0x560235c5e210 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5fa27bb740 10 client.0 _ll_register_callbacks cb 0x7f5f90007ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5f9d890640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5f9d890640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35738/0 (socket says 172.21.15.38:35738) 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5f9d890640 1 -- 172.21.15.38:0/4279542449 learned_addr learned my addr 172.21.15.38:0/4279542449 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.798+0000 7f5f9e091640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.799+0000 7f5f9d890640 1 -- 172.21.15.38:0/4279542449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 msgr2=0x560235c589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.799+0000 7f5f9d890640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.799+0000 7f5f9d890640 1 -- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c55440 msgr2=0x560235c58490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.799+0000 7f5f9d890640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c55440 0x560235c58490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:00.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.799+0000 7f5f9d890640 1 -- 172.21.15.38:0/4279542449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560235cd3b40 con 0x560235c54b20 2024-04-22T00:04:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.801+0000 7f5f9d890640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5f94004f50 tx=0x7f5f940bb290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded82327847ad4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:00.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.801+0000 7f5f817fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.802+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f940db060 con 0x560235c54b20 2024-04-22T00:04:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.802+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5f940ee120 con 0x560235c54b20 2024-04-22T00:04:00.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.802+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f90018070 con 0x560235c54b20 2024-04-22T00:04:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.803+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5f940cd050 con 0x560235c54b20 2024-04-22T00:04:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.803+0000 7f5fa27bb740 10 client.16512 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:00.795 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.803+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560235cd4eb0 con 0x560235c54b20 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.804+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f90018070 con 0x560235c54b20 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.804+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f5f940c5560 con 0x560235c54b20 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.804+0000 7f5fa27bb740 10 client.16512 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.804+0000 7f5fa27bb740 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.805+0000 7f5fa27bb740 20 client.16512 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.805+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235cd55c0 0x560235cd79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.805+0000 7f5fa27bb740 4 client.16512 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:04:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.805+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560235bc0400 con 0x560235cd55c0 2024-04-22T00:04:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.812+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235cd55c0 0x560235cd79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.814+0000 7f5f9d08f640 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235cd55c0 0x560235cd79e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8201da4f87421a61 server_cookie=5bcc6f18a5f34cfd in_seq=0 out_seq=0 2024-04-22T00:04:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:00.814+0000 7f5f817fa640 10 client.16512 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:02.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:02 smithi111 ceph-mon[31543]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 740 MiB/s wr, 506 op/s 2024-04-22T00:04:02.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:02 smithi038 ceph-mon[27972]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 740 MiB/s wr, 506 op/s 2024-04-22T00:04:02.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:02 smithi203 ceph-mon[31431]: pgmap v61: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 740 MiB/s wr, 506 op/s 2024-04-22T00:04:04.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:04 smithi111 ceph-mon[31543]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 761 MiB/s wr, 1.01k op/s 2024-04-22T00:04:04.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:04 smithi038 ceph-mon[27972]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 761 MiB/s wr, 1.01k op/s 2024-04-22T00:04:04.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:04 smithi203 ceph-mon[31431]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 761 MiB/s wr, 1.01k op/s 2024-04-22T00:04:06.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:06 smithi111 ceph-mon[31543]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 589 MiB/s wr, 947 op/s 2024-04-22T00:04:06.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:06 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:06.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:06 smithi038 ceph-mon[27972]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 589 MiB/s wr, 947 op/s 2024-04-22T00:04:06.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:06 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:06.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:06 smithi203 ceph-mon[31431]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 589 MiB/s wr, 947 op/s 2024-04-22T00:04:06.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:06 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:07.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:07.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:07.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:07.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:07.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:07.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5f817fa640 1 -- 172.21.15.38:0/4279542449 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560235bc0400 con 0x560235cd55c0 2024-04-22T00:04:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5f817fa640 10 client.16512 handle_command_reply: tid=0 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5fa27bb740 1 client.16512 shutdown 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 msgr2=0x560235c57f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f5f94004f50 tx=0x7f5f940bb290 comp rx=0 tx=0).stop 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.731+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 shutdown_connections 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560235c5e210 0x560235c589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560235c55440 0x560235c58490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560235cd55c0 0x560235cd79e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 --2- 172.21.15.38:0/4279542449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560235c54b20 0x560235c57f50 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 >> 172.21.15.38:0/4279542449 conn(0x560235c4f440 msgr2=0x560235c4dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5f817fa640 0 client.16512 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 shutdown_connections 2024-04-22T00:04:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 1 -- 172.21.15.38:0/4279542449 wait complete. 2024-04-22T00:04:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.732+0000 7f5fa27bb740 20 client.16512 trim_cache size 0 max 16384 2024-04-22T00:04:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:04:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:04:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:04:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 msgr2=0x7f5f840a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 0x7f5f840a52e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5f9810d9e0 tx=0x7f5f90004a60 comp rx=0 tx=0).stop 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 msgr2=0x7f5f981a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 0x7f5f981a6ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009850 tx=0x7f5f940042e0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 shutdown_connections 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f840a2e20 0x7f5f840a52e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f981ab790 0x7f5f981abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f9811af20 0x7f5f981a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 --2- 172.21.15.38:0/2530017007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9810ff30 0x7f5f981a6ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 >> 172.21.15.38:0/2530017007 conn(0x7f5f9806c810 msgr2=0x7f5f9810dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 shutdown_connections 2024-04-22T00:04:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:07.733+0000 7f5f9faf4640 1 -- 172.21.15.38:0/2530017007 wait complete. 2024-04-22T00:04:08.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:08 smithi111 ceph-mon[31543]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 628 MiB/s wr, 6.64k op/s 2024-04-22T00:04:08.430 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:08 smithi203 ceph-mon[31431]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 628 MiB/s wr, 6.64k op/s 2024-04-22T00:04:08.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:08 smithi038 ceph-mon[27972]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 628 MiB/s wr, 6.64k op/s 2024-04-22T00:04:10.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:10 smithi038 ceph-mon[27972]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 284 MiB/s wr, 6.48k op/s 2024-04-22T00:04:10.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:10 smithi203 ceph-mon[31431]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 284 MiB/s wr, 6.48k op/s 2024-04-22T00:04:10.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:10 smithi111 ceph-mon[31543]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 284 MiB/s wr, 6.48k op/s 2024-04-22T00:04:12.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:12 smithi038 ceph-mon[27972]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 289 MiB/s wr, 7.24k op/s 2024-04-22T00:04:12.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:12 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:04:12.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:12 smithi203 ceph-mon[31431]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 289 MiB/s wr, 7.24k op/s 2024-04-22T00:04:12.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:12 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:04:12.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:12 smithi111 ceph-mon[31543]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 289 MiB/s wr, 7.24k op/s 2024-04-22T00:04:12.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:12 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:04:13.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:13 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:04:13.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:13 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:04:13.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:13 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:04:13.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:13 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:04:13.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:13 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:04:13.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:13 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:04:13.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:13 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:04:13.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:13 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:04:13.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:13 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:04:14.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:14 smithi038 ceph-mon[27972]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 287 MiB/s wr, 12.11k op/s 2024-04-22T00:04:14.477 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:14 smithi111 ceph-mon[31543]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 287 MiB/s wr, 12.11k op/s 2024-04-22T00:04:14.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:14 smithi203 ceph-mon[31431]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 287 MiB/s wr, 12.11k op/s 2024-04-22T00:04:16.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:16 smithi038 ceph-mon[27972]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 81 MiB/s wr, 11.50k op/s 2024-04-22T00:04:16.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:16 smithi203 ceph-mon[31431]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 81 MiB/s wr, 11.50k op/s 2024-04-22T00:04:16.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:16 smithi111 ceph-mon[31543]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 81 MiB/s wr, 11.50k op/s 2024-04-22T00:04:18.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:18 smithi038 ceph-mon[27972]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 120 MiB/s wr, 16.90k op/s 2024-04-22T00:04:18.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:18 smithi203 ceph-mon[31431]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 120 MiB/s wr, 16.90k op/s 2024-04-22T00:04:18.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:18 smithi111 ceph-mon[31543]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 120 MiB/s wr, 16.90k op/s 2024-04-22T00:04:20.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:20 smithi038 ceph-mon[27972]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:20.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:20 smithi203 ceph-mon[31431]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:20.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:20 smithi111 ceph-mon[31543]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.921+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3603150252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a8108740 msgr2=0x7f71a8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.921+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3603150252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a8108740 0x7f71a8108b20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f719c009900 tx=0x7f719c02bce0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3603150252 shutdown_connections 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3603150252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 0x7f71a8116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3603150252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 0x7f71a8114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3603150252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a8108740 0x7f71a8108b20 secure :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f719c009900 tx=0x7f719c02bce0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3603150252 >> 172.21.15.38:0/3603150252 conn(0x7f71a806c810 msgr2=0x7f71a806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.923+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3603150252 shutdown_connections 2024-04-22T00:04:20.917 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.924+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3603150252 wait complete. 2024-04-22T00:04:20.918 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.925+0000 7f71ad9e5640 1 Processor -- start 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.925+0000 7f71ad9e5640 1 -- start start 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 0x7f71a8111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 0x7f71a8111d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f71a810be80 con 0x7f71a81122a0 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f71a810bd00 con 0x7f71a8114890 2024-04-22T00:04:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.926+0000 7f71ad9e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f71a810c000 con 0x7f71a8109060 2024-04-22T00:04:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.927+0000 7f71a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.927+0000 7f71a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40298/0 (socket says 172.21.15.38:40298) 2024-04-22T00:04:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.927+0000 7f71a77fe640 1 -- 172.21.15.38:0/3752712214 learned_addr learned my addr 172.21.15.38:0/3752712214 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.927+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 0x7f71a8111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.927+0000 7f71a67fc640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 0x7f71a8111d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 -- 172.21.15.38:0/3752712214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 msgr2=0x7f71a8111820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 0x7f71a8111820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 -- 172.21.15.38:0/3752712214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 msgr2=0x7f71a8111d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 0x7f71a8111d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 -- 172.21.15.38:0/3752712214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a81b36b0 con 0x7f71a81122a0 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7198009fd0 tx=0x7f719800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.928+0000 7f71ac9e3640 1 -- 172.21.15.38:0/3752712214 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198011830 con 0x7f71a81122a0 2024-04-22T00:04:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.929+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a81b39a0 con 0x7f71a81122a0 2024-04-22T00:04:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.929+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a81b3ef0 con 0x7f71a81122a0 2024-04-22T00:04:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.929+0000 7f71ac9e3640 1 -- 172.21.15.38:0/3752712214 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7198016070 con 0x7f71a81122a0 2024-04-22T00:04:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.929+0000 7f71ac9e3640 1 -- 172.21.15.38:0/3752712214 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198010380 con 0x7f71a81122a0 2024-04-22T00:04:20.926 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.933+0000 7f71ac9e3640 1 -- 172.21.15.38:0/3752712214 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7198010520 con 0x7f71a81122a0 2024-04-22T00:04:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.936+0000 7f71ac9e3640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 0x7f71900a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.936+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 0x7f71900a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.936+0000 7f71ac9e3640 1 -- 172.21.15.38:0/3752712214 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f71980cec70 con 0x7f71a81122a0 2024-04-22T00:04:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.937+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 0x7f71900a5310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f719c009230 tx=0x7f719c005a20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.939+0000 7f71b05d4740 1 Processor -- start 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 -- start start 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d4cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d55cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d58600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0cb0b70 con 0x55f5f0d4d440 2024-04-22T00:04:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0c59470 con 0x55f5f0d56210 2024-04-22T00:04:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71b05d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0cacb30 con 0x55f5f0d4cb20 2024-04-22T00:04:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.940+0000 7f71a67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d55cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d4cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d4cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48646/0 (socket says 172.21.15.38:48646) 2024-04-22T00:04:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a6ffd640 1 -- 172.21.15.38:0/369522772 learned_addr learned my addr 172.21.15.38:0/369522772 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a77fe640 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d58600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a67fc640 1 -- 172.21.15.38:0/369522772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 msgr2=0x55f5f0d4cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a67fc640 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d4cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a67fc640 1 -- 172.21.15.38:0/369522772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 msgr2=0x55f5f0d58600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a67fc640 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d58600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.941+0000 7f71a67fc640 1 -- 172.21.15.38:0/369522772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5f09c5210 con 0x55f5f0d4d440 2024-04-22T00:04:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.942+0000 7f71a67fc640 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d55cd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f71a8113020 tx=0x7f71a0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc7ce3ffa4e175a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.942+0000 7f718dffb640 1 -- 172.21.15.38:0/369522772 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0016070 con 0x55f5f0d4d440 2024-04-22T00:04:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.943+0000 7f718dffb640 1 -- 172.21.15.38:0/369522772 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a0004d20 con 0x55f5f0d4d440 2024-04-22T00:04:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.943+0000 7f718dffb640 1 -- 172.21.15.38:0/369522772 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0012910 con 0x55f5f0d4d440 2024-04-22T00:04:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.944+0000 7f71b05d4740 1 -- 172.21.15.38:0/369522772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 msgr2=0x55f5f0d55cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.944+0000 7f71b05d4740 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d55cd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f71a8113020 tx=0x7f71a0010740 comp rx=0 tx=0).stop 2024-04-22T00:04:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.945+0000 7f71b05d4740 1 -- 172.21.15.38:0/369522772 shutdown_connections 2024-04-22T00:04:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.945+0000 7f71b05d4740 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d58600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.945+0000 7f71b05d4740 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d55cd0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.945+0000 7f71b05d4740 1 --2- 172.21.15.38:0/369522772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d4cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.945+0000 7f71b05d4740 1 -- 172.21.15.38:0/369522772 >> 172.21.15.38:0/369522772 conn(0x55f5f0d47440 msgr2=0x55f5f0d47820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.946+0000 7f71b05d4740 1 -- 172.21.15.38:0/369522772 shutdown_connections 2024-04-22T00:04:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.946+0000 7f71b05d4740 1 -- 172.21.15.38:0/369522772 wait complete. 2024-04-22T00:04:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.946+0000 7f71b05d4740 1 -- start start 2024-04-22T00:04:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.947+0000 7f71b05d4740 1 Processor -- start 2024-04-22T00:04:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d53cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71b05d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0d58e40 con 0x55f5f0d4d440 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71b05d4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0c6ca10 con 0x55f5f0d56210 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48660/0 (socket says 172.21.15.38:48660) 2024-04-22T00:04:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.948+0000 7f71a6ffd640 1 -- 172.21.15.38:0/3043918906 learned_addr learned my addr 172.21.15.38:0/3043918906 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f5f0d58fc0 con 0x55f5f0d4cb20 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d53cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71b05d4740 10 client.0 _ll_register_callbacks cb 0x55f5f0ca7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a67fc640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 -- 172.21.15.38:0/3043918906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 msgr2=0x55f5f0d53270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 -- 172.21.15.38:0/3043918906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 msgr2=0x55f5f0d537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.949+0000 7f71a77fe640 1 -- 172.21.15.38:0/3043918906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5f0a91620 con 0x55f5f0d56210 2024-04-22T00:04:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.950+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:04:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.951+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d53cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f71980cc160 tx=0x7f71980047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe9d88b23680b268 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.951+0000 7f7186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.951+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719809f310 con 0x55f5f0d56210 2024-04-22T00:04:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.952+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f71980cc300 con 0x55f5f0d56210 2024-04-22T00:04:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.952+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71a0004d20 con 0x55f5f0d56210 2024-04-22T00:04:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.952+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f71980a3bd0 con 0x55f5f0d56210 2024-04-22T00:04:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.952+0000 7f71b05d4740 10 client.25940 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.952+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f5f0d64f60 con 0x55f5f0d56210 2024-04-22T00:04:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.954+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71980975d0 con 0x55f5f0d56210 2024-04-22T00:04:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.954+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f719809a4c0 con 0x55f5f0d56210 2024-04-22T00:04:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.955+0000 7f71b05d4740 10 client.25940 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.955+0000 7f71b05d4740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.955+0000 7f71b05d4740 20 client.25940 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.955+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0d66a70 0x55f5f0d68eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:20.948 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.955+0000 7f71b05d4740 4 client.25940 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.956+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f71a0004d20 con 0x55f5f0d66a70 2024-04-22T00:04:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.957+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0d66a70 0x55f5f0d68eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:20.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.971+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0d66a70 0x55f5f0d68eb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73efd743be7dc4d3 server_cookie=7c9a5f718378735d in_seq=0 out_seq=0 2024-04-22T00:04:20.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.971+0000 7f7186ffd640 10 client.25940 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.974+0000 7f7186ffd640 1 -- 172.21.15.38:0/3043918906 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f71a0004d20 con 0x55f5f0d66a70 2024-04-22T00:04:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.974+0000 7f7186ffd640 10 client.25940 handle_command_reply: tid=0 2024-04-22T00:04:20.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.974+0000 7f71b05d4740 1 client.25940 shutdown 2024-04-22T00:04:20.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.978+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 msgr2=0x55f5f0d53cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:20.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.978+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d53cf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f71980cc160 tx=0x7f71980047d0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 shutdown_connections 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d56210 0x55f5f0d53cf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0d66a70 0x55f5f0d68eb0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d4d440 0x55f5f0d537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3043918906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d53270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 >> 172.21.15.38:0/3043918906 conn(0x55f5f0d47440 msgr2=0x55f5f0d49190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.979+0000 7f7186ffd640 0 client.25940 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.987+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 shutdown_connections 2024-04-22T00:04:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.987+0000 7f71b05d4740 1 -- 172.21.15.38:0/3043918906 wait complete. 2024-04-22T00:04:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:20.987+0000 7f71b05d4740 20 client.25940 trim_cache size 0 max 16384 2024-04-22T00:04:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 Processor -- start 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 -- start start 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0d53940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f719400fd20 con 0x55f5f0d56210 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f71940119e0 con 0x55f5f0d4d440 2024-04-22T00:04:21.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.019+0000 7f71b05d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f71940103f0 con 0x55f5f0d4cb20 2024-04-22T00:04:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.020+0000 7f71a67fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.020+0000 7f71a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.021+0000 7f71a67fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60666/0 (socket says 172.21.15.38:60666) 2024-04-22T00:04:21.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.021+0000 7f71a67fc640 1 -- 172.21.15.38:0/2989349889 learned_addr learned my addr 172.21.15.38:0/2989349889 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:21.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a77fe640 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0d53940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:21.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a67fc640 1 -- 172.21.15.38:0/2989349889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 msgr2=0x55f5f0d559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:21.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a67fc640 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a67fc640 1 -- 172.21.15.38:0/2989349889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 msgr2=0x55f5f0d53940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a67fc640 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0d53940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.022+0000 7f71a67fc640 1 -- 172.21.15.38:0/2989349889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5f0d46c40 con 0x55f5f0d4d440 2024-04-22T00:04:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.024+0000 7f71a67fc640 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71a00124c0 tx=0x7f71a00124f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5708817102b39d89 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.029+0000 7f718d7fa640 1 -- 172.21.15.38:0/2989349889 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0029070 con 0x55f5f0d4d440 2024-04-22T00:04:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.029+0000 7f718d7fa640 1 -- 172.21.15.38:0/2989349889 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a00099e0 con 0x55f5f0d4d440 2024-04-22T00:04:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.029+0000 7f718d7fa640 1 -- 172.21.15.38:0/2989349889 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0016050 con 0x55f5f0d4d440 2024-04-22T00:04:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.030+0000 7f71b05d4740 1 -- 172.21.15.38:0/2989349889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 msgr2=0x55f5f0d61d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.030+0000 7f71b05d4740 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f71a00124c0 tx=0x7f71a00124f0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.031+0000 7f71b05d4740 1 -- 172.21.15.38:0/2989349889 shutdown_connections 2024-04-22T00:04:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.031+0000 7f71b05d4740 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0d53940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.031+0000 7f71b05d4740 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4d440 0x55f5f0d61d00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.031+0000 7f71b05d4740 1 --2- 172.21.15.38:0/2989349889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.031+0000 7f71b05d4740 1 -- 172.21.15.38:0/2989349889 >> 172.21.15.38:0/2989349889 conn(0x55f5f0d47440 msgr2=0x55f5f0d53230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:21.027 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.035+0000 7f71b05d4740 1 -- 172.21.15.38:0/2989349889 shutdown_connections 2024-04-22T00:04:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.036+0000 7f71b05d4740 1 -- 172.21.15.38:0/2989349889 wait complete. 2024-04-22T00:04:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.037+0000 7f71b05d4740 1 -- start start 2024-04-22T00:04:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.037+0000 7f71b05d4740 1 Processor -- start 2024-04-22T00:04:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d5e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4d440 0x55f5f0d5e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7194011250 con 0x55f5f0d56210 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7194010b80 con 0x55f5f0d4cb20 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f719400dd00 con 0x55f5f0d4d440 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.039+0000 7f71b05d4740 10 client.0 _ll_register_callbacks cb 0x7f719809ab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40338/0 (socket says 172.21.15.38:40338) 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 -- 172.21.15.38:0/3216715506 learned_addr learned my addr 172.21.15.38:0/3216715506 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 -- 172.21.15.38:0/3216715506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4d440 msgr2=0x55f5f0d5e750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4d440 0x55f5f0d5e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 -- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4cb20 msgr2=0x55f5f0d5e210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d5e210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.040+0000 7f71a77fe640 1 -- 172.21.15.38:0/3216715506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5f0dc8cc0 con 0x55f5f0d56210 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.041+0000 7f71a77fe640 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f71980973c0 tx=0x7f719809ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5761b6d742d08106 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.041+0000 7f7186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.043+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719809a8c0 con 0x55f5f0d56210 2024-04-22T00:04:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.043+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f719809a3b0 con 0x55f5f0d56210 2024-04-22T00:04:21.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.043+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71a00099e0 con 0x55f5f0d56210 2024-04-22T00:04:21.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.045+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f71980cc450 con 0x55f5f0d56210 2024-04-22T00:04:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.046+0000 7f71b05d4740 10 client.16530 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:21.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.046+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f5f0dde040 con 0x55f5f0d56210 2024-04-22T00:04:21.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.047+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a00099e0 con 0x55f5f0d56210 2024-04-22T00:04:21.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f71980b9ba0 con 0x55f5f0d56210 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 10 client.16530 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 10 client.16530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 20 client.16530 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0dde6d0 0x55f5f0de0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 4 client.16530 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.049+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f5f0cbe0b0 con 0x55f5f0dde6d0 2024-04-22T00:04:21.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.050+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0dde6d0 0x55f5f0de0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.055+0000 7f7186ffd640 10 client.16530 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:21.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:21.055+0000 7f71a6ffd640 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0dde6d0 0x55f5f0de0b10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e475326002ab6c3a server_cookie=b10f6ddd93b0dd56 in_seq=0 out_seq=0 2024-04-22T00:04:22.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:22 smithi038 ceph-mon[27972]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:22.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:22 smithi203 ceph-mon[31431]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:22.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:22 smithi111 ceph-mon[31543]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 MiB/s wr, 11.20k op/s 2024-04-22T00:04:24.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:24 smithi038 ceph-mon[27972]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 113 MiB/s wr, 15.79k op/s 2024-04-22T00:04:24.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:24 smithi203 ceph-mon[31431]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 113 MiB/s wr, 15.79k op/s 2024-04-22T00:04:24.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:24 smithi111 ceph-mon[31543]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 113 MiB/s wr, 15.79k op/s 2024-04-22T00:04:24.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:04:24.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:04:25.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.184+0000 7f6d59a78640 1 -- 172.21.15.38:0/3710990588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 msgr2=0x7f6d5406a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:25.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.184+0000 7f6d59a78640 1 --2- 172.21.15.38:0/3710990588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d5406a700 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009900 tx=0x7f6d4802bc80 comp rx=0 tx=0).stop 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 -- 172.21.15.38:0/3710990588 shutdown_connections 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 --2- 172.21.15.38:0/3710990588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54104210 0x7f6d541046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 --2- 172.21.15.38:0/3710990588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d5406a700 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 --2- 172.21.15.38:0/3710990588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54106710 0x7f6d54069d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 -- 172.21.15.38:0/3710990588 >> 172.21.15.38:0/3710990588 conn(0x7f6d54077620 msgr2=0x7f6d54077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 -- 172.21.15.38:0/3710990588 shutdown_connections 2024-04-22T00:04:25.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.188+0000 7f6d59a78640 1 -- 172.21.15.38:0/3710990588 wait complete. 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.189+0000 7f6d59a78640 1 Processor -- start 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.189+0000 7f6d59a78640 1 -- start start 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 0x7f6d54115020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54106710 0x7f6d5410dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541181f0 con 0x7f6d5406a2a0 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54118070 con 0x7f6d54104210 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.190+0000 7f6d59a78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54118370 con 0x7f6d54106710 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46714/0 (socket says 172.21.15.38:46714) 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d537fe640 1 -- 172.21.15.38:0/1233505971 learned_addr learned my addr 172.21.15.38:0/1233505971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54106710 0x7f6d5410dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 0x7f6d54115020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 -- 172.21.15.38:0/1233505971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54106710 msgr2=0x7f6d5410dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:25.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54106710 0x7f6d5410dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 -- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 msgr2=0x7f6d54114ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:25.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.191+0000 7f6d52ffd640 1 -- 172.21.15.38:0/1233505971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5410e470 con 0x7f6d54104210 2024-04-22T00:04:25.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.192+0000 7f6d537fe640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:25.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.194+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 0x7f6d54115020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009850 tx=0x7f6d4802c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:25.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.195+0000 7f6d50ff9640 1 -- 172.21.15.38:0/1233505971 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48041020 con 0x7f6d54104210 2024-04-22T00:04:25.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.195+0000 7f6d50ff9640 1 -- 172.21.15.38:0/1233505971 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4802cd50 con 0x7f6d54104210 2024-04-22T00:04:25.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.196+0000 7f6d50ff9640 1 -- 172.21.15.38:0/1233505971 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4803d6b0 con 0x7f6d54104210 2024-04-22T00:04:25.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.197+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5410e700 con 0x7f6d54104210 2024-04-22T00:04:25.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.197+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5410e9f0 con 0x7f6d54104210 2024-04-22T00:04:25.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 Processor -- start 2024-04-22T00:04:25.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 -- start start 2024-04-22T00:04:25.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x5594370fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437109820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559437062560 con 0x5594370fe850 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55943700a730 con 0x559437107430 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55943705e520 con 0x5594370fdf30 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46718/0 (socket says 172.21.15.38:46718) 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.201+0000 7f6d537fe640 1 -- 172.21.15.38:0/588989698 learned_addr learned my addr 172.21.15.38:0/588989698 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d53fff640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x5594370fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d537fe640 1 -- 172.21.15.38:0/588989698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 msgr2=0x5594370fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d537fe640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x5594370fe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d537fe640 1 -- 172.21.15.38:0/588989698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 msgr2=0x559437109820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437109820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d537fe640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437109820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d537fe640 1 -- 172.21.15.38:0/588989698 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559436e407d0 con 0x5594370fe850 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d53fff640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x5594370fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.202+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.203+0000 7f6d537fe640 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c5c0 tx=0x7f6d4c009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef7f1df14741a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.203+0000 7f6d39ffb640 1 -- 172.21.15.38:0/588989698 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c018020 con 0x5594370fe850 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.203+0000 7f6d39ffb640 1 -- 172.21.15.38:0/588989698 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4c014d70 con 0x5594370fe850 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.204+0000 7f6d50ff9640 1 -- 172.21.15.38:0/1233505971 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6d48044020 con 0x7f6d54104210 2024-04-22T00:04:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.204+0000 7f6d39ffb640 1 -- 172.21.15.38:0/588989698 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c013e20 con 0x5594370fe850 2024-04-22T00:04:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.204+0000 7f6d5c65c740 1 -- 172.21.15.38:0/588989698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 msgr2=0x559437106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.204+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00c5c0 tx=0x7f6d4c009770 comp rx=0 tx=0).stop 2024-04-22T00:04:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.205+0000 7f6d50ff9640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 0x7f6d3c0a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.205+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 0x7f6d3c0a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.205+0000 7f6d50ff9640 1 -- 172.21.15.38:0/1233505971 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6d480f0dd0 con 0x7f6d54104210 2024-04-22T00:04:25.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 0x7f6d3c0a5270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6d44006fd0 tx=0x7f6d44008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:25.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d5c65c740 1 -- 172.21.15.38:0/588989698 shutdown_connections 2024-04-22T00:04:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x559437106ef0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/588989698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x5594370fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.206+0000 7f6d5c65c740 1 -- 172.21.15.38:0/588989698 >> 172.21.15.38:0/588989698 conn(0x5594370f8660 msgr2=0x5594370f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.207+0000 7f6d5c65c740 1 -- 172.21.15.38:0/588989698 shutdown_connections 2024-04-22T00:04:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.209+0000 7f6d5c65c740 1 -- 172.21.15.38:0/588989698 wait complete. 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.210+0000 7f6d5c65c740 1 -- start start 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.210+0000 7f6d5c65c740 1 Processor -- start 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x559437104860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437104da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55943710a2c0 con 0x5594370fdf30 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55943701de00 con 0x559437107430 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55943710a440 con 0x5594370fe850 2024-04-22T00:04:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.211+0000 7f6d5c65c740 10 client.0 _ll_register_callbacks cb 0x559437058eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:25.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.213+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.213+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46726/0 (socket says 172.21.15.38:46726) 2024-04-22T00:04:25.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.213+0000 7f6d537fe640 1 -- 172.21.15.38:0/3162776805 learned_addr learned my addr 172.21.15.38:0/3162776805 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d53fff640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437104da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x559437104860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d537fe640 1 -- 172.21.15.38:0/3162776805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 msgr2=0x559437104860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d537fe640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x559437104860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d537fe640 1 -- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 msgr2=0x559437104da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d537fe640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437104da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:25.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.214+0000 7f6d537fe640 1 -- 172.21.15.38:0/3162776805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559436d763f0 con 0x5594370fdf30 2024-04-22T00:04:25.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.215+0000 7f6d537fe640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0139c0 tx=0x7f6d4c013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cc3d914ec49b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:25.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.215+0000 7f6d32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:25.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.216+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c018040 con 0x5594370fdf30 2024-04-22T00:04:25.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.216+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d4c014d70 con 0x5594370fdf30 2024-04-22T00:04:25.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.216+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6d4c014890 con 0x5594370fdf30 2024-04-22T00:04:25.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.217+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6d4c032050 con 0x5594370fdf30 2024-04-22T00:04:25.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.218+0000 7f6d5c65c740 10 client.16542 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:25.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.218+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559436d360f0 con 0x5594370fdf30 2024-04-22T00:04:25.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4c014d70 con 0x5594370fdf30 2024-04-22T00:04:25.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f6d4c01e430 con 0x5594370fdf30 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 10 client.16542 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 20 client.16542 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559437117b10 0x559437119f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 4 client.16542 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.222+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55943706faa0 con 0x559437117b10 2024-04-22T00:04:25.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.223+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559437117b10 0x559437119f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:25.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.225+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559437117b10 0x559437119f70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a4aec9c10bce7c6 server_cookie=eb4e1264f80195e in_seq=0 out_seq=0 2024-04-22T00:04:25.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:25.226+0000 7f6d32ffd640 10 client.16542 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:26.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:26 smithi038 ceph-mon[27972]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 76 MiB/s wr, 10.74k op/s 2024-04-22T00:04:26.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:26 smithi203 ceph-mon[31431]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 76 MiB/s wr, 10.74k op/s 2024-04-22T00:04:26.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:26 smithi111 ceph-mon[31543]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 76 MiB/s wr, 10.74k op/s 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.839+0000 7f7186ffd640 1 -- 172.21.15.38:0/3216715506 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f5f0cbe0b0 con 0x55f5f0dde6d0 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f7186ffd640 10 client.16530 handle_command_reply: tid=0 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 client.16530 shutdown 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 msgr2=0x55f5f0dc8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f71980973c0 tx=0x7f719809ae60 comp rx=0 tx=0).stop 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 shutdown_connections 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f5f0dde6d0 0x55f5f0de0b10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f5f0d56210 0x55f5f0dc8490 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5f0d4d440 0x55f5f0d5e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 --2- 172.21.15.38:0/3216715506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f5f0d4cb20 0x55f5f0d5e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 >> 172.21.15.38:0/3216715506 conn(0x55f5f0d47440 msgr2=0x55f5f0d464c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f6d32ffd640 1 -- 172.21.15.38:0/3162776805 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55943706faa0 con 0x559437117b10 2024-04-22T00:04:27.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f6d32ffd640 10 client.16542 handle_command_reply: tid=0 2024-04-22T00:04:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f7186ffd640 0 client.16530 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:27.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f6d5c65c740 1 client.16542 shutdown 2024-04-22T00:04:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 shutdown_connections 2024-04-22T00:04:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 1 -- 172.21.15.38:0/3216715506 wait complete. 2024-04-22T00:04:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:04:27.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.840+0000 7f71b05d4740 20 client.16530 trim_cache size 0 max 16384 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.841+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 msgr2=0x7f71900a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.841+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 0x7f71900a5310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f719c009230 tx=0x7f719c005a20 comp rx=0 tx=0).stop 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.841+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 msgr2=0x7f71a81b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.841+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7198009fd0 tx=0x7f719800b8e0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 msgr2=0x559437104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0139c0 tx=0x7f6d4c013ba0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 shutdown_connections 2024-04-22T00:04:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f71900a2e50 0x7f71900a5310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71a81122a0 0x7f71a81b2f60 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71a8114890 0x7f71a8111d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 --2- 172.21.15.38:0/3752712214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71a8109060 0x7f71a8111820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 >> 172.21.15.38:0/3752712214 conn(0x7f71a806c810 msgr2=0x7f71a8070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 shutdown_connections 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x559437104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x559437104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559437117b10 0x559437119f70 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/3162776805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x559437104320 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 >> 172.21.15.38:0/3162776805 conn(0x5594370f8660 msgr2=0x5594370fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 shutdown_connections 2024-04-22T00:04:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f71ad9e5640 1 -- 172.21.15.38:0/3752712214 wait complete. 2024-04-22T00:04:27.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d32ffd640 0 client.16542 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:27.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 shutdown_connections 2024-04-22T00:04:27.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.842+0000 7f6d5c65c740 1 -- 172.21.15.38:0/3162776805 wait complete. 2024-04-22T00:04:27.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.843+0000 7f6d5c65c740 20 client.16542 trim_cache size 0 max 16384 2024-04-22T00:04:27.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.877+0000 7f6d5c65c740 1 Processor -- start 2024-04-22T00:04:27.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.877+0000 7f6d5c65c740 1 -- start start 2024-04-22T00:04:27.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x5594371131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x5594370fa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x5594370fa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541bd360 con 0x5594370fdf30 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541bf020 con 0x559437107430 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d5c65c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541bda30 con 0x5594370fe850 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d52ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x5594370fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d52ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x5594370fa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50012/0 (socket says 172.21.15.38:50012) 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d52ffd640 1 -- 172.21.15.38:0/194771245 learned_addr learned my addr 172.21.15.38:0/194771245 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d53fff640 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x5594370fa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d53fff640 1 -- 172.21.15.38:0/194771245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 msgr2=0x5594370fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d53fff640 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x5594370fa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d53fff640 1 -- 172.21.15.38:0/194771245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 msgr2=0x5594371131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.878+0000 7f6d53fff640 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x5594371131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.879+0000 7f6d53fff640 1 -- 172.21.15.38:0/194771245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594370fff00 con 0x559437107430 2024-04-22T00:04:27.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.879+0000 7f6d53fff640 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x5594370fa5d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d44010970 tx=0x7f6d44013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ce489570c57f9e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:27.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.880+0000 7f6d397fa640 1 -- 172.21.15.38:0/194771245 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4401c040 con 0x559437107430 2024-04-22T00:04:27.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.880+0000 7f6d397fa640 1 -- 172.21.15.38:0/194771245 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44026070 con 0x559437107430 2024-04-22T00:04:27.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.880+0000 7f6d397fa640 1 -- 172.21.15.38:0/194771245 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400f040 con 0x559437107430 2024-04-22T00:04:27.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 -- 172.21.15.38:0/194771245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 msgr2=0x5594370fa5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x5594370fa5d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d44010970 tx=0x7f6d44013000 comp rx=0 tx=0).stop 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 -- 172.21.15.38:0/194771245 shutdown_connections 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x5594370fa5d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fe850 0x5594370fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/194771245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fdf30 0x5594371131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.881+0000 7f6d5c65c740 1 -- 172.21.15.38:0/194771245 >> 172.21.15.38:0/194771245 conn(0x5594370f8660 msgr2=0x5594370f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.882+0000 7f6d5c65c740 1 -- 172.21.15.38:0/194771245 shutdown_connections 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.882+0000 7f6d5c65c740 1 -- 172.21.15.38:0/194771245 wait complete. 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.882+0000 7f6d5c65c740 1 -- start start 2024-04-22T00:04:27.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.882+0000 7f6d5c65c740 1 Processor -- start 2024-04-22T00:04:27.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.883+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x55943710a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.883+0000 7f6d5c65c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.883+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x55943710a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.883+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x55943710a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50022/0 (socket says 172.21.15.38:50022) 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.883+0000 7f6d52ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d52ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46738/0 (socket says 172.21.15.38:46738) 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d537fe640 1 -- 172.21.15.38:0/1714349951 learned_addr learned my addr 172.21.15.38:0/1714349951 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:27.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x55943710b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541be890 con 0x5594370fe850 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541be1c0 con 0x559437107430 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541bb340 con 0x5594370fdf30 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d5c65c740 10 client.0 _ll_register_callbacks cb 0x7f6d4c00c490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x55943710b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 -- 172.21.15.38:0/1714349951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 msgr2=0x55943710a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x55943710a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 -- 172.21.15.38:0/1714349951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 msgr2=0x55943710abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d53fff640 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55943717c0e0 con 0x559437107430 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.884+0000 7f6d52ffd640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:27.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.885+0000 7f6d53fff640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x55943710b100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009140 tx=0x7f6d44015ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e57d93799a24d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.885+0000 7f6d32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.885+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44030070 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.885+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6d4402c850 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.886+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d44026070 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.886+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6d4403c050 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.886+0000 7f6d5c65c740 10 client.25952 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.886+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55943717d330 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d44026070 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f6d4402a5b0 con 0x559437107430 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 10 client.25952 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 10 client.25952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 20 client.25952 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55943717da70 0x55943717fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 4 client.25952 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:04:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.887+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559437069df0 con 0x55943717da70 2024-04-22T00:04:27.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.888+0000 7f6d537fe640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55943717da70 0x55943717fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d537fe640 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55943717da70 0x55943717fe90 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804c2cce90f4ee56 server_cookie=bde0771dbfd998a1 in_seq=0 out_seq=0 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d32ffd640 10 client.25952 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d32ffd640 1 -- 172.21.15.38:0/1714349951 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559437069df0 con 0x55943717da70 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d32ffd640 10 client.25952 handle_command_reply: tid=0 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d5c65c740 1 client.25952 shutdown 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 msgr2=0x55943710b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.890+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x55943710b100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009140 tx=0x7f6d44015ca0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 shutdown_connections 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559437107430 0x55943710b100 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55943717da70 0x55943717fe90 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5594370fe850 0x55943710abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d32ffd640 0 client.25952 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 --2- 172.21.15.38:0/1714349951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594370fdf30 0x55943710a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.891+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 >> 172.21.15.38:0/1714349951 conn(0x5594370f8660 msgr2=0x5594370fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.892+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 shutdown_connections 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.892+0000 7f6d5c65c740 1 -- 172.21.15.38:0/1714349951 wait complete. 2024-04-22T00:04:27.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.892+0000 7f6d5c65c740 20 client.25952 trim_cache size 0 max 16384 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.895+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 msgr2=0x7f6d3c0a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.895+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 0x7f6d3c0a5270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6d44006fd0 tx=0x7f6d44008040 comp rx=0 tx=0).stop 2024-04-22T00:04:27.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.895+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 msgr2=0x7f6d54115020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:27.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.895+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 0x7f6d54115020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009850 tx=0x7f6d4802c950 comp rx=0 tx=0).stop 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 shutdown_connections 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6d3c0a2db0 0x7f6d3c0a5270 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d54106710 0x7f6d5410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d54104210 0x7f6d54115020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 --2- 172.21.15.38:0/1233505971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6d5406a2a0 0x7f6d54114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.896+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 >> 172.21.15.38:0/1233505971 conn(0x7f6d54077620 msgr2=0x7f6d54102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:27.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.897+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 shutdown_connections 2024-04-22T00:04:27.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:27.897+0000 7f6d59a78640 1 -- 172.21.15.38:0/1233505971 wait complete. 2024-04-22T00:04:27.907 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:04:27.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:04:28.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.088+0000 7ff543fff640 1 -- 172.21.15.38:0/1981763344 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548004c40 con 0x7ff54c100610 2024-04-22T00:04:28.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 -- 172.21.15.38:0/1981763344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100610 msgr2=0x7ff54c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 --2- 172.21.15.38:0/1981763344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100610 0x7ff54c100a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff548013150 tx=0x7ff548033fc0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 -- 172.21.15.38:0/1981763344 shutdown_connections 2024-04-22T00:04:28.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 --2- 172.21.15.38:0/1981763344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c100fd0 0x7ff54c069d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 --2- 172.21.15.38:0/1981763344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100610 0x7ff54c100a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 --2- 172.21.15.38:0/1981763344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c1066b0 0x7ff54c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.089+0000 7ff55386b640 1 -- 172.21.15.38:0/1981763344 >> 172.21.15.38:0/1981763344 conn(0x7ff54c0769c0 msgr2=0x7ff54c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.090+0000 7ff55386b640 1 -- 172.21.15.38:0/1981763344 shutdown_connections 2024-04-22T00:04:28.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.090+0000 7ff55386b640 1 -- 172.21.15.38:0/1981763344 wait complete. 2024-04-22T00:04:28.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 Processor -- start 2024-04-22T00:04:28.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 -- start start 2024-04-22T00:04:28.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c100610 0x7ff54c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c1066b0 0x7ff54c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c10fa80 con 0x7ff54c100610 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c10f900 con 0x7ff54c1066b0 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.091+0000 7ff55386b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c10fc00 con 0x7ff54c100fd0 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.092+0000 7ff550e06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.092+0000 7ff550e06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50044/0 (socket says 172.21.15.38:50044) 2024-04-22T00:04:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.092+0000 7ff550e06640 1 -- 172.21.15.38:0/1417145963 learned_addr learned my addr 172.21.15.38:0/1417145963 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.092+0000 7ff551e08640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c1066b0 0x7ff54c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.092+0000 7ff551607640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c100610 0x7ff54c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 -- 172.21.15.38:0/1417145963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c1066b0 msgr2=0x7ff54c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c1066b0 0x7ff54c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 -- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c100610 msgr2=0x7ff54c195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c100610 0x7ff54c195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 -- 172.21.15.38:0/1417145963 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c19ae30 con 0x7ff54c100fd0 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff550e06640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff5480345d0 tx=0x7ff548004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480110d0 con 0x7ff54c100fd0 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.093+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54c19b0c0 con 0x7ff54c100fd0 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.094+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54c1b39f0 con 0x7ff54c100fd0 2024-04-22T00:04:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.094+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff548011750 con 0x7ff54c100fd0 2024-04-22T00:04:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.094+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54800d460 con 0x7ff54c100fd0 2024-04-22T00:04:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.096+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff530005290 con 0x7ff54c100fd0 2024-04-22T00:04:28.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.100+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff548011270 con 0x7ff54c100fd0 2024-04-22T00:04:28.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.101+0000 7ff5427fc640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 0x7ff5380a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.102+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff5480f1470 con 0x7ff54c100fd0 2024-04-22T00:04:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.102+0000 7ff551607640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 0x7ff5380a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.104+0000 7ff551607640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 0x7ff5380a5390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff53c006fd0 tx=0x7ff53c008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.104+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff54804a070 con 0x7ff54c100fd0 2024-04-22T00:04:28.208 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:28 smithi038 ceph-mon[27972]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 MiB/s wr, 16.05k op/s 2024-04-22T00:04:28.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:28 smithi203 ceph-mon[31431]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 MiB/s wr, 16.05k op/s 2024-04-22T00:04:28.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.578+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff530005580 con 0x7ff54c100fd0 2024-04-22T00:04:28.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.579+0000 7ff5427fc640 1 -- 172.21.15.38:0/1417145963 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11949 (secure 0 0 0) 0x7ff5480bde80 con 0x7ff54c100fd0 2024-04-22T00:04:28.572 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:04:28.572 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:03:41.555018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:04:28.573 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 32 2024-04-22T00:04:28.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.582+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 msgr2=0x7ff5380a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.582+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 0x7ff5380a5390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff53c006fd0 tx=0x7ff53c008040 comp rx=0 tx=0).stop 2024-04-22T00:04:28.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.582+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 msgr2=0x7ff54c1963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.582+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff5480345d0 tx=0x7ff548004430 comp rx=0 tx=0).stop 2024-04-22T00:04:28.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 shutdown_connections 2024-04-22T00:04:28.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5380a2ed0 0x7ff5380a5390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c1066b0 0x7ff54c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff54c100fd0 0x7ff54c1963b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 --2- 172.21.15.38:0/1417145963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff54c100610 0x7ff54c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.583+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 >> 172.21.15.38:0/1417145963 conn(0x7ff54c0769c0 msgr2=0x7ff54c1041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:28 smithi111 ceph-mon[31543]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 MiB/s wr, 16.05k op/s 2024-04-22T00:04:28.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.584+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 shutdown_connections 2024-04-22T00:04:28.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.585+0000 7ff55386b640 1 -- 172.21.15.38:0/1417145963 wait complete. 2024-04-22T00:04:28.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 -- 172.21.15.38:0/2574214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc100b20 msgr2=0x7f06dc100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 --2- 172.21.15.38:0/2574214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc100b20 0x7f06dc100fa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009900 tx=0x7f06cc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 -- 172.21.15.38:0/2574214157 shutdown_connections 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 --2- 172.21.15.38:0/2574214157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc1014e0 0x7f06dc10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 --2- 172.21.15.38:0/2574214157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc100b20 0x7f06dc100fa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 --2- 172.21.15.38:0/2574214157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1066a0 0x7f06dc106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 -- 172.21.15.38:0/2574214157 >> 172.21.15.38:0/2574214157 conn(0x7f06dc0761c0 msgr2=0x7f06dc0765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 -- 172.21.15.38:0/2574214157 shutdown_connections 2024-04-22T00:04:28.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.768+0000 7f06e3da7640 1 -- 172.21.15.38:0/2574214157 wait complete. 2024-04-22T00:04:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 Processor -- start 2024-04-22T00:04:28.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 -- start start 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 0x7f06dc1022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1014e0 0x7f06dc1027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc110a30 con 0x7f06dc100b20 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1108b0 con 0x7f06dc1066a0 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e3da7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc110bb0 con 0x7f06dc1014e0 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 0x7f06dc1022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e2344640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e1342640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1014e0 0x7f06dc1027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e2344640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42160/0 (socket says 172.21.15.38:42160) 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 -- 172.21.15.38:0/498613250 learned_addr learned my addr 172.21.15.38:0/498613250 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.769+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 0x7f06dc1022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46762/0 (socket says 172.21.15.38:46762) 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 -- 172.21.15.38:0/498613250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1014e0 msgr2=0x7f06dc1027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1014e0 0x7f06dc1027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 -- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 msgr2=0x7f06dc1022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 0x7f06dc1022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 -- 172.21.15.38:0/498613250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc1a6540 con 0x7f06dc1066a0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06e2344640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06d400eb10 tx=0x7f06d400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06d2ffd640 1 -- 172.21.15.38:0/498613250 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4004b20 con 0x7f06dc1066a0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06d2ffd640 1 -- 172.21.15.38:0/498613250 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d4004cc0 con 0x7f06dc1066a0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.770+0000 7f06d2ffd640 1 -- 172.21.15.38:0/498613250 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4010760 con 0x7f06dc1066a0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.771+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc1a6830 con 0x7f06dc1066a0 2024-04-22T00:04:28.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.771+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc1a6d00 con 0x7f06dc1066a0 2024-04-22T00:04:28.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.774+0000 7f06d2ffd640 1 -- 172.21.15.38:0/498613250 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f06d4010900 con 0x7f06dc1066a0 2024-04-22T00:04:28.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.774+0000 7f06e698a740 1 Processor -- start 2024-04-22T00:04:28.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.774+0000 7f06e698a740 1 -- start start 2024-04-22T00:04:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 0x5630c4c56400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06d2ffd640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 0x7f06c80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 0x5630c4c5efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c61910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5630c4bba530 con 0x5630c4c56940 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5630c4b62830 con 0x5630c4c5f520 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e2344640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c61910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e2344640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c61910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42176/0 (socket says 172.21.15.38:42176) 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e2344640 1 -- 172.21.15.38:0/1548815503 learned_addr learned my addr 172.21.15.38:0/1548815503 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5630c4bb64f0 con 0x5630c4c56020 2024-04-22T00:04:28.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.775+0000 7f06d2ffd640 1 -- 172.21.15.38:0/498613250 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f06d40cfd00 con 0x7f06dc1066a0 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.776+0000 7f06e1342640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 0x7f06c80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.776+0000 7f06e1342640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 0x5630c4c5efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 0x5630c4c56400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 -- 172.21.15.38:0/1548815503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 msgr2=0x5630c4c61910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c61910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 -- 172.21.15.38:0/1548815503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 msgr2=0x5630c4c5efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 0x5630c4c5efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 -- 172.21.15.38:0/1548815503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c4b862c0 con 0x5630c4c56020 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1342640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 0x5630c4c5efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1342640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 0x7f06c80a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02efd0 tx=0x7f06cc004120 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.777+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 0x5630c4c56400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009bd0 tx=0x7f06d8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3eae3ef14e2d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06c7fff640 1 -- 172.21.15.38:0/1548815503 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d801e070 con 0x5630c4c56020 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06c7fff640 1 -- 172.21.15.38:0/1548815503 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d8009d80 con 0x5630c4c56020 2024-04-22T00:04:28.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06c7fff640 1 -- 172.21.15.38:0/1548815503 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8019450 con 0x5630c4c56020 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 msgr2=0x5630c4c56400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06e698a740 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 0x5630c4c56400 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009bd0 tx=0x7f06d8011810 comp rx=0 tx=0).stop 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 shutdown_connections 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.778+0000 7f06e698a740 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c61910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56940 0x5630c4c5efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 --2- 172.21.15.38:0/1548815503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56020 0x5630c4c56400 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 >> 172.21.15.38:0/1548815503 conn(0x5630c4c50750 msgr2=0x5630c4c50b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 shutdown_connections 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 -- 172.21.15.38:0/1548815503 wait complete. 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 -- start start 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.779+0000 7f06e698a740 1 Processor -- start 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 0x5630c4c5eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c5c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5630c4c62310 con 0x5630c4c56020 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5630c4b75f10 con 0x5630c4c5f520 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5630c4c62490 con 0x5630c4c56940 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e698a740 10 client.0 _ll_register_callbacks cb 0x5630c48cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 0x5630c4c5eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 0x5630c4c5eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46776/0 (socket says 172.21.15.38:46776) 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e1342640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e2344640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c5c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e1b43640 1 -- 172.21.15.38:0/3979972055 learned_addr learned my addr 172.21.15.38:0/3979972055 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e2344640 1 -- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 msgr2=0x5630c4c5c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e2344640 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e2344640 1 -- 172.21.15.38:0/3979972055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 msgr2=0x5630c4c5eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.780+0000 7f06e2344640 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 0x5630c4c5eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06e2344640 1 -- 172.21.15.38:0/3979972055 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c4b47c80 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06e2344640 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c5c7f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06d40988b0 tx=0x7f06d409bae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8f915aace4b807 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d409bee0 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f06d40bd310 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d8009d80 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f06d40a0290 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06e698a740 10 client.25967 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.781+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5630c48ce3f0 con 0x5630c4c5f520 2024-04-22T00:04:28.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d8009d80 con 0x5630c4c5f520 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f06d409b330 con 0x5630c4c5f520 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 10 client.25967 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 10 client.25967 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 20 client.25967 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4c6f2e0 0x5630c4c71710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 4 client.25967 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630c4bc7a70 con 0x5630c4c6f2e0 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.782+0000 7f06e1b43640 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4c6f2e0 0x5630c4c71710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.783+0000 7f06e1b43640 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4c6f2e0 0x5630c4c71710 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fab3bf00f2b68c95 server_cookie=d05003b8290ba15a in_seq=0 out_seq=0 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.783+0000 7f06c57fa640 10 client.25967 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06c57fa640 1 -- 172.21.15.38:0/3979972055 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5630c4bc7a70 con 0x5630c4c6f2e0 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06c57fa640 10 client.25967 handle_command_reply: tid=0 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 client.25967 shutdown 2024-04-22T00:04:28.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 msgr2=0x5630c4c5c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c5c7f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06d40988b0 tx=0x7f06d409bae0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 shutdown_connections 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4c6f2e0 0x5630c4c71710 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c5f520 0x5630c4c5c7f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 --2- 172.21.15.38:0/3979972055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c56020 0x5630c4c5eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 >> 172.21.15.38:0/3979972055 conn(0x5630c4c50750 msgr2=0x5630c4c524a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06c57fa640 0 client.25967 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 shutdown_connections 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.784+0000 7f06e698a740 1 -- 172.21.15.38:0/3979972055 wait complete. 2024-04-22T00:04:28.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.785+0000 7f06e698a740 20 client.25967 trim_cache size 0 max 16384 2024-04-22T00:04:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.807+0000 7f06e698a740 1 Processor -- start 2024-04-22T00:04:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.807+0000 7f06e698a740 1 -- start start 2024-04-22T00:04:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c6b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ac7b0 con 0x5630c4c5f520 2024-04-22T00:04:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ae470 con 0x5630c4c56020 2024-04-22T00:04:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e698a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ace80 con 0x5630c4c56940 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e2344640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e2344640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46780/0 (socket says 172.21.15.38:46780) 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e1342640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e2344640 1 -- 172.21.15.38:0/983217061 learned_addr learned my addr 172.21.15.38:0/983217061 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.808+0000 7f06e1b43640 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c6b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 -- 172.21.15.38:0/983217061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 msgr2=0x5630c4c5b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 -- 172.21.15.38:0/983217061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 msgr2=0x5630c4c6b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c6b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 -- 172.21.15.38:0/983217061 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c4c56770 con 0x5630c4c5f520 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.809+0000 7f06e2344640 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f06d4098340 tx=0x7f06d400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96c5d2ded043446 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.810+0000 7f06e1b43640 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c6b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.810+0000 7f06c77fe640 1 -- 172.21.15.38:0/983217061 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d409bcb0 con 0x5630c4c5f520 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.810+0000 7f06c77fe640 1 -- 172.21.15.38:0/983217061 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d40b7070 con 0x5630c4c5f520 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.810+0000 7f06c77fe640 1 -- 172.21.15.38:0/983217061 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d40a46c0 con 0x5630c4c5f520 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- 172.21.15.38:0/983217061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 msgr2=0x5630c4c4f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f06d4098340 tx=0x7f06d400cf70 comp rx=0 tx=0).stop 2024-04-22T00:04:28.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- 172.21.15.38:0/983217061 shutdown_connections 2024-04-22T00:04:28.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c4f6d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c5b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 --2- 172.21.15.38:0/983217061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c6b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- 172.21.15.38:0/983217061 >> 172.21.15.38:0/983217061 conn(0x5630c4c50750 msgr2=0x5630c4c4efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- 172.21.15.38:0/983217061 shutdown_connections 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- 172.21.15.38:0/983217061 wait complete. 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 -- start start 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.811+0000 7f06e698a740 1 Processor -- start 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c62dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c63300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1adce0 con 0x5630c4c5f520 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ad610 con 0x5630c4c56020 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1aa7d0 con 0x5630c4c56940 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e698a740 10 client.0 _ll_register_callbacks cb 0x7f06d4004e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42198/0 (socket says 172.21.15.38:42198) 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 -- 172.21.15.38:0/1783439661 learned_addr learned my addr 172.21.15.38:0/1783439661 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1342640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c62dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 -- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 msgr2=0x5630c4c62dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c62dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 -- 172.21.15.38:0/1783439661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 msgr2=0x5630c4c63300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c63300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1b43640 1 -- 172.21.15.38:0/1783439661 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c49987d0 con 0x5630c4c56020 2024-04-22T00:04:28.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.812+0000 7f06e1342640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c62dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:04:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.813+0000 7f06e1b43640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009b20 tx=0x7f06d8003a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d5958c0343e88e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.813+0000 7f06c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8031060 con 0x5630c4c56020 2024-04-22T00:04:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f06d8003ba0 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d40b7070 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f06d8038050 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 10 client.25973 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5630c4cd54d0 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d40b7070 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f06d802b6c0 con 0x5630c4c56020 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 10 client.25973 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 10 client.25973 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 20 client.25973 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.814+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4cd5a30 0x5630c4cd7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.815+0000 7f06e698a740 4 client.25973 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.815+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630c4bc1dc0 con 0x5630c4cd5a30 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.815+0000 7f06e1342640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4cd5a30 0x5630c4cd7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06e1342640 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4cd5a30 0x5630c4cd7e50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4fbe741282745d3e server_cookie=3fd815b407926877 in_seq=0 out_seq=0 2024-04-22T00:04:28.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06c57fa640 10 client.25973 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06c57fa640 1 -- 172.21.15.38:0/1783439661 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630c4bc1dc0 con 0x5630c4cd5a30 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06c57fa640 10 client.25973 handle_command_reply: tid=0 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06e698a740 1 client.25973 shutdown 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 msgr2=0x5630c4c67950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.816+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009b20 tx=0x7f06d8003a60 comp rx=0 tx=0).stop 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 shutdown_connections 2024-04-22T00:04:28.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5630c4cd5a30 0x5630c4cd7e50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630c4c5f520 0x5630c4c63300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5630c4c56940 0x5630c4c62dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 --2- 172.21.15.38:0/1783439661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630c4c56020 0x5630c4c67950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 >> 172.21.15.38:0/1783439661 conn(0x5630c4c50750 msgr2=0x5630c4c5b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06c57fa640 0 client.25973 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 shutdown_connections 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 1 -- 172.21.15.38:0/1783439661 wait complete. 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.817+0000 7f06e698a740 20 client.25973 trim_cache size 0 max 16384 2024-04-22T00:04:28.812 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:04:28.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 msgr2=0x7f06c80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 0x7f06c80a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f06cc02efd0 tx=0x7f06cc004120 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 msgr2=0x7f06dc102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06d400eb10 tx=0x7f06d400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 shutdown_connections 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f06c80a2d80 0x7f06c80a5240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06dc1066a0 0x7f06dc102d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06dc1014e0 0x7f06dc1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 --2- 172.21.15.38:0/498613250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06dc100b20 0x7f06dc1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 >> 172.21.15.38:0/498613250 conn(0x7f06dc0761c0 msgr2=0x7f06dc0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.818+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 shutdown_connections 2024-04-22T00:04:28.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.819+0000 7f06e3da7640 1 -- 172.21.15.38:0/498613250 wait complete. 2024-04-22T00:04:28.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:04:28.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 -- 172.21.15.38:0/132561385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 msgr2=0x7f32e0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/132561385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e0106a90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009930 tx=0x7f32d402bc30 comp rx=0 tx=0).stop 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 -- 172.21.15.38:0/132561385 shutdown_connections 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/132561385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100fd0 0x7f32e010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/132561385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100610 0x7f32e0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/132561385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e0106a90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 -- 172.21.15.38:0/132561385 >> 172.21.15.38:0/132561385 conn(0x7f32e0077740 msgr2=0x7f32e0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 -- 172.21.15.38:0/132561385 shutdown_connections 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.992+0000 7f32e6dfe640 1 -- 172.21.15.38:0/132561385 wait complete. 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 Processor -- start 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 -- start start 2024-04-22T00:04:28.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100610 0x7f32e0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100fd0 0x7f32e0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0111b30 con 0x7f32e01066b0 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01119b0 con 0x7f32e0100fd0 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e6dfe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0111cb0 con 0x7f32e0100610 2024-04-22T00:04:28.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46784/0 (socket says 172.21.15.38:46784) 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 -- 172.21.15.38:0/955781279 learned_addr learned my addr 172.21.15.38:0/955781279 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32dffff640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100fd0 0x7f32e0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100610 0x7f32e0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 -- 172.21.15.38:0/955781279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100610 msgr2=0x7f32e0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100610 0x7f32e0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 -- 172.21.15.38:0/955781279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100fd0 msgr2=0x7f32e0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100fd0 0x7f32e0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.993+0000 7f32e539b640 1 -- 172.21.15.38:0/955781279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e019cfb0 con 0x7f32e01066b0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32e539b640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f32d800eb10 tx=0x7f32d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32ddffb640 1 -- 172.21.15.38:0/955781279 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004b20 con 0x7f32e01066b0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32ddffb640 1 -- 172.21.15.38:0/955781279 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d8004cc0 con 0x7f32e01066b0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e019d2a0 con 0x7f32e01066b0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32ddffb640 1 -- 172.21.15.38:0/955781279 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8010760 con 0x7f32e01066b0 2024-04-22T00:04:28.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.994+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e01a4b30 con 0x7f32e01066b0 2024-04-22T00:04:28.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.996+0000 7f32ddffb640 1 -- 172.21.15.38:0/955781279 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f32d8005020 con 0x7f32e01066b0 2024-04-22T00:04:28.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32ddffb640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 0x7f32d00a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 Processor -- start 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32ddffb640 1 -- 172.21.15.38:0/955781279 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f32d80cfe10 con 0x7f32e01066b0 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 0x7f32d00a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 -- start start 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 0x55c1468cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468ded20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c146833530 con 0x55c1468cf670 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c1467db830 con 0x55c1468ced50 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e99e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c14682f4f0 con 0x55c1468dc930 2024-04-22T00:04:28.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32dffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 0x7f32d00a52c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f32d4000c00 tx=0x7f32d4005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e4b9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 0x55c1468cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32dffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46798/0 (socket says 172.21.15.38:46798) 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32dffff640 1 -- 172.21.15.38:0/3273773571 learned_addr learned my addr 172.21.15.38:0/3273773571 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.997+0000 7f32e539b640 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 -- 172.21.15.38:0/3273773571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 msgr2=0x55c1468ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468ded20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 -- 172.21.15.38:0/3273773571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 msgr2=0x55c1468cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 0x55c1468cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 -- 172.21.15.38:0/3273773571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1467ff2c0 con 0x55c1468cf670 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 0x55c1468cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32dffff640 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009c90 tx=0x7f32cc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6534725f97b19e29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32caffd640 1 -- 172.21.15.38:0/3273773571 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc01e070 con 0x55c1468cf670 2024-04-22T00:04:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32caffd640 1 -- 172.21.15.38:0/3273773571 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32cc009e40 con 0x55c1468cf670 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.998+0000 7f32caffd640 1 -- 172.21.15.38:0/3273773571 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc00ea20 con 0x55c1468cf670 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- 172.21.15.38:0/3273773571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 msgr2=0x55c1468dc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009c90 tx=0x7f32cc011810 comp rx=0 tx=0).stop 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- 172.21.15.38:0/3273773571 shutdown_connections 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468ded20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468dc3f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3273773571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468ced50 0x55c1468cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- 172.21.15.38:0/3273773571 >> 172.21.15.38:0/3273773571 conn(0x55c1468c9750 msgr2=0x55c1468c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- 172.21.15.38:0/3273773571 shutdown_connections 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- 172.21.15.38:0/3273773571 wait complete. 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 -- start start 2024-04-22T00:04:29.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:28.999+0000 7f32e99e1740 1 Processor -- start 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468d9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc930 0x55c1468d9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c1468df6f0 con 0x55c1468ced50 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c1467eef10 con 0x55c1468dc930 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c1468df870 con 0x55c1468cf670 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e99e1740 10 client.0 _ll_register_callbacks cb 0x55c146546be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32dffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46802/0 (socket says 172.21.15.38:46802) 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 -- 172.21.15.38:0/1331015166 learned_addr learned my addr 172.21.15.38:0/1331015166 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 -- 172.21.15.38:0/1331015166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 msgr2=0x55c1468d9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468d9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 -- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc930 msgr2=0x55c1468d9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc930 0x55c1468d9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 -- 172.21.15.38:0/1331015166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1467c0c80 con 0x55c1468ced50 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32dffff640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.000+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f32d4005ab0 tx=0x7f32d402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be28a26dc2e00e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32bffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4004c50 con 0x55c1468ced50 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f32d4004f50 con 0x55c1468ced50 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c146840a70 con 0x55c1468ced50 2024-04-22T00:04:29.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f32d4005150 con 0x55c1468ced50 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32e99e1740 10 client.16575 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c1465473f0 con 0x55c1468ced50 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.001+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c146840a70 con 0x55c1468ced50 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f32d4034440 con 0x55c1468ced50 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 10 client.16575 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 10 client.16575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 20 client.16575 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c1468d13d0 0x55c1468d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 4 client.16575 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c14683adc0 con 0x55c1468d13d0 2024-04-22T00:04:29.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.002+0000 7f32dffff640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c1468d13d0 0x55c1468d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32bffff640 10 client.16575 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32dffff640 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c1468d13d0 0x55c1468d3860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=46d43d13149d39b server_cookie=1726eba31bd4df84 in_seq=0 out_seq=0 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32bffff640 1 -- 172.21.15.38:0/1331015166 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c14683adc0 con 0x55c1468d13d0 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32bffff640 10 client.16575 handle_command_reply: tid=0 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32e99e1740 1 client.16575 shutdown 2024-04-22T00:04:29.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 msgr2=0x55c1468dc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.004+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f32d4005ab0 tx=0x7f32d402bc30 comp rx=0 tx=0).stop 2024-04-22T00:04:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 shutdown_connections 2024-04-22T00:04:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc930 0x55c1468d9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c1468d13d0 0x55c1468d3860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 --2- 172.21.15.38:0/1331015166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468ced50 0x55c1468dc270 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 >> 172.21.15.38:0/1331015166 conn(0x55c1468c9750 msgr2=0x55c1468cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32bffff640 0 client.16575 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 shutdown_connections 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 1 -- 172.21.15.38:0/1331015166 wait complete. 2024-04-22T00:04:29.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.005+0000 7f32e99e1740 20 client.16575 trim_cache size 0 max 16384 2024-04-22T00:04:29.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 Processor -- start 2024-04-22T00:04:29.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 -- start start 2024-04-22T00:04:29.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01ac8e0 con 0x55c1468cf670 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01ae5a0 con 0x55c1468dc270 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.023+0000 7f32e99e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01acfb0 con 0x55c1468dc930 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e4b9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e4b9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46814/0 (socket says 172.21.15.38:46814) 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e4b9a640 1 -- 172.21.15.38:0/3470005985 learned_addr learned my addr 172.21.15.38:0/3470005985 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e539b640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 -- 172.21.15.38:0/3470005985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 msgr2=0x55c1468f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 -- 172.21.15.38:0/3470005985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 msgr2=0x55c1468e85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 -- 172.21.15.38:0/3470005985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1466117d0 con 0x55c1468dc930 2024-04-22T00:04:29.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32e539b640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.024+0000 7f32dffff640 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468dbd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f32cc000c00 tx=0x7f32cc00e580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a347cbf4f031eec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.025+0000 7f32ca7fc640 1 -- 172.21.15.38:0/3470005985 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc01e040 con 0x55c1468dc930 2024-04-22T00:04:29.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.025+0000 7f32ca7fc640 1 -- 172.21.15.38:0/3470005985 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32cc01f040 con 0x55c1468dc930 2024-04-22T00:04:29.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.025+0000 7f32ca7fc640 1 -- 172.21.15.38:0/3470005985 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc01e040 con 0x55c1468dc930 2024-04-22T00:04:29.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- 172.21.15.38:0/3470005985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 msgr2=0x55c1468dbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468dbd30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f32cc000c00 tx=0x7f32cc00e580 comp rx=0 tx=0).stop 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- 172.21.15.38:0/3470005985 shutdown_connections 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468dc930 0x55c1468dbd30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 --2- 172.21.15.38:0/3470005985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468cf670 0x55c1468e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- 172.21.15.38:0/3470005985 >> 172.21.15.38:0/3470005985 conn(0x55c1468c9750 msgr2=0x55c1468c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- 172.21.15.38:0/3470005985 shutdown_connections 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- 172.21.15.38:0/3470005985 wait complete. 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.026+0000 7f32e99e1740 1 -- start start 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 Processor -- start 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468e00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01ade10 con 0x55c1468dc930 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01ad740 con 0x55c1468dc270 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f32e01aa8c0 con 0x55c1468cf670 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e99e1740 10 client.0 _ll_register_callbacks cb 0x7f32d4042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e539b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32dffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468e00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e539b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46830/0 (socket says 172.21.15.38:46830) 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e539b640 1 -- 172.21.15.38:0/54506100 learned_addr learned my addr 172.21.15.38:0/54506100 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.027+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 -- 172.21.15.38:0/54506100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 msgr2=0x55c1468e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 -- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 msgr2=0x55c1468e00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468e00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 -- 172.21.15.38:0/54506100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1468e0b60 con 0x55c1468dc930 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:04:29.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e539b640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f32d809bf20 tx=0x7f32d809be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fc92de582b5fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32bffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d80b4070 con 0x55c1468dc930 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f32d80a4970 con 0x55c1468dc930 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32cc01f040 con 0x55c1468dc930 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f32d80cd650 con 0x55c1468dc930 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e99e1740 10 client.16581 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c146957300 con 0x55c1468dc930 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.028+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32cc01f040 con 0x55c1468dc930 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f32d809b3e0 con 0x55c1468dc930 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 10 client.16581 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 10 client.16581 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 20 client.16581 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c146957a20 0x55c146959e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 4 client.16581 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1467b86e0 con 0x55c146957a20 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c146957a20 0x55c146959e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.029+0000 7f32bffff640 10 client.16581 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e4b9a640 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c146957a20 0x55c146959e40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b718bfc7174d0d86 server_cookie=59ae8137b89264c7 in_seq=0 out_seq=0 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32bffff640 1 -- 172.21.15.38:0/54506100 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1467b86e0 con 0x55c146957a20 2024-04-22T00:04:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32bffff640 10 client.16581 handle_command_reply: tid=0 2024-04-22T00:04:29.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 client.16581 shutdown 2024-04-22T00:04:29.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 msgr2=0x55c1468e0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f32d809bf20 tx=0x7f32d809be90 comp rx=0 tx=0).stop 2024-04-22T00:04:29.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 shutdown_connections 2024-04-22T00:04:29.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55c146957a20 0x55c146959e40 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32bffff640 0 client.16581 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c1468dc930 0x55c1468e0620 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c1468dc270 0x55c1468e00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 --2- 172.21.15.38:0/54506100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c1468cf670 0x55c1468e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 >> 172.21.15.38:0/54506100 conn(0x55c1468c9750 msgr2=0x55c1468c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 shutdown_connections 2024-04-22T00:04:29.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.030+0000 7f32e99e1740 1 -- 172.21.15.38:0/54506100 wait complete. 2024-04-22T00:04:29.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e99e1740 20 client.16581 trim_cache size 0 max 16384 2024-04-22T00:04:29.040 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:04:29.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 msgr2=0x7f32d00a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 0x7f32d00a52c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f32d4000c00 tx=0x7f32d4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 msgr2=0x7f32e019c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f32d800eb10 tx=0x7f32d800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 shutdown_connections 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f32d00a2e00 0x7f32d00a52c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32e01066b0 0x7f32e019c880 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32e0100fd0 0x7f32e0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 --2- 172.21.15.38:0/955781279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32e0100610 0x7f32e0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 >> 172.21.15.38:0/955781279 conn(0x7f32e0077740 msgr2=0x7f32e00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 shutdown_connections 2024-04-22T00:04:29.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.031+0000 7f32e6dfe640 1 -- 172.21.15.38:0/955781279 wait complete. 2024-04-22T00:04:29.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:04:29.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.187+0000 7ff8aef9a640 1 -- 172.21.15.38:0/846693233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a8107720 msgr2=0x7ff8a8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.187+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/846693233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a8107720 0x7ff8a8107b00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009930 tx=0x7ff89c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:04:29.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.188+0000 7ff8aef9a640 1 -- 172.21.15.38:0/846693233 shutdown_connections 2024-04-22T00:04:29.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.188+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/846693233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a81020d0 0x7ff8a810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.188+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/846693233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8101710 0x7ff8a8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.188+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/846693233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a8107720 0x7ff8a8107b00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.188+0000 7ff8aef9a640 1 -- 172.21.15.38:0/846693233 >> 172.21.15.38:0/846693233 conn(0x7ff8a8078b90 msgr2=0x7ff8a8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.189+0000 7ff8aef9a640 1 -- 172.21.15.38:0/846693233 shutdown_connections 2024-04-22T00:04:29.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.190+0000 7ff8aef9a640 1 -- 172.21.15.38:0/846693233 wait complete. 2024-04-22T00:04:29.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 Processor -- start 2024-04-22T00:04:29.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 -- start start 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 0x7ff8a810e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 0x7ff8a810b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8107720 0x7ff8a810b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 0x7ff8a810b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 0x7ff8a810b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46852/0 (socket says 172.21.15.38:46852) 2024-04-22T00:04:29.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 0x7ff8a810e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8a7fff640 1 -- 172.21.15.38:0/785364873 learned_addr learned my addr 172.21.15.38:0/785364873 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.191+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a8075150 con 0x7ff8a81020d0 2024-04-22T00:04:29.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a8074fd0 con 0x7ff8a8107720 2024-04-22T00:04:29.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a80752d0 con 0x7ff8a8101710 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8ad537640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8107720 0x7ff8a810b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 -- 172.21.15.38:0/785364873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8107720 msgr2=0x7ff8a810b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8107720 0x7ff8a810b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 -- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 msgr2=0x7ff8a810b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 0x7ff8a810b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a810bf30 con 0x7ff8a8101710 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.192+0000 7ff8acd36640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 0x7ff8a810e220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff89c036040 tx=0x7ff89c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.193+0000 7ff8a5ffb640 1 -- 172.21.15.38:0/785364873 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c043020 con 0x7ff8a8101710 2024-04-22T00:04:29.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.193+0000 7ff8a5ffb640 1 -- 172.21.15.38:0/785364873 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff89c005000 con 0x7ff8a8101710 2024-04-22T00:04:29.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.193+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a81aad30 con 0x7ff8a8101710 2024-04-22T00:04:29.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.193+0000 7ff8a5ffb640 1 -- 172.21.15.38:0/785364873 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c003a10 con 0x7ff8a8101710 2024-04-22T00:04:29.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.193+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a81ab0e0 con 0x7ff8a8101710 2024-04-22T00:04:29.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.196+0000 7ff8a5ffb640 1 -- 172.21.15.38:0/785364873 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff89c046020 con 0x7ff8a8101710 2024-04-22T00:04:29.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.196+0000 7ff8b1b85740 1 Processor -- start 2024-04-22T00:04:29.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 -- start start 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3d1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1940 0x55941e3d9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55941e335530 con 0x55941e3da520 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55941e2dd830 con 0x55941e3d1940 2024-04-22T00:04:29.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.197+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55941e3314f0 con 0x55941e3d1020 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1940 0x55941e3d9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46856/0 (socket says 172.21.15.38:46856) 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 -- 172.21.15.38:0/2591062262 learned_addr learned my addr 172.21.15.38:0/2591062262 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8ad537640 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a5ffb640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 0x7ff8980a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a5ffb640 1 -- 172.21.15.38:0/785364873 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff89c0f1530 con 0x7ff8a8101710 2024-04-22T00:04:29.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 -- 172.21.15.38:0/2591062262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 msgr2=0x55941e3d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3d1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 -- 172.21.15.38:0/2591062262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1940 msgr2=0x55941e3d9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.198+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1940 0x55941e3d9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.199+0000 7ff8a7fff640 1 -- 172.21.15.38:0/2591062262 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55941e3012c0 con 0x55941e3da520 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.199+0000 7ff8ad537640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 0x7ff8980a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.199+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff894009d00 tx=0x7ff894011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8788fcbac6eed3fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.199+0000 7ff892ffd640 1 -- 172.21.15.38:0/2591062262 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894005220 con 0x55941e3da520 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.199+0000 7ff892ffd640 1 -- 172.21.15.38:0/2591062262 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff894004c80 con 0x55941e3da520 2024-04-22T00:04:29.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff892ffd640 1 -- 172.21.15.38:0/2591062262 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894019850 con 0x55941e3da520 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2591062262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 msgr2=0x55941e3dc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8ad537640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 0x7ff8980a5280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0005fd0 tx=0x7ff8a0015040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff894009d00 tx=0x7ff894011810 comp rx=0 tx=0).stop 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2591062262 shutdown_connections 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3da520 0x55941e3dc910 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1940 0x55941e3d9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2591062262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.200+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2591062262 >> 172.21.15.38:0/2591062262 conn(0x55941e3cb750 msgr2=0x55941e3cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.201+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2591062262 shutdown_connections 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.201+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2591062262 wait complete. 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.201+0000 7ff8b1b85740 1 -- start start 2024-04-22T00:04:29.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.201+0000 7ff8b1b85740 1 Processor -- start 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1940 0x55941e3d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55941e3dd360 con 0x55941e3d1020 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55941e2f0f10 con 0x55941e3da520 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55941e3dd4e0 con 0x55941e3d1940 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8b1b85740 10 client.0 _ll_register_callbacks cb 0x55941e048be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46872/0 (socket says 172.21.15.38:46872) 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8acd36640 1 -- 172.21.15.38:0/1658436480 learned_addr learned my addr 172.21.15.38:0/1658436480 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1940 0x55941e3d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.202+0000 7ff8ad537640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 -- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1940 msgr2=0x55941e3d71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1940 0x55941e3d71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 -- 172.21.15.38:0/1658436480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 msgr2=0x55941e3d9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 -- 172.21.15.38:0/1658436480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55941e2c2c80 con 0x55941e3da520 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8acd36640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff8ad537640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3d76f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000fc50 tx=0x7ff8a0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e734da1ae0e65a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.203+0000 7ff887fff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.204+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0010e40 con 0x55941e3da520 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.204+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff8a001d430 con 0x55941e3da520 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.204+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff894004c80 con 0x55941e3da520 2024-04-22T00:04:29.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.205+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8a001d630 con 0x55941e3da520 2024-04-22T00:04:29.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.205+0000 7ff8b1b85740 10 client.26000 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.205+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55941e0493f0 con 0x55941e3da520 2024-04-22T00:04:29.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff894004c80 con 0x55941e3da520 2024-04-22T00:04:29.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7ff8a001c070 con 0x55941e3da520 2024-04-22T00:04:29.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 10 client.26000 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 10 client.26000 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 20 client.26000 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e3ea5d0 0x55941e3eca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 4 client.26000 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.206+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55941e342a70 con 0x55941e3ea5d0 2024-04-22T00:04:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.207+0000 7ff8acd36640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e3ea5d0 0x55941e3eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.207+0000 7ff887fff640 10 client.26000 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.207+0000 7ff8acd36640 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e3ea5d0 0x55941e3eca70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c58d0574c4918c4d server_cookie=59b12553134fa597 in_seq=0 out_seq=0 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff887fff640 1 -- 172.21.15.38:0/1658436480 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55941e342a70 con 0x55941e3ea5d0 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff887fff640 10 client.26000 handle_command_reply: tid=0 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 client.26000 shutdown 2024-04-22T00:04:29.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 msgr2=0x55941e3d76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3d76f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000fc50 tx=0x7ff8a0004ab0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 shutdown_connections 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e3ea5d0 0x55941e3eca70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3d76f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1940 0x55941e3d71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1658436480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1020 0x55941e3d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 >> 172.21.15.38:0/1658436480 conn(0x55941e3cb750 msgr2=0x55941e3cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff887fff640 0 client.26000 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.208+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 shutdown_connections 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.209+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1658436480 wait complete. 2024-04-22T00:04:29.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.209+0000 7ff8b1b85740 20 client.26000 trim_cache size 0 max 16384 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.232+0000 7ff8b1b85740 1 Processor -- start 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 -- start start 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3d6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b50b0 con 0x55941e3d1940 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b6d70 con 0x55941e3da520 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8b1b85740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b5780 con 0x55941e3d1020 2024-04-22T00:04:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50226/0 (socket says 172.21.15.38:50226) 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8acd36640 1 -- 172.21.15.38:0/1734367406 learned_addr learned my addr 172.21.15.38:0/1734367406 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8ad537640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.233+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3d6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 -- 172.21.15.38:0/1734367406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 msgr2=0x55941e3e6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 -- 172.21.15.38:0/1734367406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 msgr2=0x55941e3cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 -- 172.21.15.38:0/1734367406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55941e1137d0 con 0x55941e3d1940 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8acd36640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3d6310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff89401b040 tx=0x7ff894009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6079c99237fe57 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8927fc640 1 -- 172.21.15.38:0/1734367406 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894028070 con 0x55941e3d1940 2024-04-22T00:04:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8927fc640 1 -- 172.21.15.38:0/1734367406 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8940035f0 con 0x55941e3d1940 2024-04-22T00:04:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.234+0000 7ff8927fc640 1 -- 172.21.15.38:0/1734367406 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8940246a0 con 0x55941e3d1940 2024-04-22T00:04:29.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1734367406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 msgr2=0x55941e3d6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3d6310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff89401b040 tx=0x7ff894009930 comp rx=0 tx=0).stop 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1734367406 shutdown_connections 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3da520 0x55941e3cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3d6310 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/1734367406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3d1020 0x55941e3e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1734367406 >> 172.21.15.38:0/1734367406 conn(0x55941e3cb750 msgr2=0x55941e3ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1734367406 shutdown_connections 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- 172.21.15.38:0/1734367406 wait complete. 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 -- start start 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.235+0000 7ff8b1b85740 1 Processor -- start 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3da520 0x55941e3de190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8acd36640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42290/0 (socket says 172.21.15.38:42290) 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46894/0 (socket says 172.21.15.38:46894) 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8acd36640 1 -- 172.21.15.38:0/2911400845 learned_addr learned my addr 172.21.15.38:0/2911400845 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b65e0 con 0x55941e3d1940 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b5f10 con 0x55941e3d1020 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a81b3090 con 0x55941e3da520 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8ad537640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3da520 0x55941e3de190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.236+0000 7ff8b1b85740 10 client.0 _ll_register_callbacks cb 0x7ff8a0009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 -- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3da520 msgr2=0x55941e3de190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3da520 0x55941e3de190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 -- 172.21.15.38:0/2911400845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 msgr2=0x55941e3ddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55941e44eeb0 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8acd36640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0bd640 tx=0x7ff89c0f3040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4caf3ff958667fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff887fff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c0d1e60 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff89c0b9f20 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8940035f0 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff89c0bd2a0 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8b1b85740 10 client.26006 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55941e450190 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.237+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8940035f0 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7ff89c0054d0 con 0x55941e3d1020 2024-04-22T00:04:29.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 10 client.26006 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 10 client.26006 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 20 client.26006 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e450890 0x55941e452cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 4 client.26006 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55941e33cdc0 con 0x55941e450890 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e450890 0x55941e452cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff8a7fff640 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e450890 0x55941e452cd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a26a1fb34109cce server_cookie=9fc65169aaff3613 in_seq=0 out_seq=0 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.238+0000 7ff887fff640 10 client.26006 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff887fff640 1 -- 172.21.15.38:0/2911400845 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55941e33cdc0 con 0x55941e450890 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff887fff640 10 client.26006 handle_command_reply: tid=0 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 client.26006 shutdown 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 msgr2=0x55941e3dd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0bd640 tx=0x7ff89c0f3040 comp rx=0 tx=0).stop 2024-04-22T00:04:29.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 shutdown_connections 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55941e450890 0x55941e452cd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff887fff640 0 client.26006 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55941e3da520 0x55941e3de190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55941e3d1940 0x55941e3ddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 --2- 172.21.15.38:0/2911400845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55941e3d1020 0x55941e3dd6f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 >> 172.21.15.38:0/2911400845 conn(0x55941e3cb750 msgr2=0x55941e3ccb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 shutdown_connections 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 1 -- 172.21.15.38:0/2911400845 wait complete. 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.239+0000 7ff8b1b85740 20 client.26006 trim_cache size 0 max 16384 2024-04-22T00:04:29.238 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:04:29.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 msgr2=0x7ff8980a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 0x7ff8980a5280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0005fd0 tx=0x7ff8a0015040 comp rx=0 tx=0).stop 2024-04-22T00:04:29.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 msgr2=0x7ff8a810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 0x7ff8a810e220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff89c036040 tx=0x7ff89c004270 comp rx=0 tx=0).stop 2024-04-22T00:04:29.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 shutdown_connections 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff8980a2dc0 0x7ff8980a5280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8a8107720 0x7ff8a810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8a81020d0 0x7ff8a810b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 --2- 172.21.15.38:0/785364873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8a8101710 0x7ff8a810e220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 >> 172.21.15.38:0/785364873 conn(0x7ff8a8078b90 msgr2=0x7ff8a80fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 shutdown_connections 2024-04-22T00:04:29.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.240+0000 7ff8aef9a640 1 -- 172.21.15.38:0/785364873 wait complete. 2024-04-22T00:04:29.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:04:29.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 -- 172.21.15.38:0/1605442749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec1066b0 msgr2=0x7f17ec106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 --2- 172.21.15.38:0/1605442749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec1066b0 0x7f17ec106a90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009930 tx=0x7f17dc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:04:29.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 -- 172.21.15.38:0/1605442749 shutdown_connections 2024-04-22T00:04:29.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 --2- 172.21.15.38:0/1605442749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec100fd0 0x7f17ec10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 --2- 172.21.15.38:0/1605442749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100610 0x7f17ec100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 --2- 172.21.15.38:0/1605442749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec1066b0 0x7f17ec106a90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 -- 172.21.15.38:0/1605442749 >> 172.21.15.38:0/1605442749 conn(0x7f17ec077740 msgr2=0x7f17ec077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 -- 172.21.15.38:0/1605442749 shutdown_connections 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.372+0000 7f17f153e640 1 -- 172.21.15.38:0/1605442749 wait complete. 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 Processor -- start 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 -- start start 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100fd0 0x7f17ec19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1066b0 0x7f17ec19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec110a80 con 0x7f17ec100610 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec110900 con 0x7f17ec100fd0 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.373+0000 7f17f153e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec110c00 con 0x7f17ec1066b0 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100fd0 0x7f17ec19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46910/0 (socket says 172.21.15.38:46910) 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 -- 172.21.15.38:0/465188003 learned_addr learned my addr 172.21.15.38:0/465188003 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1066b0 0x7f17ec19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 -- 172.21.15.38:0/465188003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1066b0 msgr2=0x7f17ec19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1066b0 0x7f17ec19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 -- 172.21.15.38:0/465188003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100fd0 msgr2=0x7f17ec19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100fd0 0x7f17ec19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 -- 172.21.15.38:0/465188003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec19f190 con 0x7f17ec100610 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17eaffd640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009900 tx=0x7f17dc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17d3fff640 1 -- 172.21.15.38:0/465188003 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc043020 con 0x7f17ec100610 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec19f420 con 0x7f17ec100610 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17d3fff640 1 -- 172.21.15.38:0/465188003 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc0054e0 con 0x7f17ec100610 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17d3fff640 1 -- 172.21.15.38:0/465188003 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc003bb0 con 0x7f17ec100610 2024-04-22T00:04:29.369 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.374+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1a6d10 con 0x7f17ec100610 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17d3fff640 1 -- 172.21.15.38:0/465188003 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f17dc046020 con 0x7f17ec100610 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 Processor -- start 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 -- start start 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a407c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a4084fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 0x5642a4087910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642a3fe0530 con 0x5642a407c940 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5642a3f88830 con 0x5642a407c020 2024-04-22T00:04:29.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17f4207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5642a3fdc4f0 con 0x5642a4085520 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17d3fff640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a4084fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a407c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 0x5642a4087910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a407c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42320/0 (socket says 172.21.15.38:42320) 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17eb7fe640 1 -- 172.21.15.38:0/2701691657 learned_addr learned my addr 172.21.15.38:0/2701691657 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17d3fff640 1 -- 172.21.15.38:0/465188003 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f17dc0f1650 con 0x7f17ec100610 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.377+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 -- 172.21.15.38:0/2701691657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 msgr2=0x5642a407c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a407c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 -- 172.21.15.38:0/2701691657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 msgr2=0x5642a4084fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a4084fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 -- 172.21.15.38:0/2701691657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a3fac2c0 con 0x5642a4085520 2024-04-22T00:04:29.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 0x5642a4087910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009bf0 tx=0x7f17e0011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d7155500185a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 0x7f17d40a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17e4005f40 tx=0x7f17e4009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17d17fa640 1 -- 172.21.15.38:0/2701691657 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0004310 con 0x5642a4085520 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17d17fa640 1 -- 172.21.15.38:0/2701691657 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e00044b0 con 0x5642a4085520 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.378+0000 7f17d17fa640 1 -- 172.21.15.38:0/2701691657 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0003850 con 0x5642a4085520 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- 172.21.15.38:0/2701691657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 msgr2=0x5642a4087910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 0x5642a4087910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009bf0 tx=0x7f17e0011440 comp rx=0 tx=0).stop 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- 172.21.15.38:0/2701691657 shutdown_connections 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4085520 0x5642a4087910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a4084fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 --2- 172.21.15.38:0/2701691657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a407c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- 172.21.15.38:0/2701691657 >> 172.21.15.38:0/2701691657 conn(0x5642a4076750 msgr2=0x5642a4076b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- 172.21.15.38:0/2701691657 shutdown_connections 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- 172.21.15.38:0/2701691657 wait complete. 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 -- start start 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.379+0000 7f17f4207740 1 Processor -- start 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 0x5642a40826b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642a4088310 con 0x5642a4085520 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5642a3f9bf10 con 0x5642a407c940 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5642a4088490 con 0x5642a407c020 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17f4207740 10 client.0 _ll_register_callbacks cb 0x5642a3cf3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50266/0 (socket says 172.21.15.38:50266) 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42334/0 (socket says 172.21.15.38:42334) 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 -- 172.21.15.38:0/429867322 learned_addr learned my addr 172.21.15.38:0/429867322 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 0x5642a40826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 -- 172.21.15.38:0/429867322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 msgr2=0x5642a4082170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 -- 172.21.15.38:0/429867322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 msgr2=0x5642a40826b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 0x5642a40826b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 -- 172.21.15.38:0/429867322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642a3f6dc80 con 0x5642a407c020 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 0x5642a40826b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17eaffd640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0ba310 tx=0x7f17dc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b68cbc08ca3cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.380+0000 7f17ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0d5020 con 0x5642a407c020 2024-04-22T00:04:29.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f17dc0b9fb0 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e00044b0 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f17dc0d1e60 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 10 client.25900 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642a3cf43f0 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e00044b0 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f17dc0d61f0 con 0x5642a407c020 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 10 client.25900 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 10 client.25900 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 20 client.25900 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4095510 0x5642a4097980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 4 client.25900 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.381+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642a3feda70 con 0x5642a4095510 2024-04-22T00:04:29.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.382+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4095510 0x5642a4097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.382+0000 7f17ceff5640 10 client.25900 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:04:29.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.382+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4095510 0x5642a4097980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75c53e2579d2e863 server_cookie=1dbd0826d3d0b06e in_seq=0 out_seq=0 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17ceff5640 1 -- 172.21.15.38:0/429867322 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642a3feda70 con 0x5642a4095510 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17ceff5640 10 client.25900 handle_command_reply: tid=0 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 client.25900 shutdown 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 msgr2=0x5642a4084de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0ba310 tx=0x7f17dc009260 comp rx=0 tx=0).stop 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 shutdown_connections 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4095510 0x5642a4097980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a4085520 0x5642a40826b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c940 0x5642a4082170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17ceff5640 0 client.25900 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 --2- 172.21.15.38:0/429867322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c020 0x5642a4084de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 >> 172.21.15.38:0/429867322 conn(0x5642a4076750 msgr2=0x5642a40784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 shutdown_connections 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 1 -- 172.21.15.38:0/429867322 wait complete. 2024-04-22T00:04:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.383+0000 7f17f4207740 20 client.25900 trim_cache size 0 max 16384 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 Processor -- start 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 -- start start 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 0x5642a40911d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1b0af0 con 0x5642a407c020 2024-04-22T00:04:29.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1b27b0 con 0x5642a4076750 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17f4207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1b11c0 con 0x5642a407c940 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50272/0 (socket says 172.21.15.38:50272) 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eb7fe640 1 -- 172.21.15.38:0/3004552896 learned_addr learned my addr 172.21.15.38:0/3004552896 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46944/0 (socket says 172.21.15.38:46944) 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 0x5642a40911d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 -- 172.21.15.38:0/3004552896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 msgr2=0x5642a40848a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 -- 172.21.15.38:0/3004552896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 msgr2=0x5642a407d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eaffd640 1 -- 172.21.15.38:0/3004552896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a3dbe7d0 con 0x5642a4076750 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.405+0000 7f17ea7fc640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.406+0000 7f17eaffd640 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 0x5642a40911d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f17dc006010 tx=0x7f17dc005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d00f6f940c40b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.406+0000 7f17d0ff9640 1 -- 172.21.15.38:0/3004552896 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0d5050 con 0x5642a4076750 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.406+0000 7f17d0ff9640 1 -- 172.21.15.38:0/3004552896 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc004c80 con 0x5642a4076750 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.406+0000 7f17d0ff9640 1 -- 172.21.15.38:0/3004552896 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0c65b0 con 0x5642a4076750 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- 172.21.15.38:0/3004552896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 msgr2=0x5642a40911d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 0x5642a40911d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f17dc006010 tx=0x7f17dc005be0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- 172.21.15.38:0/3004552896 shutdown_connections 2024-04-22T00:04:29.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a407c940 0x5642a40848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c020 0x5642a407d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- 172.21.15.38:0/3004552896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a4076750 0x5642a40911d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- 172.21.15.38:0/3004552896 >> 172.21.15.38:0/3004552896 conn(0x5642a4082810 msgr2=0x5642a407ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- 172.21.15.38:0/3004552896 shutdown_connections 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- 172.21.15.38:0/3004552896 wait complete. 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- start start 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 Processor -- start 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4076750 0x5642a40886a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a4088be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1b2020 con 0x5642a407c940 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1b1950 con 0x5642a407c020 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec1aead0 con 0x5642a4076750 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.407+0000 7f17f4207740 10 client.0 _ll_register_callbacks cb 0x7f17dc0b9c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46954/0 (socket says 172.21.15.38:46954) 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 -- 172.21.15.38:0/1647730268 learned_addr learned my addr 172.21.15.38:0/1647730268 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17ea7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a4088be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eaffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4076750 0x5642a40886a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 -- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4076750 msgr2=0x5642a40886a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4076750 0x5642a40886a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 -- 172.21.15.38:0/1647730268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 msgr2=0x5642a4088be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a4088be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 -- 172.21.15.38:0/1647730268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642a40f9fd0 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17eb7fe640 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f17e4003e90 tx=0x7f17e4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fda1bef11596d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004b20 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17dc004c80 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f17e4004cc0 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f17e4035050 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17f4207740 10 client.16617 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.408+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642a40fb180 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17dc004c80 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f17e401d070 con 0x5642a407c940 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 10 client.16617 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 10 client.16617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 20 client.16617 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4085520 0x5642a40fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 4 client.16617 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642a3fe7dc0 con 0x5642a4085520 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17eaffd640 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4085520 0x5642a40fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17ceff5640 10 client.16617 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.409+0000 7f17eaffd640 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4085520 0x5642a40fd8b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6889fc84541c3da1 server_cookie=b16b65fdbe3f389a in_seq=0 out_seq=0 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17ceff5640 1 -- 172.21.15.38:0/1647730268 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642a3fe7dc0 con 0x5642a4085520 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17ceff5640 10 client.16617 handle_command_reply: tid=0 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 client.16617 shutdown 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 msgr2=0x5642a40f9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f17e4003e90 tx=0x7f17e4004ab0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 shutdown_connections 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5642a4085520 0x5642a40fd8b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642a407c940 0x5642a40f9a90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5642a407c020 0x5642a4088be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17ceff5640 0 client.16617 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 --2- 172.21.15.38:0/1647730268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5642a4076750 0x5642a40886a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 >> 172.21.15.38:0/1647730268 conn(0x5642a4082810 msgr2=0x5642a4075850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 shutdown_connections 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 1 -- 172.21.15.38:0/1647730268 wait complete. 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f4207740 20 client.16617 trim_cache size 0 max 16384 2024-04-22T00:04:29.411 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 msgr2=0x7f17d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 0x7f17d40a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17e4005f40 tx=0x7f17e4009500 comp rx=0 tx=0).stop 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 msgr2=0x7f17ec19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.410+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009900 tx=0x7f17dc004360 comp rx=0 tx=0).stop 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 shutdown_connections 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1066b0 0x7f17ec19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17ec100fd0 0x7f17ec19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 --2- 172.21.15.38:0/465188003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17ec100610 0x7f17ec19a1a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 >> 172.21.15.38:0/465188003 conn(0x7f17ec077740 msgr2=0x7f17ec0688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 shutdown_connections 2024-04-22T00:04:29.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.411+0000 7f17f153e640 1 -- 172.21.15.38:0/465188003 wait complete. 2024-04-22T00:04:29.414 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:29 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1417145963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:04:29.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:04:29.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:29 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1417145963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:04:29.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 -- 172.21.15.38:0/1408980603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 msgr2=0x7fcbac10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/1408980603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac10e1a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009ca0 tx=0x7fcba402c230 comp rx=0 tx=0).stop 2024-04-22T00:04:29.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 -- 172.21.15.38:0/1408980603 shutdown_connections 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/1408980603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac10e1a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/1408980603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1032e0 0x7fcbac10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/1408980603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1029c0 0x7fcbac102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.542+0000 7fcbb33f5640 1 -- 172.21.15.38:0/1408980603 >> 172.21.15.38:0/1408980603 conn(0x7fcbac077740 msgr2=0x7fcbac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.543+0000 7fcbb33f5640 1 -- 172.21.15.38:0/1408980603 shutdown_connections 2024-04-22T00:04:29.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.543+0000 7fcbb33f5640 1 -- 172.21.15.38:0/1408980603 wait complete. 2024-04-22T00:04:29.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 Processor -- start 2024-04-22T00:04:29.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 -- start start 2024-04-22T00:04:29.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 0x7fcbac198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac111b00 con 0x7fcbac10bd00 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac111980 con 0x7fcbac1032e0 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb33f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac111c80 con 0x7fcbac1029c0 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50290/0 (socket says 172.21.15.38:50290) 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1191640 1 -- 172.21.15.38:0/4263091247 learned_addr learned my addr 172.21.15.38:0/4263091247 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 0x7fcbac198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1992640 1 -- 172.21.15.38:0/4263091247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 msgr2=0x7fcbac197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1992640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1992640 1 -- 172.21.15.38:0/4263091247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 msgr2=0x7fcbac198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.544+0000 7fcbb1992640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 0x7fcbac198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb1992640 1 -- 172.21.15.38:0/4263091247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac19cfb0 con 0x7fcbac10bd00 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb1191640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb0990640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 0x7fcbac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb1992640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac19c880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009bf0 tx=0x7fcba4035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcba27fc640 1 -- 172.21.15.38:0/4263091247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba4039040 con 0x7fcbac10bd00 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbac19d240 con 0x7fcbac10bd00 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcba27fc640 1 -- 172.21.15.38:0/4263091247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba4044070 con 0x7fcbac10bd00 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcba27fc640 1 -- 172.21.15.38:0/4263091247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba40042a0 con 0x7fcbac10bd00 2024-04-22T00:04:29.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.545+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbac100540 con 0x7fcbac10bd00 2024-04-22T00:04:29.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcba27fc640 1 -- 172.21.15.38:0/4263091247 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcba4032d00 con 0x7fcbac10bd00 2024-04-22T00:04:29.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 Processor -- start 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 -- start start 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e020 0x55f4a6d3e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d49910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6ca2530 con 0x55f4a6d47520 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6c4a830 con 0x55f4a6d3e940 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.548+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6c9e4f0 con 0x55f4a6d3e020 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcba27fc640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 0x7fcb980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42358/0 (socket says 172.21.15.38:42358) 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d49910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 -- 172.21.15.38:0/2016104153 learned_addr learned my addr 172.21.15.38:0/2016104153 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb0990640 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e020 0x55f4a6d3e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1992640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 0x7fcb980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 -- 172.21.15.38:0/2016104153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e020 msgr2=0x55f4a6d3e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e020 0x55f4a6d3e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 -- 172.21.15.38:0/2016104153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 msgr2=0x55f4a6d49910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d49910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.549+0000 7fcbb1191640 1 -- 172.21.15.38:0/2016104153 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a6c6e2c0 con 0x55f4a6d3e940 2024-04-22T00:04:29.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.550+0000 7fcba27fc640 1 -- 172.21.15.38:0/4263091247 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcba40f1940 con 0x7fcbac10bd00 2024-04-22T00:04:29.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.550+0000 7fcbb1992640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 0x7fcb980a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcba4004af0 tx=0x7fcba4045c40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.551+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcba8011650 tx=0x7fcba8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fedae8cdb06fe29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.552+0000 7fcb977fe640 1 -- 172.21.15.38:0/2016104153 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba801e070 con 0x55f4a6d3e940 2024-04-22T00:04:29.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.552+0000 7fcb977fe640 1 -- 172.21.15.38:0/2016104153 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba800fba0 con 0x55f4a6d3e940 2024-04-22T00:04:29.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.552+0000 7fcb977fe640 1 -- 172.21.15.38:0/2016104153 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8019cd0 con 0x55f4a6d3e940 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2016104153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 msgr2=0x55f4a6d46fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcba8011650 tx=0x7fcba8009f90 comp rx=0 tx=0).stop 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2016104153 shutdown_connections 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d49910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46fe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2016104153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e020 0x55f4a6d3e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2016104153 >> 172.21.15.38:0/2016104153 conn(0x55f4a6d38750 msgr2=0x55f4a6d38b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2016104153 shutdown_connections 2024-04-22T00:04:29.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.553+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2016104153 wait complete. 2024-04-22T00:04:29.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 -- start start 2024-04-22T00:04:29.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 Processor -- start 2024-04-22T00:04:29.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e940 0x55f4a6d442d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d448e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6d4a310 con 0x55f4a6d3e940 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6c5df10 con 0x55f4a6d3e020 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f4a6d4a490 con 0x55f4a6d47520 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb5fdb740 10 client.0 _ll_register_callbacks cb 0x55f4a69b5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e940 0x55f4a6d442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42362/0 (socket says 172.21.15.38:42362) 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb0990640 1 -- 172.21.15.38:0/1291365606 learned_addr learned my addr 172.21.15.38:0/1291365606 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.554+0000 7fcbb0990640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 -- 172.21.15.38:0/1291365606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 msgr2=0x55f4a6d448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 -- 172.21.15.38:0/1291365606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e940 msgr2=0x55f4a6d442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e940 0x55f4a6d442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 -- 172.21.15.38:0/1291365606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a6c2fc80 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb0990640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d50 tx=0x7fcb9c012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d160032aa78d62 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcb94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c020e60 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcb9c005020 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba800fba0 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb9c031050 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb5fdb740 10 client.26036 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.555+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4a69b63f0 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba800fba0 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcb9c004c00 con 0x55f4a6d3e020 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 10 client.26036 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 10 client.26036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:29.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 20 client.26036 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6d57730 0x55f4a6d59b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 4 client.26036 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:29.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.556+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4a6cafa70 con 0x55f4a6d57730 2024-04-22T00:04:29.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.557+0000 7fcbb1191640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6d57730 0x55f4a6d59b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.557+0000 7fcb94ff9640 10 client.26036 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.557+0000 7fcbb1191640 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6d57730 0x55f4a6d59b60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a3916a42b55142 server_cookie=fc16a476f4b8441d in_seq=0 out_seq=0 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcb94ff9640 1 -- 172.21.15.38:0/1291365606 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4a6cafa70 con 0x55f4a6d57730 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcb94ff9640 10 client.26036 handle_command_reply: tid=0 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 client.26036 shutdown 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 msgr2=0x55f4a6d46f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d50 tx=0x7fcb9c012040 comp rx=0 tx=0).stop 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 shutdown_connections 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.558+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6d57730 0x55f4a6d59b60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e940 0x55f4a6d442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/1291365606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d46f40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcb94ff9640 0 client.26036 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 >> 172.21.15.38:0/1291365606 conn(0x55f4a6d38750 msgr2=0x55f4a6d3a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 shutdown_connections 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/1291365606 wait complete. 2024-04-22T00:04:29.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.559+0000 7fcbb5fdb740 20 client.26036 trim_cache size 0 max 16384 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 Processor -- start 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 -- start start 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1ac8a0 con 0x55f4a6d47520 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1ae560 con 0x55f4a6d3e020 2024-04-22T00:04:29.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.573+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1acf70 con 0x55f4a6d3e940 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47004/0 (socket says 172.21.15.38:47004) 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb1191640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50318/0 (socket says 172.21.15.38:50318) 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 -- 172.21.15.38:0/461150268 learned_addr learned my addr 172.21.15.38:0/461150268 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 -- 172.21.15.38:0/461150268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 msgr2=0x55f4a6d46a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 -- 172.21.15.38:0/461150268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 msgr2=0x55f4a6d531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 -- 172.21.15.38:0/461150268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a6a807d0 con 0x55f4a6d47520 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb1191640 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:29.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcbb0990640 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x55f4a6d45870 tx=0x7fcb9c024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d08f6cd5ae3d585 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcb96ffd640 1 -- 172.21.15.38:0/461150268 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c020e30 con 0x55f4a6d47520 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcb96ffd640 1 -- 172.21.15.38:0/461150268 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb9c009720 con 0x55f4a6d47520 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.574+0000 7fcb96ffd640 1 -- 172.21.15.38:0/461150268 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c004a00 con 0x55f4a6d47520 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/461150268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 msgr2=0x55f4a6d46f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x55f4a6d45870 tx=0x7fcb9c024040 comp rx=0 tx=0).stop 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/461150268 shutdown_connections 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d47520 0x55f4a6d46f40 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d3e940 0x55f4a6d46a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/461150268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e020 0x55f4a6d531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/461150268 >> 172.21.15.38:0/461150268 conn(0x55f4a6d38750 msgr2=0x55f4a6d37850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/461150268 shutdown_connections 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.575+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/461150268 wait complete. 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 -- start start 2024-04-22T00:04:29.570 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 Processor -- start 2024-04-22T00:04:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d41080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1addd0 con 0x55f4a6d3e020 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1ad700 con 0x55f4a6d3e940 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac1aa880 con 0x55f4a6d47520 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb5fdb740 10 client.0 _ll_register_callbacks cb 0x7fcb9c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb0990640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47012/0 (socket says 172.21.15.38:47012) 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb0990640 1 -- 172.21.15.38:0/2741075412 learned_addr learned my addr 172.21.15.38:0/2741075412 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb0990640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.576+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 -- 172.21.15.38:0/2741075412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 msgr2=0x55f4a6d41080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d41080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 -- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 msgr2=0x55f4a6d40b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d40b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 -- 172.21.15.38:0/2741075412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a6d415c0 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb0990640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d50 tx=0x7fcb9c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef39210d62ac391 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcb94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c04b060 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcb9c0035f0 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb9c009720 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb9c0037f0 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb5fdb740 10 client.16635 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4a6dbd980 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.577+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb9c009720 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcb9c01eb90 con 0x55f4a6d3e020 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 10 client.16635 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 20 client.16635 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6dbe0b0 0x55f4a6dc04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 4 client.16635 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4a6ca9dc0 con 0x55f4a6dbe0b0 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6dbe0b0 0x55f4a6dc04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcb94ff9640 10 client.16635 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb1191640 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6dbe0b0 0x55f4a6dc04d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b61f329e9bf1578 server_cookie=a92f4bab1b671b29 in_seq=0 out_seq=0 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcb94ff9640 1 -- 172.21.15.38:0/2741075412 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4a6ca9dc0 con 0x55f4a6dbe0b0 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcb94ff9640 10 client.16635 handle_command_reply: tid=0 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 1 client.16635 shutdown 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 msgr2=0x55f4a6d4f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.578+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009d50 tx=0x7fcb9c005570 comp rx=0 tx=0).stop 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 shutdown_connections 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f4a6d47520 0x55f4a6d41080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4a6d3e940 0x55f4a6d40b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f4a6dbe0b0 0x55f4a6dc04d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 --2- 172.21.15.38:0/2741075412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4a6d3e020 0x55f4a6d4f8d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcb94ff9640 0 client.16635 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 >> 172.21.15.38:0/2741075412 conn(0x55f4a6d38750 msgr2=0x55f4a6d3ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 shutdown_connections 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 1 -- 172.21.15.38:0/2741075412 wait complete. 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb5fdb740 20 client.16635 trim_cache size 0 max 16384 2024-04-22T00:04:29.578 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:04:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 msgr2=0x7fcb980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 0x7fcb980a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcba4004af0 tx=0x7fcba4045c40 comp rx=0 tx=0).stop 2024-04-22T00:04:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 msgr2=0x7fcbac19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac19c880 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009bf0 tx=0x7fcba4035d10 comp rx=0 tx=0).stop 2024-04-22T00:04:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 shutdown_connections 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb980a2d00 0x7fcb980a51c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbac10bd00 0x7fcbac19c880 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbac1032e0 0x7fcbac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 --2- 172.21.15.38:0/4263091247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbac1029c0 0x7fcbac197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 >> 172.21.15.38:0/4263091247 conn(0x7fcbac077740 msgr2=0x7fcbac0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 shutdown_connections 2024-04-22T00:04:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:29.579+0000 7fcbb33f5640 1 -- 172.21.15.38:0/4263091247 wait complete. 2024-04-22T00:04:29.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:29 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1417145963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:04:30.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:30 smithi038 ceph-mon[27972]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.35k op/s 2024-04-22T00:04:30.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:30 smithi203 ceph-mon[31431]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.35k op/s 2024-04-22T00:04:30.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:30 smithi111 ceph-mon[31543]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.35k op/s 2024-04-22T00:04:30.584 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:04:30.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:04:30.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:04:30.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 -- 172.21.15.38:0/1650869092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 msgr2=0x7f908810e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 --2- 172.21.15.38:0/1650869092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f908810e5d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9080009cd0 tx=0x7f908002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 -- 172.21.15.38:0/1650869092 shutdown_connections 2024-04-22T00:04:30.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 --2- 172.21.15.38:0/1650869092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f908810e5d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 --2- 172.21.15.38:0/1650869092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f9088101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 --2- 172.21.15.38:0/1650869092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 0x7f9088107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 -- 172.21.15.38:0/1650869092 >> 172.21.15.38:0/1650869092 conn(0x7f90880fd1b0 msgr2=0x7f90880ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.751+0000 7f908edf5640 1 -- 172.21.15.38:0/1650869092 shutdown_connections 2024-04-22T00:04:30.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.752+0000 7f908edf5640 1 -- 172.21.15.38:0/1650869092 wait complete. 2024-04-22T00:04:30.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.752+0000 7f908edf5640 1 Processor -- start 2024-04-22T00:04:30.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.752+0000 7f908edf5640 1 -- start start 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f90881a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 0x7f90881ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f908810f910 con 0x7f90881020c0 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f908810f790 con 0x7f9088101700 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f908edf5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f908810fa90 con 0x7f9088107710 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f9087fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f9087fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47036/0 (socket says 172.21.15.38:47036) 2024-04-22T00:04:30.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.753+0000 7f9087fff640 1 -- 172.21.15.38:0/4070911713 learned_addr learned my addr 172.21.15.38:0/4070911713 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f908d392640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 0x7f90881ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f908cb91640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f90881a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 -- 172.21.15.38:0/4070911713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 msgr2=0x7f90881ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 0x7f90881ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 -- 172.21.15.38:0/4070911713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 msgr2=0x7f90881a6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f90881a6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 -- 172.21.15.38:0/4070911713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90881abdf0 con 0x7f90881020c0 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f908cb91640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f90881a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.754+0000 7f9087fff640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f907c00bc70 tx=0x7f907c008600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.755+0000 7f9085ffb640 1 -- 172.21.15.38:0/4070911713 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c008d10 con 0x7f90881020c0 2024-04-22T00:04:30.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.755+0000 7f9085ffb640 1 -- 172.21.15.38:0/4070911713 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c0158e0 con 0x7f90881020c0 2024-04-22T00:04:30.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.755+0000 7f9085ffb640 1 -- 172.21.15.38:0/4070911713 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c014840 con 0x7f90881020c0 2024-04-22T00:04:30.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.755+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90881ac0e0 con 0x7f90881020c0 2024-04-22T00:04:30.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.756+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90881b3970 con 0x7f90881020c0 2024-04-22T00:04:30.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.759+0000 7f90919d9740 1 Processor -- start 2024-04-22T00:04:30.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 -- start start 2024-04-22T00:04:30.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f9085ffb640 1 -- 172.21.15.38:0/4070911713 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f907c015a90 con 0x7f90881020c0 2024-04-22T00:04:30.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557629304170 con 0x5576293a1720 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5576292ac7c0 con 0x5576293a9db0 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f90919d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557629300130 con 0x5576293a0a40 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.760+0000 7f9087fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42410/0 (socket says 172.21.15.38:42410) 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 -- 172.21.15.38:0/3132081716 learned_addr learned my addr 172.21.15.38:0/3132081716 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f908d392640 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 -- 172.21.15.38:0/3132081716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 msgr2=0x5576293a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 -- 172.21.15.38:0/3132081716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 msgr2=0x5576293a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.761+0000 7f9087fff640 1 -- 172.21.15.38:0/3132081716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576292bde60 con 0x5576293a9db0 2024-04-22T00:04:30.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.762+0000 7f9085ffb640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 0x7f90700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.762+0000 7f9087fff640 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f907c0cc590 tx=0x7f907c0d1530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61dde9f1d8dde0f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.763+0000 7f9085ffb640 1 -- 172.21.15.38:0/4070911713 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f907c0d3060 con 0x7f90881020c0 2024-04-22T00:04:30.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.763+0000 7f906effd640 1 -- 172.21.15.38:0/3132081716 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c097690 con 0x5576293a9db0 2024-04-22T00:04:30.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.763+0000 7f906effd640 1 -- 172.21.15.38:0/3132081716 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c0d2020 con 0x5576293a9db0 2024-04-22T00:04:30.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.763+0000 7f906effd640 1 -- 172.21.15.38:0/3132081716 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c09ad50 con 0x5576293a9db0 2024-04-22T00:04:30.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f908d392640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 0x7f90700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 -- 172.21.15.38:0/3132081716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 msgr2=0x5576293ac1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f907c0cc590 tx=0x7f907c0d1530 comp rx=0 tx=0).stop 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f908d392640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 0x7f90700a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9080006010 tx=0x7f9080005d80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 -- 172.21.15.38:0/3132081716 shutdown_connections 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ac1a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.764+0000 7f90919d9740 1 --2- 172.21.15.38:0/3132081716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.765+0000 7f90919d9740 1 -- 172.21.15.38:0/3132081716 >> 172.21.15.38:0/3132081716 conn(0x5576293674f0 msgr2=0x55762939b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.765+0000 7f90919d9740 1 -- 172.21.15.38:0/3132081716 shutdown_connections 2024-04-22T00:04:30.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.765+0000 7f90919d9740 1 -- 172.21.15.38:0/3132081716 wait complete. 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.765+0000 7f90919d9740 1 -- start start 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 Processor -- start 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293a6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 0x5576293a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5576293acb60 con 0x5576293a1720 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5576292bfd40 con 0x5576293a0a40 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5576293acce0 con 0x5576293a9db0 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f90919d9740 10 client.0 _ll_register_callbacks cb 0x5576292fb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f908cb91640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293a6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f908d392640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 0x5576293a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f908cb91640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293a6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42412/0 (socket says 172.21.15.38:42412) 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f908d392640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 0x5576293a78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50372/0 (socket says 172.21.15.38:50372) 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f908cb91640 1 -- 172.21.15.38:0/3394909758 learned_addr learned my addr 172.21.15.38:0/3394909758 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.766+0000 7f9087fff640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 -- 172.21.15.38:0/3394909758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 msgr2=0x5576293a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 0x5576293a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 -- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 msgr2=0x5576293a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293a6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 -- 172.21.15.38:0/3394909758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576290e07e0 con 0x5576293a1720 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9087fff640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a73a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f907c09b0f0 tx=0x7f907c00d540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbe577633c96b91 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9067fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:30.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c09af30 con 0x5576293a1720 2024-04-22T00:04:30.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.767+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f907c097a60 con 0x5576293a1720 2024-04-22T00:04:30.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f907c0d2020 con 0x5576293a1720 2024-04-22T00:04:30.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f907c09b1c0 con 0x5576293a1720 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f90919d9740 10 client.16647 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557629140f70 con 0x5576293a1720 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c0d2020 con 0x5576293a1720 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.768+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f907c0d1c70 con 0x5576293a1720 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 10 client.16647 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 20 client.16647 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576293b9d40 0x5576293bc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 4 client.16647 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.769+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576292d8450 con 0x5576293b9d40 2024-04-22T00:04:30.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.770+0000 7f908cb91640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576293b9d40 0x5576293bc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.771+0000 7f908cb91640 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576293b9d40 0x5576293bc130 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff06eaec423ac598 server_cookie=29dffa2eb7c4cb31 in_seq=0 out_seq=0 2024-04-22T00:04:30.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.771+0000 7f9067fff640 10 client.16647 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:30.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.772+0000 7f9067fff640 1 -- 172.21.15.38:0/3394909758 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576292d8450 con 0x5576293b9d40 2024-04-22T00:04:30.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.772+0000 7f9067fff640 10 client.16647 handle_command_reply: tid=0 2024-04-22T00:04:30.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.773+0000 7f90919d9740 1 client.16647 shutdown 2024-04-22T00:04:30.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.773+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 msgr2=0x5576293a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.773+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a73a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f907c09b0f0 tx=0x7f907c00d540 comp rx=0 tx=0).stop 2024-04-22T00:04:30.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 shutdown_connections 2024-04-22T00:04:30.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a9db0 0x5576293a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576293b9d40 0x5576293bc130 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a1720 0x5576293a73a0 secure :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f907c09b0f0 tx=0x7f907c00d540 comp rx=0 tx=0).stop 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 --2- 172.21.15.38:0/3394909758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293a6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 >> 172.21.15.38:0/3394909758 conn(0x5576293674f0 msgr2=0x55762939d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f9067fff640 0 client.16647 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:30.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 shutdown_connections 2024-04-22T00:04:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 1 -- 172.21.15.38:0/3394909758 wait complete. 2024-04-22T00:04:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.774+0000 7f90919d9740 20 client.16647 trim_cache size 0 max 16384 2024-04-22T00:04:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.804+0000 7f90919d9740 1 Processor -- start 2024-04-22T00:04:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.804+0000 7f90919d9740 1 -- start start 2024-04-22T00:04:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.804+0000 7f90919d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.804+0000 7f90919d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.804+0000 7f90919d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f908d392640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f908d392640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47054/0 (socket says 172.21.15.38:47054) 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f908cb91640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f908cb91640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50388/0 (socket says 172.21.15.38:50388) 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f9087fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f9087fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42422/0 (socket says 172.21.15.38:42422) 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f908d392640 1 -- 172.21.15.38:0/2111020130 learned_addr learned my addr 172.21.15.38:0/2111020130 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90881bdab0 con 0x5576293a5da0 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90881bf770 con 0x5576293a9db0 2024-04-22T00:04:30.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.805+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f90881be180 con 0x5576293a0a40 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 -- 172.21.15.38:0/2111020130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 msgr2=0x5576293ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 -- 172.21.15.38:0/2111020130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 msgr2=0x5576293ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 -- 172.21.15.38:0/2111020130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557628fd8170 con 0x5576293a0a40 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908d392640 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f9087fff640 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:30.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.807+0000 7f908cb91640 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5576292ac7c0 tx=0x7f9078005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aba9a3d56dfc400 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.808+0000 7f906e7fc640 1 -- 172.21.15.38:0/2111020130 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078013110 con 0x5576293a0a40 2024-04-22T00:04:30.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.808+0000 7f906e7fc640 1 -- 172.21.15.38:0/2111020130 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5576292d8450 con 0x5576293a0a40 2024-04-22T00:04:30.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.808+0000 7f906e7fc640 1 -- 172.21.15.38:0/2111020130 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907801b590 con 0x5576293a0a40 2024-04-22T00:04:30.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 msgr2=0x5576293a9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5576292ac7c0 tx=0x7f9078005f90 comp rx=0 tx=0).stop 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 shutdown_connections 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a5da0 0x5576293ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a9db0 0x5576293ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 --2- 172.21.15.38:0/2111020130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a0a40 0x5576293a9590 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 >> 172.21.15.38:0/2111020130 conn(0x5576293674f0 msgr2=0x55762939a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 shutdown_connections 2024-04-22T00:04:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.809+0000 7f90919d9740 1 -- 172.21.15.38:0/2111020130 wait complete. 2024-04-22T00:04:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.810+0000 7f90919d9740 1 -- start start 2024-04-22T00:04:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.810+0000 7f90919d9740 1 Processor -- start 2024-04-22T00:04:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.811+0000 7f90919d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.811+0000 7f90919d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 0x5576293bc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.811+0000 7f90919d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a9db0 0x5576293bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.811+0000 7f90919d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f90881befe0 con 0x5576293a9db0 2024-04-22T00:04:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f90919d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f90881be910 con 0x5576293a0a40 2024-04-22T00:04:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f90919d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f90881bba90 con 0x5576293a5da0 2024-04-22T00:04:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f90919d9740 10 client.0 _ll_register_callbacks cb 0x7f9080037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f9087fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 0x5576293bc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f9087fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 0x5576293bc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50402/0 (socket says 172.21.15.38:50402) 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f9087fff640 1 -- 172.21.15.38:0/3403707702 learned_addr learned my addr 172.21.15.38:0/3403707702 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f908cb91640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.812+0000 7f908d392640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a9db0 0x5576293bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 -- 172.21.15.38:0/3403707702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 msgr2=0x5576293bc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 0x5576293bc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 -- 172.21.15.38:0/3403707702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a9db0 msgr2=0x5576293bc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a9db0 0x5576293bc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 -- 172.21.15.38:0/3403707702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576293bcf70 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f908cb91640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293bf5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5576292ac7c0 tx=0x7f9078013340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e018ae63687e3a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.813+0000 7f9067fff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078004260 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9078004400 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576292d8450 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f90780035f0 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f90919d9740 10 client.26057 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.814+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557629421060 con 0x5576293a0a40 2024-04-22T00:04:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5576292d8450 con 0x5576293a0a40 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f9078024a80 con 0x5576293a0a40 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 10 client.26057 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 20 client.26057 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576294217d0 0x557629423bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 4 client.26057 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.815+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5576293116b0 con 0x5576294217d0 2024-04-22T00:04:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.816+0000 7f9087fff640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576294217d0 0x557629423bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:30.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.817+0000 7f9087fff640 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576294217d0 0x557629423bf0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d1011f1bfd475d4 server_cookie=28b8eaaa658c10c in_seq=0 out_seq=0 2024-04-22T00:04:30.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.817+0000 7f9067fff640 10 client.26057 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:30.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.822+0000 7f9067fff640 1 -- 172.21.15.38:0/3403707702 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5576293116b0 con 0x5576294217d0 2024-04-22T00:04:30.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.822+0000 7f9067fff640 10 client.26057 handle_command_reply: tid=0 2024-04-22T00:04:30.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.822+0000 7f90919d9740 1 client.26057 shutdown 2024-04-22T00:04:30.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.823+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 msgr2=0x5576293bf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.823+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293bf5f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5576292ac7c0 tx=0x7f9078013340 comp rx=0 tx=0).stop 2024-04-22T00:04:30.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.824+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 shutdown_connections 2024-04-22T00:04:30.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.824+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5576294217d0 0x557629423bf0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.824+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576293a9db0 0x5576293bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576293a5da0 0x5576293bc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f90919d9740 1 --2- 172.21.15.38:0/3403707702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5576293a0a40 0x5576293bf5f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 >> 172.21.15.38:0/3403707702 conn(0x5576293674f0 msgr2=0x5576293a6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f9067fff640 0 client.26057 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:30.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 shutdown_connections 2024-04-22T00:04:30.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.825+0000 7f90919d9740 1 -- 172.21.15.38:0/3403707702 wait complete. 2024-04-22T00:04:30.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.826+0000 7f90919d9740 20 client.26057 trim_cache size 0 max 16384 2024-04-22T00:04:30.820 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:04:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:04:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "cb1f0aa3-99f9-46d8-89ad-93f122837c78", 2024-04-22T00:04:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:04:30.820 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:04:30.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.828+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 msgr2=0x7f90700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.828+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 0x7f90700a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9080006010 tx=0x7f9080005d80 comp rx=0 tx=0).stop 2024-04-22T00:04:30.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.828+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 msgr2=0x7f90881a7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:30.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.828+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f907c00bc70 tx=0x7f907c008600 comp rx=0 tx=0).stop 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 shutdown_connections 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f90700a2d80 0x7f90700a5240 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088107710 0x7f90881ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90881020c0 0x7f90881a7340 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 --2- 172.21.15.38:0/4070911713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9088101700 0x7f90881a6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 >> 172.21.15.38:0/4070911713 conn(0x7f90880fd1b0 msgr2=0x7f90880fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 shutdown_connections 2024-04-22T00:04:30.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:30.829+0000 7f908edf5640 1 -- 172.21.15.38:0/4070911713 wait complete. 2024-04-22T00:04:30.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:04:30.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.016+0000 7f3636acd640 1 -- 172.21.15.38:0/4286190176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 msgr2=0x7f363010d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.016+0000 7f3636acd640 1 --2- 172.21.15.38:0/4286190176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363010d500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3628009830 tx=0x7f362802e000 comp rx=0 tx=0).stop 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.016+0000 7f3636acd640 1 -- 172.21.15.38:0/4286190176 shutdown_connections 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 --2- 172.21.15.38:0/4286190176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f36301090e0 0x7f3630114f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 --2- 172.21.15.38:0/4286190176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630108740 0x7f3630108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 --2- 172.21.15.38:0/4286190176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363010d500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 -- 172.21.15.38:0/4286190176 >> 172.21.15.38:0/4286190176 conn(0x7f363006ccb0 msgr2=0x7f363006d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 -- 172.21.15.38:0/4286190176 shutdown_connections 2024-04-22T00:04:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.017+0000 7f3636acd640 1 -- 172.21.15.38:0/4286190176 wait complete. 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.018+0000 7f3636acd640 1 Processor -- start 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.018+0000 7f3636acd640 1 -- start start 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 0x7f363019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36301090e0 0x7f363019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3630118170 con 0x7f3630108740 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3630117ff0 con 0x7f363010d120 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3636acd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36301182f0 con 0x7f36301090e0 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f363506a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3634869640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 0x7f363019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3634869640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 0x7f363019e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47090/0 (socket says 172.21.15.38:47090) 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f3634869640 1 -- 172.21.15.38:0/1939703632 learned_addr learned my addr 172.21.15.38:0/1939703632 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f363506a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42446/0 (socket says 172.21.15.38:42446) 2024-04-22T00:04:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.019+0000 7f362ffff640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36301090e0 0x7f363019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 -- 172.21.15.38:0/1939703632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36301090e0 msgr2=0x7f363019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36301090e0 0x7f363019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 -- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 msgr2=0x7f363019e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 0x7f363019e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 -- 172.21.15.38:0/1939703632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301a2910 con 0x7f363010d120 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.020+0000 7f363506a640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f362000ea10 tx=0x7f362000eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.021+0000 7f362dffb640 1 -- 172.21.15.38:0/1939703632 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362000cbc0 con 0x7f363010d120 2024-04-22T00:04:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.021+0000 7f362dffb640 1 -- 172.21.15.38:0/1939703632 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362000cd60 con 0x7f363010d120 2024-04-22T00:04:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.021+0000 7f362dffb640 1 -- 172.21.15.38:0/1939703632 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362000cbc0 con 0x7f363010d120 2024-04-22T00:04:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.021+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36301a2c00 con 0x7f363010d120 2024-04-22T00:04:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.021+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301a3100 con 0x7f363010d120 2024-04-22T00:04:31.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.025+0000 7f362dffb640 1 -- 172.21.15.38:0/1939703632 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f362001d020 con 0x7f363010d120 2024-04-22T00:04:31.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.025+0000 7f3639799740 1 Processor -- start 2024-04-22T00:04:31.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.025+0000 7f3639799740 1 -- start start 2024-04-22T00:04:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 0x5641ce65d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362dffb640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 0x7f36180a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 0x5641ce665ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce668820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5641ce5c1560 con 0x5641ce666430 2024-04-22T00:04:31.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5641ce569730 con 0x5641ce65d850 2024-04-22T00:04:31.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3639799740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5641ce5bd520 con 0x5641ce65cf30 2024-04-22T00:04:31.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f3634869640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 0x7f36180a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 0x5641ce65d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 0x5641ce665ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 0x5641ce65d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50428/0 (socket says 172.21.15.38:50428) 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362ffff640 1 -- 172.21.15.38:0/3245851074 learned_addr learned my addr 172.21.15.38:0/3245851074 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f362dffb640 1 -- 172.21.15.38:0/1939703632 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f36200cee50 con 0x7f363010d120 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.026+0000 7f363506a640 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce668820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f3634869640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 0x7f36180a53c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3628000f80 tx=0x7f3628030040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 -- 172.21.15.38:0/3245851074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 msgr2=0x5641ce65d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 0x5641ce65d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 -- 172.21.15.38:0/3245851074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 msgr2=0x5641ce668820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce668820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 -- 172.21.15.38:0/3245851074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641ce39f7d0 con 0x5641ce65d850 2024-04-22T00:04:31.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.027+0000 7f362ffff640 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 0x5641ce665ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3624013790 tx=0x7f3624004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=166b3585d7616362 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.030+0000 7f3616ffd640 1 -- 172.21.15.38:0/3245851074 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624013dd0 con 0x5641ce65d850 2024-04-22T00:04:31.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.030+0000 7f3616ffd640 1 -- 172.21.15.38:0/3245851074 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3624005500 con 0x5641ce65d850 2024-04-22T00:04:31.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.030+0000 7f3616ffd640 1 -- 172.21.15.38:0/3245851074 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624004d80 con 0x5641ce65d850 2024-04-22T00:04:31.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 -- 172.21.15.38:0/3245851074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 msgr2=0x5641ce665ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 0x5641ce665ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3624013790 tx=0x7f3624004730 comp rx=0 tx=0).stop 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 -- 172.21.15.38:0/3245851074 shutdown_connections 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce668820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65d850 0x5641ce665ef0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 --2- 172.21.15.38:0/3245851074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65cf30 0x5641ce65d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.031+0000 7f3639799740 1 -- 172.21.15.38:0/3245851074 >> 172.21.15.38:0/3245851074 conn(0x5641ce657660 msgr2=0x5641ce657a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.032+0000 7f3639799740 1 -- 172.21.15.38:0/3245851074 shutdown_connections 2024-04-22T00:04:31.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.033+0000 7f3639799740 1 -- 172.21.15.38:0/3245851074 wait complete. 2024-04-22T00:04:31.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.033+0000 7f3639799740 1 -- start start 2024-04-22T00:04:31.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.033+0000 7f3639799740 1 Processor -- start 2024-04-22T00:04:31.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce65d850 0x5641ce663a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3634869640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3634869640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42464/0 (socket says 172.21.15.38:42464) 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3634869640 1 -- 172.21.15.38:0/499901575 learned_addr learned my addr 172.21.15.38:0/499901575 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce666430 0x5641ce663fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5641ce669220 con 0x5641ce65d850 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5641ce57ce00 con 0x5641ce65cf30 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.034+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5641ce6693a0 con 0x5641ce666430 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f363506a640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce666430 0x5641ce663fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3639799740 10 client.0 _ll_register_callbacks cb 0x5641ce5b7eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3634869640 1 -- 172.21.15.38:0/499901575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce666430 msgr2=0x5641ce663fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3634869640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce666430 0x5641ce663fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3634869640 1 -- 172.21.15.38:0/499901575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce65d850 msgr2=0x5641ce663a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3634869640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce65d850 0x5641ce663a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3634869640 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641ce2d53f0 con 0x5641ce65cf30 2024-04-22T00:04:31.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.035+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3624005500 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.036+0000 7f3634869640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3628039040 tx=0x7f3628009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c6b2c218ecafe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.036+0000 7f360ffff640 10 client.26069 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.036+0000 7f360ffff640 1 -- 172.21.15.38:0/499901575 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628034820 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.036+0000 7f360ffff640 1 -- 172.21.15.38:0/499901575 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3628003970 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.036+0000 7f360ffff640 1 -- 172.21.15.38:0/499901575 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3628003bb0 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.037+0000 7f3639799740 10 client.26069 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.037+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5641ce2950f0 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f360ffff640 1 -- 172.21.15.38:0/499901575 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f3628004040 con 0x5641ce65cf30 2024-04-22T00:04:31.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 10 client.26069 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 20 client.26069 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6763d0 0x5641ce678890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 4 client.26069 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641ce5ceaa0 con 0x5641ce6763d0 2024-04-22T00:04:31.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.038+0000 7f362ffff640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6763d0 0x5641ce678890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.039+0000 7f362ffff640 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6763d0 0x5641ce678890 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cecd0b16b407d50 server_cookie=49f0d0b11a68389b in_seq=0 out_seq=0 2024-04-22T00:04:31.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.039+0000 7f360ffff640 10 client.26069 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f360ffff640 1 -- 172.21.15.38:0/499901575 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641ce5ceaa0 con 0x5641ce6763d0 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f360ffff640 10 client.26069 handle_command_reply: tid=0 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 client.26069 shutdown 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 msgr2=0x5641ce663550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3628039040 tx=0x7f3628009e50 comp rx=0 tx=0).stop 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 shutdown_connections 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce666430 0x5641ce663fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6763d0 0x5641ce678890 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce65d850 0x5641ce663a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 --2- 172.21.15.38:0/499901575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce663550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f360ffff640 0 client.26069 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:31.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 >> 172.21.15.38:0/499901575 conn(0x5641ce657660 msgr2=0x5641ce6593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 shutdown_connections 2024-04-22T00:04:31.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 1 -- 172.21.15.38:0/499901575 wait complete. 2024-04-22T00:04:31.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.041+0000 7f3639799740 20 client.26069 trim_cache size 0 max 16384 2024-04-22T00:04:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 Processor -- start 2024-04-22T00:04:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 -- start start 2024-04-22T00:04:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce6720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce663df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c6170 con 0x5641ce666430 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c7e30 con 0x5641ce65cf30 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.075+0000 7f3639799740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c6840 con 0x5641ce65d850 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47122/0 (socket says 172.21.15.38:47122) 2024-04-22T00:04:31.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 -- 172.21.15.38:0/353837092 learned_addr learned my addr 172.21.15.38:0/353837092 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f362ffff640 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce663df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 -- 172.21.15.38:0/353837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 msgr2=0x5641ce663df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce663df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 -- 172.21.15.38:0/353837092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 msgr2=0x5641ce6720e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce6720e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.076+0000 7f363506a640 1 -- 172.21.15.38:0/353837092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641ce652cf0 con 0x5641ce666430 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.077+0000 7f363506a640 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f362009af70 tx=0x7f36200bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c947f80b02beb322 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.077+0000 7f36167fc640 1 -- 172.21.15.38:0/353837092 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362001a040 con 0x5641ce666430 2024-04-22T00:04:31.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.077+0000 7f36167fc640 1 -- 172.21.15.38:0/353837092 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36200a3ca0 con 0x5641ce666430 2024-04-22T00:04:31.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.077+0000 7f36167fc640 1 -- 172.21.15.38:0/353837092 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36200a3810 con 0x5641ce666430 2024-04-22T00:04:31.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.079+0000 7f3639799740 1 -- 172.21.15.38:0/353837092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 msgr2=0x5641ce656560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.079+0000 7f3639799740 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f362009af70 tx=0x7f36200bb040 comp rx=0 tx=0).stop 2024-04-22T00:04:31.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.080+0000 7f3639799740 1 -- 172.21.15.38:0/353837092 shutdown_connections 2024-04-22T00:04:31.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.080+0000 7f3639799740 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce656560 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.080+0000 7f3639799740 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce663df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.080+0000 7f3639799740 1 --2- 172.21.15.38:0/353837092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce6720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.080+0000 7f3639799740 1 -- 172.21.15.38:0/353837092 >> 172.21.15.38:0/353837092 conn(0x5641ce657660 msgr2=0x5641ce655e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.084+0000 7f3639799740 1 -- 172.21.15.38:0/353837092 shutdown_connections 2024-04-22T00:04:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.084+0000 7f3639799740 1 -- 172.21.15.38:0/353837092 wait complete. 2024-04-22T00:04:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.084+0000 7f3639799740 1 -- start start 2024-04-22T00:04:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 Processor -- start 2024-04-22T00:04:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce6600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce660830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c76a0 con 0x5641ce666430 2024-04-22T00:04:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c6fd0 con 0x5641ce65cf30 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36301c4150 con 0x5641ce65d850 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.085+0000 7f3639799740 10 client.0 _ll_register_callbacks cb 0x7f3620005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f363506a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce660830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:42474/0 (socket says 172.21.15.38:42474) 2024-04-22T00:04:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 -- 172.21.15.38:0/749816239 learned_addr learned my addr 172.21.15.38:0/749816239 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 -- 172.21.15.38:0/749816239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 msgr2=0x5641ce6600f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce6600f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 -- 172.21.15.38:0/749816239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 msgr2=0x5641ce660830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce660830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.086+0000 7f3634869640 1 -- 172.21.15.38:0/749816239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641ce6dba10 con 0x5641ce65cf30 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.087+0000 7f3634869640 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362802e610 tx=0x7f3628042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7653fe673347c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.087+0000 7f360ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:31.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.088+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628066060 con 0x5641ce65cf30 2024-04-22T00:04:31.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.088+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36200a3ca0 con 0x5641ce65cf30 2024-04-22T00:04:31.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.088+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3628034620 con 0x5641ce65cf30 2024-04-22T00:04:31.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.091+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f362806d050 con 0x5641ce65cf30 2024-04-22T00:04:31.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.091+0000 7f3639799740 10 client.26072 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:31.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.091+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641ce6dcbf0 con 0x5641ce65cf30 2024-04-22T00:04:31.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36200a3ca0 con 0x5641ce65cf30 2024-04-22T00:04:31.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f3628056450 con 0x5641ce65cf30 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 10 client.26072 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 10 client.26072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 20 client.26072 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6dd440 0x5641ce6df860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 4 client.26072 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:04:31.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.092+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5641ce5c8df0 con 0x5641ce6dd440 2024-04-22T00:04:31.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.094+0000 7f362ffff640 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6dd440 0x5641ce6df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:31.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.095+0000 7f362ffff640 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6dd440 0x5641ce6df860 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99c8d373b843e4d2 server_cookie=812ce260ced32170 in_seq=0 out_seq=0 2024-04-22T00:04:31.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.095+0000 7f360ffff640 10 client.26072 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:31.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.097+0000 7f360ffff640 1 -- 172.21.15.38:0/749816239 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5641ce5c8df0 con 0x5641ce6dd440 2024-04-22T00:04:31.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.097+0000 7f360ffff640 10 client.26072 handle_command_reply: tid=0 2024-04-22T00:04:31.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.097+0000 7f3639799740 1 client.26072 shutdown 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.098+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 msgr2=0x5641ce66eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.098+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362802e610 tx=0x7f3628042000 comp rx=0 tx=0).stop 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 shutdown_connections 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5641ce6dd440 0x5641ce6df860 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5641ce666430 0x5641ce660830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5641ce65d850 0x5641ce6600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 --2- 172.21.15.38:0/749816239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5641ce65cf30 0x5641ce66eaf0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f362802e610 tx=0x7f3628042000 comp rx=0 tx=0).stop 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f360ffff640 0 client.26072 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 >> 172.21.15.38:0/749816239 conn(0x5641ce657660 msgr2=0x5641ce6566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 shutdown_connections 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.099+0000 7f3639799740 1 -- 172.21.15.38:0/749816239 wait complete. 2024-04-22T00:04:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.100+0000 7f3639799740 20 client.26072 trim_cache size 0 max 16384 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "cb1f0aa3-99f9-46d8-89ad-93f122837c78": { 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "cb1f0aa3-99f9-46d8-89ad-93f122837c78", 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:04:31.093 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:04:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 msgr2=0x7f36180a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 0x7f36180a53c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3628000f80 tx=0x7f3628030040 comp rx=0 tx=0).stop 2024-04-22T00:04:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 msgr2=0x7f363019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f362000ea10 tx=0x7f362000eed0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 shutdown_connections 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f36180a2f00 0x7f36180a53c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f363010d120 0x7f363019ef50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36301090e0 0x7f363019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 --2- 172.21.15.38:0/1939703632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3630108740 0x7f363019e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 >> 172.21.15.38:0/1939703632 conn(0x7f363006ccb0 msgr2=0x7f36300710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 shutdown_connections 2024-04-22T00:04:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:04:31.101+0000 7f3636acd640 1 -- 172.21.15.38:0/1939703632 wait complete. 2024-04-22T00:04:31.112 INFO:tasks.cephfs.filesystem:scrub status for tag:cb1f0aa3-99f9-46d8-89ad-93f122837c78 - {'path': '/', 'tag': 'cb1f0aa3-99f9-46d8-89ad-93f122837c78', 'options': 'recursive,force'} 2024-04-22T00:04:31.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:31 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:04:31.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:31 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:31.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:31 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:31.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:31 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:04:31.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:31 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:31.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:31 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:31.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:31 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:04:31.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:31 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:31.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:31 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:32.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:32 smithi038 ceph-mon[27972]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.42k op/s 2024-04-22T00:04:32.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:32 smithi203 ceph-mon[31431]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.42k op/s 2024-04-22T00:04:32.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:32 smithi111 ceph-mon[31543]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 82 MiB/s wr, 11.42k op/s 2024-04-22T00:04:34.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:34 smithi038 ceph-mon[27972]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 100 MiB/s wr, 14.09k op/s 2024-04-22T00:04:34.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:34 smithi203 ceph-mon[31431]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 100 MiB/s wr, 14.09k op/s 2024-04-22T00:04:34.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:34 smithi111 ceph-mon[31543]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 100 MiB/s wr, 14.09k op/s 2024-04-22T00:04:36.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:36 smithi038 ceph-mon[27972]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 63 MiB/s wr, 8.74k op/s 2024-04-22T00:04:36.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:36 smithi203 ceph-mon[31431]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 63 MiB/s wr, 8.74k op/s 2024-04-22T00:04:36.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:36 smithi111 ceph-mon[31543]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 63 MiB/s wr, 8.74k op/s 2024-04-22T00:04:38.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:38 smithi038 ceph-mon[27972]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 MiB/s wr, 10.77k op/s 2024-04-22T00:04:38.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:38 smithi203 ceph-mon[31431]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 MiB/s wr, 10.77k op/s 2024-04-22T00:04:38.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:38 smithi111 ceph-mon[31543]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 MiB/s wr, 10.77k op/s 2024-04-22T00:04:40.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:40 smithi038 ceph-mon[27972]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 39 MiB/s wr, 5.63k op/s 2024-04-22T00:04:40.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:40 smithi203 ceph-mon[31431]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 39 MiB/s wr, 5.63k op/s 2024-04-22T00:04:40.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:40 smithi111 ceph-mon[31543]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 39 MiB/s wr, 5.63k op/s 2024-04-22T00:04:40.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.995+0000 7f7528af3640 1 -- 172.21.15.38:0/611956698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7524108c80 msgr2=0x7f7524109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:40.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.995+0000 7f7528af3640 1 --2- 172.21.15.38:0/611956698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7524108c80 0x7f7524109100 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f751c013490 tx=0x7f751c034370 comp rx=0 tx=0).stop 2024-04-22T00:04:40.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 -- 172.21.15.38:0/611956698 shutdown_connections 2024-04-22T00:04:40.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 --2- 172.21.15.38:0/611956698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7524108c80 0x7f7524109100 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 --2- 172.21.15.38:0/611956698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410d850 0x7f7524108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 --2- 172.21.15.38:0/611956698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410cf30 0x7f752410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 -- 172.21.15.38:0/611956698 >> 172.21.15.38:0/611956698 conn(0x7f752406cca0 msgr2=0x7f752406d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:40.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 -- 172.21.15.38:0/611956698 shutdown_connections 2024-04-22T00:04:40.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.996+0000 7f7528af3640 1 -- 172.21.15.38:0/611956698 wait complete. 2024-04-22T00:04:40.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 Processor -- start 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 -- start start 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7524108c80 0x7f752419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 0x7f75241a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.997+0000 7f7528af3640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7524112110 con 0x7f752410cf30 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7528af3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7524111f90 con 0x7f752410d850 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7528af3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7524118350 con 0x7f7524108c80 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50028/0 (socket says 172.21.15.38:50028) 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 -- 172.21.15.38:0/897271463 learned_addr learned my addr 172.21.15.38:0/897271463 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7522d9d640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 0x7f75241a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f752259c640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7524108c80 0x7f752419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 -- 172.21.15.38:0/897271463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7524108c80 msgr2=0x7f752419e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7524108c80 0x7f752419e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 -- 172.21.15.38:0/897271463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 msgr2=0x7f75241a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 0x7f75241a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7521d9b640 1 -- 172.21.15.38:0/897271463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75241a34f0 con 0x7f752410cf30 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.998+0000 7f7522d9d640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 0x7f75241a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:40.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.999+0000 7f7521d9b640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f751800b450 tx=0x7f751800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:40.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.999+0000 7f75137fe640 1 -- 172.21.15.38:0/897271463 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518015020 con 0x7f752410cf30 2024-04-22T00:04:40.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.999+0000 7f75137fe640 1 -- 172.21.15.38:0/897271463 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7518011d70 con 0x7f752410cf30 2024-04-22T00:04:40.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.999+0000 7f75137fe640 1 -- 172.21.15.38:0/897271463 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751800f340 con 0x7f752410cf30 2024-04-22T00:04:40.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:40.999+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75241a37e0 con 0x7f752410cf30 2024-04-22T00:04:40.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.000+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75241ab070 con 0x7f752410cf30 2024-04-22T00:04:40.995 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 Processor -- start 2024-04-22T00:04:40.996 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 -- start start 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f75137fe640 1 -- 172.21.15.38:0/897271463 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7518018020 con 0x7f752410cf30 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aa3150bb70 con 0x55aa315a7b20 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55aa314b4470 con 0x55aa315b1210 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.003+0000 7f752b7b0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55aa31507b30 con 0x55aa315a8440 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50034/0 (socket says 172.21.15.38:50034) 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 -- 172.21.15.38:0/403410209 learned_addr learned my addr 172.21.15.38:0/403410209 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f752259c640 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7521d9b640 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 -- 172.21.15.38:0/403410209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 msgr2=0x55aa315b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 -- 172.21.15.38:0/403410209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 msgr2=0x55aa315b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:40.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 -- 172.21.15.38:0/403410209 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa31220210 con 0x55aa315a7b20 2024-04-22T00:04:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f75137fe640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 0x7f750c0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.004+0000 7f7522d9d640 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f751c013490 tx=0x7f751c00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc08a974c93aa8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.005+0000 7f752259c640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 0x7f750c0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.005+0000 7f75137fe640 1 -- 172.21.15.38:0/897271463 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f75180cf070 con 0x7f752410cf30 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.005+0000 7f7510ff9640 1 -- 172.21.15.38:0/403410209 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c04b030 con 0x55aa315a7b20 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.005+0000 7f7510ff9640 1 -- 172.21.15.38:0/403410209 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f751c03b400 con 0x55aa315a7b20 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.005+0000 7f7510ff9640 1 -- 172.21.15.38:0/403410209 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c040040 con 0x55aa315a7b20 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 -- 172.21.15.38:0/403410209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 msgr2=0x55aa315a7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f751c013490 tx=0x7f751c00ff90 comp rx=0 tx=0).stop 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752259c640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 0x7f750c0a5330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55aa315a75d0 tx=0x7f751400a2d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 -- 172.21.15.38:0/403410209 shutdown_connections 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 --2- 172.21.15.38:0/403410209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315a7f00 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 -- 172.21.15.38:0/403410209 >> 172.21.15.38:0/403410209 conn(0x55aa315a2440 msgr2=0x55aa315a2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 -- 172.21.15.38:0/403410209 shutdown_connections 2024-04-22T00:04:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.006+0000 7f752b7b0740 1 -- 172.21.15.38:0/403410209 wait complete. 2024-04-22T00:04:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.007+0000 7f752b7b0740 1 -- start start 2024-04-22T00:04:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.007+0000 7f752b7b0740 1 Processor -- start 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.007+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315ae1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50044/0 (socket says 172.21.15.38:50044) 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aa315b3e30 con 0x55aa315a7b20 2024-04-22T00:04:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55aa314c7a10 con 0x55aa315b1210 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55aa315b3fb0 con 0x55aa315a8440 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752b7b0740 10 client.0 _ll_register_callbacks cb 0x55aa31502b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7522d9d640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f752259c640 1 -- 172.21.15.38:0/890301108 learned_addr learned my addr 172.21.15.38:0/890301108 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 -- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 msgr2=0x55aa315aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 -- 172.21.15.38:0/890301108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 msgr2=0x55aa315ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.008+0000 7f7521d9b640 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa312ec620 con 0x55aa315a8440 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f7521d9b640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ae700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f75180cc530 tx=0x7f75180104a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd244333bae1d83c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f7509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751809ac60 con 0x55aa315a8440 2024-04-22T00:04:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f75180cc6a0 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f751c03b400 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f75180cc8a0 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f752b7b0740 10 client.25963 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.009+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aa315c0050 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f751809ba00 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f75180d10d0 con 0x55aa315a8440 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 10 client.25963 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 10 client.25963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 20 client.25963 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa315c1510 0x55aa315c3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 4 client.25963 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f751c03b400 con 0x55aa315c1510 2024-04-22T00:04:41.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.010+0000 7f752259c640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa315c1510 0x55aa315c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.011+0000 7f7509ffb640 10 client.25963 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.011+0000 7f752259c640 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa315c1510 0x55aa315c3950 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b85d0058e21a208 server_cookie=5e3e1d2806e14cd4 in_seq=0 out_seq=0 2024-04-22T00:04:41.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.012+0000 7f7509ffb640 1 -- 172.21.15.38:0/890301108 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f751c03b400 con 0x55aa315c1510 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.012+0000 7f7509ffb640 10 client.25963 handle_command_reply: tid=0 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.012+0000 7f752b7b0740 1 client.25963 shutdown 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 msgr2=0x55aa315ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ae700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f75180cc530 tx=0x7f75180104a0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 shutdown_connections 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ae700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa315c1510 0x55aa315c3950 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 --2- 172.21.15.38:0/890301108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315ae1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 >> 172.21.15.38:0/890301108 conn(0x55aa315a2440 msgr2=0x55aa315a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.013+0000 7f7509ffb640 0 client.25963 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.014+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 shutdown_connections 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.014+0000 7f752b7b0740 1 -- 172.21.15.38:0/890301108 wait complete. 2024-04-22T00:04:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.014+0000 7f752b7b0740 20 client.25963 trim_cache size 0 max 16384 2024-04-22T00:04:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.043+0000 7f752b7b0740 1 Processor -- start 2024-04-22T00:04:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.043+0000 7f752b7b0740 1 -- start start 2024-04-22T00:04:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b4fc0 con 0x55aa315a7b20 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f7521d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f7521d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315bccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:55386/0 (socket says 172.21.15.38:55386) 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f7521d9b640 1 -- 172.21.15.38:0/1729432476 learned_addr learned my addr 172.21.15.38:0/1729432476 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b6c80 con 0x55aa315b1210 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b5690 con 0x55aa315a8440 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f752259c640 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.044+0000 7f7522d9d640 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 -- 172.21.15.38:0/1729432476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 msgr2=0x55aa315bccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315bccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 -- 172.21.15.38:0/1729432476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 msgr2=0x55aa315aca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 -- 172.21.15.38:0/1729432476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa315ad170 con 0x55aa315a7b20 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.045+0000 7f752259c640 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315b08f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7518004790 tx=0x7f7514019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935ea83a8974f31e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.046+0000 7f750bfff640 1 -- 172.21.15.38:0/1729432476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751401bab0 con 0x55aa315a7b20 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.046+0000 7f750bfff640 1 -- 172.21.15.38:0/1729432476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f751c03b400 con 0x55aa315a7b20 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.046+0000 7f750bfff640 1 -- 172.21.15.38:0/1729432476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751400fa40 con 0x55aa315a7b20 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 msgr2=0x55aa315b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315b08f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7518004790 tx=0x7f7514019000 comp rx=0 tx=0).stop 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 shutdown_connections 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa315b1210 0x55aa315aca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315bccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 --2- 172.21.15.38:0/1729432476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315a7b20 0x55aa315b08f0 secure :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7518004790 tx=0x7f7514019000 comp rx=0 tx=0).stop 2024-04-22T00:04:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.047+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 >> 172.21.15.38:0/1729432476 conn(0x55aa315a2440 msgr2=0x55aa315a14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.048+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 shutdown_connections 2024-04-22T00:04:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.048+0000 7f752b7b0740 1 -- 172.21.15.38:0/1729432476 wait complete. 2024-04-22T00:04:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.049+0000 7f752b7b0740 1 -- start start 2024-04-22T00:04:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.049+0000 7f752b7b0740 1 Processor -- start 2024-04-22T00:04:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.049+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315b1210 0x55aa315abac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f752b7b0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f752b7b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b64f0 con 0x55aa315b1210 2024-04-22T00:04:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f752b7b0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b5e20 con 0x55aa31624ac0 2024-04-22T00:04:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f7522d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f7522d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:59502/0 (socket says 172.21.15.38:59502) 2024-04-22T00:04:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f7522d9d640 1 -- 172.21.15.38:0/4242822332 learned_addr learned my addr 172.21.15.38:0/4242822332 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75241b2fa0 con 0x55aa315a8440 2024-04-22T00:04:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.050+0000 7f7521d9b640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315b1210 0x55aa315abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.044 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f752b7b0740 10 client.0 _ll_register_callbacks cb 0x7f7518010890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f752259c640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 -- 172.21.15.38:0/4242822332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 msgr2=0x55aa315ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 -- 172.21.15.38:0/4242822332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315b1210 msgr2=0x55aa315abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315b1210 0x55aa315abac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 -- 172.21.15.38:0/4242822332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa316274c0 con 0x55aa31624ac0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f751c03b400 con 0x55aa31624ac0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7522d9d640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f751c034980 tx=0x7f751c0133f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f7a1641f2d3326 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.051+0000 7f7509ffb640 10 client.26087 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.052+0000 7f7509ffb640 1 -- 172.21.15.38:0/4242822332 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c054070 con 0x55aa31624ac0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.052+0000 7f7509ffb640 1 -- 172.21.15.38:0/4242822332 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f751c0114e0 con 0x55aa31624ac0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.052+0000 7f7509ffb640 1 -- 172.21.15.38:0/4242822332 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f751c040040 con 0x55aa31624ac0 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.052+0000 7f752b7b0740 10 client.26087 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.052+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55aa31628700 con 0x55aa31624ac0 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f7509ffb640 1 -- 172.21.15.38:0/4242822332 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f751c003970 con 0x55aa31624ac0 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 10 client.26087 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 10 client.26087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 20 client.26087 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa31628de0 0x55aa3162b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 4 client.26087 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:04:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.054+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aa315190b0 con 0x55aa31628de0 2024-04-22T00:04:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.055+0000 7f752259c640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa31628de0 0x55aa3162b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.056+0000 7f752259c640 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa31628de0 0x55aa3162b200 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37f2bb1161e4c43f server_cookie=7e7ac28761095ba6 in_seq=0 out_seq=0 2024-04-22T00:04:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:41.056+0000 7f7509ffb640 10 client.26087 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:42.378 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f7509ffb640 1 -- 172.21.15.38:0/4242822332 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aa315190b0 con 0x55aa31628de0 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f7509ffb640 10 client.26087 handle_command_reply: tid=0 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 client.26087 shutdown 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 msgr2=0x55aa31626eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f751c034980 tx=0x7f751c0133f0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 shutdown_connections 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55aa31624ac0 0x55aa31626eb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55aa31628de0 0x55aa3162b200 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aa315b1210 0x55aa315abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 --2- 172.21.15.38:0/4242822332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55aa315a8440 0x55aa315ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f7509ffb640 0 client.26087 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 >> 172.21.15.38:0/4242822332 conn(0x55aa315a2440 msgr2=0x55aa315a14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 shutdown_connections 2024-04-22T00:04:42.379 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 1 -- 172.21.15.38:0/4242822332 wait complete. 2024-04-22T00:04:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.385+0000 7f752b7b0740 20 client.26087 trim_cache size 0 max 16384 2024-04-22T00:04:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:04:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:04:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:04:42.380 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 msgr2=0x7f750c0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 0x7f750c0a5330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55aa315a75d0 tx=0x7f751400a2d0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 msgr2=0x7f752419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f751800b450 tx=0x7f751800b910 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 shutdown_connections 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f750c0a2e70 0x7f750c0a5330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f752410d850 0x7f75241a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f752410cf30 0x7f752419ea40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 --2- 172.21.15.38:0/897271463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7524108c80 0x7f752419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.386+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 >> 172.21.15.38:0/897271463 conn(0x7f752406cca0 msgr2=0x7f75240720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.387+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 shutdown_connections 2024-04-22T00:04:42.381 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:42.387+0000 7f7528af3640 1 -- 172.21.15.38:0/897271463 wait complete. 2024-04-22T00:04:42.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:42 smithi038 ceph-mon[27972]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 MiB/s wr, 5.20k op/s 2024-04-22T00:04:42.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:42 smithi203 ceph-mon[31431]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 MiB/s wr, 5.20k op/s 2024-04-22T00:04:42.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:42 smithi111 ceph-mon[31543]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 MiB/s wr, 5.20k op/s 2024-04-22T00:04:44.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:44 smithi038 ceph-mon[27972]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 MiB/s wr, 8.15k op/s 2024-04-22T00:04:44.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:44 smithi203 ceph-mon[31431]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 MiB/s wr, 8.15k op/s 2024-04-22T00:04:44.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:44 smithi111 ceph-mon[31543]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 MiB/s wr, 8.15k op/s 2024-04-22T00:04:45.827 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:04:45 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:04:45.422+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...3,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v356 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v2 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v356>,<0x10000000007/.git v3092>,<0x10000000006/ffsb v2640>,<0x10000000005/tmp v153>,<0x100000001fb/client.0 v146>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v151>,<0x10000000001/sv_1 v171>,<0x10000000000/qa v153>,<0x1/volumes v145>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v2 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:04:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 MiB/s wr, 5.48k op/s 2024-04-22T00:04:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:04:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:46.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:46 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 MiB/s wr, 5.48k op/s 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:46.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:46 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 MiB/s wr, 5.48k op/s 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:46.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:46 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:47 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:47 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:47 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:04:47.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:47 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:47 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:47 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:47 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:04:47.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:47 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:47 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:47 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:47 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:04:47.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:47 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:48.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:48 smithi038 ceph-mon[27972]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 51 MiB/s wr, 7.74k op/s 2024-04-22T00:04:48.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:48 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:48.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:48 smithi203 ceph-mon[31431]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 51 MiB/s wr, 7.74k op/s 2024-04-22T00:04:48.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:48 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:48.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:48 smithi111 ceph-mon[31543]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 51 MiB/s wr, 7.74k op/s 2024-04-22T00:04:48.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:48 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:04:50.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:50 smithi038 ceph-mon[27972]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 MiB/s wr, 6.64k op/s 2024-04-22T00:04:50.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:50 smithi203 ceph-mon[31431]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 MiB/s wr, 6.64k op/s 2024-04-22T00:04:50.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:50 smithi111 ceph-mon[31543]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 MiB/s wr, 6.64k op/s 2024-04-22T00:04:52.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:52 smithi038 ceph-mon[27972]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 MiB/s wr, 7.61k op/s 2024-04-22T00:04:52.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:52 smithi203 ceph-mon[31431]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 MiB/s wr, 7.61k op/s 2024-04-22T00:04:52.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:52 smithi111 ceph-mon[31543]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 MiB/s wr, 7.61k op/s 2024-04-22T00:04:54.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:54 smithi038 ceph-mon[27972]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 62 MiB/s wr, 9.52k op/s 2024-04-22T00:04:54.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:54 smithi203 ceph-mon[31431]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 62 MiB/s wr, 9.52k op/s 2024-04-22T00:04:54.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:54 smithi111 ceph-mon[31543]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 62 MiB/s wr, 9.52k op/s 2024-04-22T00:04:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:56 smithi038 ceph-mon[27972]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 42 MiB/s wr, 6.50k op/s 2024-04-22T00:04:56.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:56 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:56 smithi203 ceph-mon[31431]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 42 MiB/s wr, 6.50k op/s 2024-04-22T00:04:56.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:56 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:56.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:56 smithi111 ceph-mon[31543]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 42 MiB/s wr, 6.50k op/s 2024-04-22T00:04:56.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:56 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.017+0000 7f34f71f7640 1 -- 172.21.15.38:0/1186408828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f011af20 msgr2=0x7f34f011d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.017+0000 7f34f71f7640 1 --2- 172.21.15.38:0/1186408828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f011af20 0x7f34f011d310 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009c10 tx=0x7f34e802bf80 comp rx=0 tx=0).stop 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.018+0000 7f34f71f7640 1 -- 172.21.15.38:0/1186408828 shutdown_connections 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.018+0000 7f34f71f7640 1 --2- 172.21.15.38:0/1186408828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f011af20 0x7f34f011d310 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.018+0000 7f34f71f7640 1 --2- 172.21.15.38:0/1186408828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0110310 0x7f34f0108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.018+0000 7f34f71f7640 1 --2- 172.21.15.38:0/1186408828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f010ff30 0x7f34f0108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.018+0000 7f34f71f7640 1 -- 172.21.15.38:0/1186408828 >> 172.21.15.38:0/1186408828 conn(0x7f34f006c810 msgr2=0x7f34f006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.021+0000 7f34f71f7640 1 -- 172.21.15.38:0/1186408828 shutdown_connections 2024-04-22T00:04:57.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.022+0000 7f34f71f7640 1 -- 172.21.15.38:0/1186408828 wait complete. 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.022+0000 7f34f71f7640 1 Processor -- start 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.023+0000 7f34f71f7640 1 -- start start 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.023+0000 7f34f71f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.023+0000 7f34f71f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f011af20 0x7f34f01115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f71f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 0x7f34f0111fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f71f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0120920 con 0x7f34f010ff30 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f71f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01207a0 con 0x7f34f0111b30 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f71f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0120aa0 con 0x7f34f011af20 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41468/0 (socket says 172.21.15.38:41468) 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f4f93640 1 -- 172.21.15.38:0/863056344 learned_addr learned my addr 172.21.15.38:0/863056344 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 0x7f34f0111fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 -- 172.21.15.38:0/863056344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f011af20 msgr2=0x7f34f01115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f011af20 0x7f34f01115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 -- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 msgr2=0x7f34f01184a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f5794640 1 -- 172.21.15.38:0/863056344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f0115f80 con 0x7f34f0111b30 2024-04-22T00:04:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.024+0000 7f34f4f93640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:04:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.025+0000 7f34f5794640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 0x7f34f0111fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34e802c590 tx=0x7f34e80042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.025+0000 7f34edffb640 1 -- 172.21.15.38:0/863056344 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8043020 con 0x7f34f0111b30 2024-04-22T00:04:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.025+0000 7f34edffb640 1 -- 172.21.15.38:0/863056344 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e80046a0 con 0x7f34f0111b30 2024-04-22T00:04:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.025+0000 7f34edffb640 1 -- 172.21.15.38:0/863056344 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e803e870 con 0x7f34f0111b30 2024-04-22T00:04:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.026+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f0116210 con 0x7f34f0111b30 2024-04-22T00:04:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.026+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f0116620 con 0x7f34f0111b30 2024-04-22T00:04:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.030+0000 7f34edffb640 1 -- 172.21.15.38:0/863056344 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f34e8046020 con 0x7f34f0111b30 2024-04-22T00:04:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.031+0000 7f34edffb640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 0x7f34d80a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.031+0000 7f34edffb640 1 -- 172.21.15.38:0/863056344 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f34e80f1e80 con 0x7f34f0111b30 2024-04-22T00:04:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.033+0000 7f34f4f93640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 0x7f34d80a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.037+0000 7f34f4f93640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 0x7f34d80a53b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34e00058c0 tx=0x7f34e000a380 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.158+0000 7f34f9dde740 1 Processor -- start 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.158+0000 7f34f9dde740 1 -- start start 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308322b20 0x55e308322f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e30832e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e308286b70 con 0x55e30832c210 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e30822f470 con 0x55e308323440 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f9dde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e308282b30 con 0x55e308322b20 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f5794640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e30832e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34effff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34effff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55208/0 (socket says 172.21.15.38:55208) 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34effff640 1 -- 172.21.15.38:0/275319783 learned_addr learned my addr 172.21.15.38:0/275319783 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.159+0000 7f34f4f93640 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308322b20 0x55e308322f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.160+0000 7f34effff640 1 -- 172.21.15.38:0/275319783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308322b20 msgr2=0x55e308322f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.160+0000 7f34effff640 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308322b20 0x55e308322f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.160+0000 7f34effff640 1 -- 172.21.15.38:0/275319783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 msgr2=0x55e30832e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.160+0000 7f34effff640 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e30832e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.160+0000 7f34effff640 1 -- 172.21.15.38:0/275319783 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e307f9b210 con 0x55e308323440 2024-04-22T00:04:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.161+0000 7f34effff640 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b9d0 tx=0x7f34e400be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d62558aa032e0b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.162+0000 7f34d6ffd640 1 -- 172.21.15.38:0/275319783 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4016070 con 0x55e308323440 2024-04-22T00:04:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.162+0000 7f34d6ffd640 1 -- 172.21.15.38:0/275319783 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e4004220 con 0x55e308323440 2024-04-22T00:04:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.162+0000 7f34d6ffd640 1 -- 172.21.15.38:0/275319783 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4004a00 con 0x55e308323440 2024-04-22T00:04:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.163+0000 7f34f9dde740 1 -- 172.21.15.38:0/275319783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 msgr2=0x55e30832bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.163+0000 7f34f9dde740 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34e400b9d0 tx=0x7f34e400be90 comp rx=0 tx=0).stop 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 -- 172.21.15.38:0/275319783 shutdown_connections 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e30832e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e30832bcd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 --2- 172.21.15.38:0/275319783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308322b20 0x55e308322f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 -- 172.21.15.38:0/275319783 >> 172.21.15.38:0/275319783 conn(0x55e30831d440 msgr2=0x55e30831d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 -- 172.21.15.38:0/275319783 shutdown_connections 2024-04-22T00:04:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.164+0000 7f34f9dde740 1 -- 172.21.15.38:0/275319783 wait complete. 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.165+0000 7f34f9dde740 1 -- start start 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.165+0000 7f34f9dde740 1 Processor -- start 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e3083297d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e308329d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e30832ee40 con 0x55e30832c210 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e308242a10 con 0x55e308322b20 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e30832efc0 con 0x55e308323440 2024-04-22T00:04:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55212/0 (socket says 172.21.15.38:55212) 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f4f93640 1 -- 172.21.15.38:0/1111349519 learned_addr learned my addr 172.21.15.38:0/1111349519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34f9dde740 10 client.0 _ll_register_callbacks cb 0x55e30827db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.166+0000 7f34effff640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e3083297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 -- 172.21.15.38:0/1111349519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 msgr2=0x55e3083297d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e3083297d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 -- 172.21.15.38:0/1111349519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 msgr2=0x55e308329d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e308329d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 -- 172.21.15.38:0/1111349519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e308067620 con 0x55e308322b20 2024-04-22T00:04:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34f4f93640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f34e001d710 tx=0x7f34e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=888988fcd8dd2615 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34cffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:04:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.167+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e002c060 con 0x55e308322b20 2024-04-22T00:04:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.168+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f34e001b670 con 0x55e308322b20 2024-04-22T00:04:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.168+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e4004220 con 0x55e308322b20 2024-04-22T00:04:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.170+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f34e0016070 con 0x55e308322b20 2024-04-22T00:04:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.170+0000 7f34f9dde740 10 client.26099 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.170+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e30833af20 con 0x55e308322b20 2024-04-22T00:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e0005500 con 0x55e308322b20 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f34e0004830 con 0x55e308322b20 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34f9dde740 10 client.26099 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34f9dde740 10 client.26099 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34f9dde740 20 client.26099 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.171+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e30833c460 0x55e30833e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.172+0000 7f34f9dde740 4 client.26099 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.172+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f34e4004220 con 0x55e30833c460 2024-04-22T00:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.172+0000 7f34effff640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e30833c460 0x55e30833e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.172+0000 7f34cffff640 10 client.26099 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.172+0000 7f34effff640 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e30833c460 0x55e30833e8e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98f226459f974aaa server_cookie=e0fb2f2891c1141e in_seq=0 out_seq=0 2024-04-22T00:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.173+0000 7f34cffff640 1 -- 172.21.15.38:0/1111349519 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f34e4004220 con 0x55e30833c460 2024-04-22T00:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.173+0000 7f34cffff640 10 client.26099 handle_command_reply: tid=0 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 client.26099 shutdown 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 msgr2=0x55e308329290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f34e001d710 tx=0x7f34e0004730 comp rx=0 tx=0).stop 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 shutdown_connections 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e30833c460 0x55e30833e8e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e308329d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e3083297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1111349519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308329290 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.174+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 >> 172.21.15.38:0/1111349519 conn(0x55e30831d440 msgr2=0x55e30831f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.176+0000 7f34cffff640 0 client.26099 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.176+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 shutdown_connections 2024-04-22T00:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.177+0000 7f34f9dde740 1 -- 172.21.15.38:0/1111349519 wait complete. 2024-04-22T00:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.177+0000 7f34f9dde740 20 client.26099 trim_cache size 0 max 16384 2024-04-22T00:04:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.208+0000 7f34f9dde740 1 Processor -- start 2024-04-22T00:04:57.201 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.208+0000 7f34f9dde740 1 -- start start 2024-04-22T00:04:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308337d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e308329b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c5ff0 con 0x55e30832c210 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c7cb0 con 0x55e308322b20 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f9dde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c66c0 con 0x55e308323440 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f5794640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f5794640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41484/0 (socket says 172.21.15.38:41484) 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f5794640 1 -- 172.21.15.38:0/1759298026 learned_addr learned my addr 172.21.15.38:0/1759298026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34f4f93640 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308337d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.209+0000 7f34effff640 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e308329b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 -- 172.21.15.38:0/1759298026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 msgr2=0x55e308329b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e308329b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 -- 172.21.15.38:0/1759298026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 msgr2=0x55e308337d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308337d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 -- 172.21.15.38:0/1759298026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e30833e8d0 con 0x55e30832c210 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.210+0000 7f34f5794640 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f34e80baae0 tx=0x7f34e80bdff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2096b1420844a7ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.213+0000 7f34d67fc640 1 -- 172.21.15.38:0/1759298026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e80be110 con 0x55e30832c210 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.213+0000 7f34d67fc640 1 -- 172.21.15.38:0/1759298026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e80c24c0 con 0x55e30832c210 2024-04-22T00:04:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.213+0000 7f34d67fc640 1 -- 172.21.15.38:0/1759298026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8033b20 con 0x55e30832c210 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 -- 172.21.15.38:0/1759298026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 msgr2=0x55e3083441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f34e80baae0 tx=0x7f34e80bdff0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 -- 172.21.15.38:0/1759298026 shutdown_connections 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e30832c210 0x55e3083441f0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e308323440 0x55e308329b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 --2- 172.21.15.38:0/1759298026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308322b20 0x55e308337d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 -- 172.21.15.38:0/1759298026 >> 172.21.15.38:0/1759298026 conn(0x55e30831d440 msgr2=0x55e30831bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 -- 172.21.15.38:0/1759298026 shutdown_connections 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.214+0000 7f34f9dde740 1 -- 172.21.15.38:0/1759298026 wait complete. 2024-04-22T00:04:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 -- start start 2024-04-22T00:04:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 Processor -- start 2024-04-22T00:04:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e308326540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e30832c210 0x55e308326a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c7520 con 0x55e308322b20 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c6e50 con 0x55e308323440 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.215+0000 7f34f9dde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34f01c3fd0 con 0x55e30832c210 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f9dde740 10 client.0 _ll_register_callbacks cb 0x7f34e0010d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41494/0 (socket says 172.21.15.38:41494) 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 -- 172.21.15.38:0/2612978049 learned_addr learned my addr 172.21.15.38:0/2612978049 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 -- 172.21.15.38:0/2612978049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e30832c210 msgr2=0x55e308326a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e30832c210 0x55e308326a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 -- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 msgr2=0x55e308326540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e308326540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.216+0000 7f34f4f93640 1 -- 172.21.15.38:0/2612978049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3083a1b30 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.217+0000 7f34f4f93640 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f34e001d710 tx=0x7f34e0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33eb7751801fea9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.217+0000 7f34cffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.217+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0005150 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.217+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e80c24c0 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.217+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f34e00052f0 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f34e0048050 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 10 client.16701 fetch_fsmap learned FSMap version 32 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e3083a2e40 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e80c24c0 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f34e0043030 con 0x55e308322b20 2024-04-22T00:04:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 10 client.16701 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 20 client.16701 populate_metadata read hostname 'smithi038' 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e3083a3640 0x55e3083a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 4 client.16701 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.218+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e3082940b0 con 0x55e3083a3640 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.219+0000 7f34effff640 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e3083a3640 0x55e3083a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.219+0000 7f34cffff640 10 client.16701 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:57.220+0000 7f34effff640 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e3083a3640 0x55e3083a5a80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2fea98b2e627baf server_cookie=9c7f20494a475dd1 in_seq=0 out_seq=0 2024-04-22T00:04:57.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:57.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:57.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:57.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:04:57.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:57.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:57.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:57.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:04:57.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:57.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:57.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:57.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:04:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34cffff640 1 -- 172.21.15.38:0/2612978049 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e3082940b0 con 0x55e3083a3640 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34cffff640 10 client.16701 handle_command_reply: tid=0 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 client.16701 shutdown 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 msgr2=0x55e308326000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f34e001d710 tx=0x7f34e0005d40 comp rx=0 tx=0).stop 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 shutdown_connections 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e30832c210 0x55e308326a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e308323440 0x55e308326540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e3083a3640 0x55e3083a5a80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 --2- 172.21.15.38:0/2612978049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e308322b20 0x55e308326000 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34cffff640 0 client.16701 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:04:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 >> 172.21.15.38:0/2612978049 conn(0x55e30831d440 msgr2=0x55e30831c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 shutdown_connections 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 1 -- 172.21.15.38:0/2612978049 wait complete. 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.135+0000 7f34f9dde740 20 client.16701 trim_cache size 0 max 16384 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:04:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 msgr2=0x7f34d80a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 0x7f34d80a53b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34e00058c0 tx=0x7f34e000a380 comp rx=0 tx=0).stop 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 msgr2=0x7f34f0111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 0x7f34f0111fe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f34e802c590 tx=0x7f34e80042e0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 shutdown_connections 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f34d80a2ef0 0x7f34d80a53b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34f0111b30 0x7f34f0111fe0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34f011af20 0x7f34f01115f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 --2- 172.21.15.38:0/863056344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34f010ff30 0x7f34f01184a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:04:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 >> 172.21.15.38:0/863056344 conn(0x7f34f006c810 msgr2=0x7f34f010dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:04:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.136+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 shutdown_connections 2024-04-22T00:04:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:04:58.137+0000 7f34f71f7640 1 -- 172.21.15.38:0/863056344 wait complete. 2024-04-22T00:04:58.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:04:58 smithi038 ceph-mon[27972]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 MiB/s wr, 10.39k op/s 2024-04-22T00:04:58.529 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:04:58 smithi203 ceph-mon[31431]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 MiB/s wr, 10.39k op/s 2024-04-22T00:04:58.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:04:58 smithi111 ceph-mon[31543]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 MiB/s wr, 10.39k op/s 2024-04-22T00:05:00.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:00 smithi038 ceph-mon[27972]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 8.50k op/s 2024-04-22T00:05:00.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:00 smithi203 ceph-mon[31431]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 8.50k op/s 2024-04-22T00:05:00.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:00 smithi111 ceph-mon[31543]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 56 MiB/s wr, 8.50k op/s 2024-04-22T00:05:01.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:05:01.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:05:02.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:02 smithi203 ceph-mon[31431]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-22T00:05:02.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:02 smithi111 ceph-mon[31543]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-22T00:05:02.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:02 smithi038 ceph-mon[27972]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-22T00:05:02.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.944+0000 7fee0bbaf640 1 -- 172.21.15.38:0/3481018257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0411af20 msgr2=0x7fee0411d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:02.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.944+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/3481018257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0411af20 0x7fee0411d310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee0002bc80 comp rx=0 tx=0).stop 2024-04-22T00:05:02.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.945+0000 7fee0bbaf640 1 -- 172.21.15.38:0/3481018257 shutdown_connections 2024-04-22T00:05:02.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.945+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/3481018257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0411af20 0x7fee0411d310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.945+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/3481018257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410dfc0 0x7fee0410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.945+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/3481018257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee0410ff90 0x7fee0410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.945+0000 7fee0bbaf640 1 -- 172.21.15.38:0/3481018257 >> 172.21.15.38:0/3481018257 conn(0x7fee0406c850 msgr2=0x7fee0406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:02.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.951+0000 7fee0bbaf640 1 -- 172.21.15.38:0/3481018257 shutdown_connections 2024-04-22T00:05:02.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.953+0000 7fee0bbaf640 1 -- 172.21.15.38:0/3481018257 wait complete. 2024-04-22T00:05:02.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.954+0000 7fee0bbaf640 1 Processor -- start 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.959+0000 7fee0bbaf640 1 -- start start 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410ff90 0x7fee041a7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee0411af20 0x7fee041ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fee041209c0 con 0x7fee0411af20 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fee04120840 con 0x7fee0410ff90 2024-04-22T00:05:02.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.960+0000 7fee0bbaf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee04120b40 con 0x7fee0410dfc0 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.962+0000 7fee0914a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410ff90 0x7fee041a7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.962+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.962+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51264/0 (socket says 172.21.15.38:51264) 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.962+0000 7fee0994b640 1 -- 172.21.15.38:0/2391536673 learned_addr learned my addr 172.21.15.38:0/2391536673 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.963+0000 7fee0994b640 1 -- 172.21.15.38:0/2391536673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410ff90 msgr2=0x7fee041a7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.963+0000 7fee0994b640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410ff90 0x7fee041a7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.963+0000 7fee0994b640 1 -- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee0411af20 msgr2=0x7fee041ab780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:05:02.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.963+0000 7fee0994b640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee0411af20 0x7fee041ab780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.963+0000 7fee0994b640 1 -- 172.21.15.38:0/2391536673 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee041abeb0 con 0x7fee0410dfc0 2024-04-22T00:05:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.964+0000 7fee0994b640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00ba50 tx=0x7fedfc00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.964+0000 7fedfaffd640 1 -- 172.21.15.38:0/2391536673 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc019070 con 0x7fee0410dfc0 2024-04-22T00:05:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.964+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee041ac1a0 con 0x7fee0410dfc0 2024-04-22T00:05:02.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.964+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee041b3a30 con 0x7fee0410dfc0 2024-04-22T00:05:02.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.965+0000 7fedfaffd640 1 -- 172.21.15.38:0/2391536673 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc00cbe0 con 0x7fee0410dfc0 2024-04-22T00:05:02.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.965+0000 7fedfaffd640 1 -- 172.21.15.38:0/2391536673 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc007730 con 0x7fee0410dfc0 2024-04-22T00:05:02.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.968+0000 7fee0e793740 1 Processor -- start 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 -- start start 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ec9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563ae6e2d560 con 0x563ae6ec9850 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ae6dd5730 con 0x563ae6ec8f30 2024-04-22T00:05:02.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0e793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ae6e29520 con 0x563ae6ed2430 2024-04-22T00:05:02.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0914a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0914a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51274/0 (socket says 172.21.15.38:51274) 2024-04-22T00:05:02.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.969+0000 7fee0914a640 1 -- 172.21.15.38:0/784714692 learned_addr learned my addr 172.21.15.38:0/784714692 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0a14c640 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ec9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fedfaffd640 1 -- 172.21.15.38:0/2391536673 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fedfc01c070 con 0x7fee0410dfc0 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0914a640 1 -- 172.21.15.38:0/784714692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 msgr2=0x563ae6ec9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0914a640 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ec9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0914a640 1 -- 172.21.15.38:0/784714692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 msgr2=0x563ae6ed1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0914a640 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.970+0000 7fee0914a640 1 -- 172.21.15.38:0/784714692 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae6c0b7d0 con 0x563ae6ed2430 2024-04-22T00:05:02.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.972+0000 7fedfaffd640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 0x7fedf00a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.972+0000 7fee0914a640 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009a10 tx=0x7fedf4009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa2856eaf4582e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:02.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.972+0000 7fedfaffd640 1 -- 172.21.15.38:0/2391536673 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fedfc0ceb60 con 0x7fee0410dfc0 2024-04-22T00:05:02.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.973+0000 7fee0a14c640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 0x7fedf00a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.973+0000 7fedeffff640 1 -- 172.21.15.38:0/784714692 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4017070 con 0x563ae6ed2430 2024-04-22T00:05:02.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.973+0000 7fedeffff640 1 -- 172.21.15.38:0/784714692 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf4013470 con 0x563ae6ed2430 2024-04-22T00:05:02.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.973+0000 7fedeffff640 1 -- 172.21.15.38:0/784714692 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4012630 con 0x563ae6ed2430 2024-04-22T00:05:02.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.974+0000 7fee0e793740 1 -- 172.21.15.38:0/784714692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 msgr2=0x563ae6ed4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:02.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.974+0000 7fee0e793740 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009a10 tx=0x7fedf4009ed0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.975+0000 7fee0a14c640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 0x7fedf00a5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee0002c290 tx=0x7fee0002c1a0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:02.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.977+0000 7fee0e793740 1 -- 172.21.15.38:0/784714692 shutdown_connections 2024-04-22T00:05:02.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.977+0000 7fee0e793740 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ed4820 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.977+0000 7fee0e793740 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.977+0000 7fee0e793740 1 --2- 172.21.15.38:0/784714692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ec9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.977+0000 7fee0e793740 1 -- 172.21.15.38:0/784714692 >> 172.21.15.38:0/784714692 conn(0x563ae6ec3660 msgr2=0x563ae6ec3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:02.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.980+0000 7fee0e793740 1 -- 172.21.15.38:0/784714692 shutdown_connections 2024-04-22T00:05:02.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.980+0000 7fee0e793740 1 -- 172.21.15.38:0/784714692 wait complete. 2024-04-22T00:05:02.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.981+0000 7fee0e793740 1 -- start start 2024-04-22T00:05:02.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.981+0000 7fee0e793740 1 Processor -- start 2024-04-22T00:05:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.982+0000 7fee0e793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ecf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.982+0000 7fee0e793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.982+0000 7fee0e793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 0x563ae6ecff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0914a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0914a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55266/0 (socket says 172.21.15.38:55266) 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0a14c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 0x563ae6ecff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0a14c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 0x563ae6ecff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41514/0 (socket says 172.21.15.38:41514) 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ecf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ecf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51284/0 (socket says 172.21.15.38:51284) 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0914a640 1 -- 172.21.15.38:0/3034437995 learned_addr learned my addr 172.21.15.38:0/3034437995 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563ae6ed5270 con 0x563ae6ed2430 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563ae6de8e00 con 0x563ae6ec9850 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.983+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ae6ed53f0 con 0x563ae6ec8f30 2024-04-22T00:05:02.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0e793740 10 client.0 _ll_register_callbacks cb 0x563ae6e23eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:02.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 -- 172.21.15.38:0/3034437995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 msgr2=0x563ae6ecf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:02.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ecf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 -- 172.21.15.38:0/3034437995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 msgr2=0x563ae6ecff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 0x563ae6ecff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae6b413f0 con 0x563ae6ec9850 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0914a640 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedf40099e0 tx=0x7fedf4013bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc848f0af3875c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.984+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf4013470 con 0x563ae6ec9850 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.985+0000 7feded7fa640 10 client.26111 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.985+0000 7feded7fa640 1 -- 172.21.15.38:0/3034437995 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4029060 con 0x563ae6ec9850 2024-04-22T00:05:02.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.986+0000 7feded7fa640 1 -- 172.21.15.38:0/3034437995 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fedf401c660 con 0x563ae6ec9850 2024-04-22T00:05:02.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.986+0000 7feded7fa640 1 -- 172.21.15.38:0/3034437995 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fedf4017040 con 0x563ae6ec9850 2024-04-22T00:05:02.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.986+0000 7fee0e793740 10 client.26111 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.986+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=32}) -- 0x563ae6b010f0 con 0x563ae6ec9850 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7feded7fa640 1 -- 172.21.15.38:0/3034437995 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fedf401c8a0 con 0x563ae6ec9850 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 10 client.26111 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 10 client.26111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 20 client.26111 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6ee2770 0x563ae6ee4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 4 client.26111 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.987+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fedf4013470 con 0x563ae6ee2770 2024-04-22T00:05:02.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.989+0000 7fee0994b640 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6ee2770 0x563ae6ee4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:02.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.991+0000 7fee0994b640 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6ee2770 0x563ae6ee4b90 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e756e6aa57475e4 server_cookie=3ec106926a29a184 in_seq=0 out_seq=0 2024-04-22T00:05:02.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.992+0000 7feded7fa640 10 client.26111 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:02.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.992+0000 7feded7fa640 1 -- 172.21.15.38:0/3034437995 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fedf4013470 con 0x563ae6ee2770 2024-04-22T00:05:02.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.992+0000 7feded7fa640 10 client.26111 handle_command_reply: tid=0 2024-04-22T00:05:02.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.992+0000 7fee0e793740 1 client.26111 shutdown 2024-04-22T00:05:02.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 msgr2=0x563ae6ecf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:02.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedf40099e0 tx=0x7fedf4013bf0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 shutdown_connections 2024-04-22T00:05:02.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6ee2770 0x563ae6ee4b90 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ed2430 0x563ae6ecff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec9850 0x563ae6ecf9d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 --2- 172.21.15.38:0/3034437995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ecf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 >> 172.21.15.38:0/3034437995 conn(0x563ae6ec3660 msgr2=0x563ae6ec53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.993+0000 7feded7fa640 0 client.26111 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.994+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 shutdown_connections 2024-04-22T00:05:02.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.994+0000 7fee0e793740 1 -- 172.21.15.38:0/3034437995 wait complete. 2024-04-22T00:05:02.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:02.994+0000 7fee0e793740 20 client.26111 trim_cache size 0 max 16384 2024-04-22T00:05:03.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.035+0000 7fee0e793740 1 Processor -- start 2024-04-22T00:05:03.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.035+0000 7fee0e793740 1 -- start start 2024-04-22T00:05:03.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.035+0000 7fee0e793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ed1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.035+0000 7fee0e793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0e793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ecfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0e793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fede800fd40 con 0x563ae6ec9850 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0e793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fede8011a00 con 0x563ae6ec8f30 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0e793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fede8010410 con 0x563ae6ed2430 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41530/0 (socket says 172.21.15.38:41530) 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 -- 172.21.15.38:0/2513406187 learned_addr learned my addr 172.21.15.38:0/2513406187 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0a14c640 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ecfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 -- 172.21.15.38:0/2513406187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 msgr2=0x563ae6ecfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:03.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ecfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.036+0000 7fee0914a640 1 -- 172.21.15.38:0/2513406187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 msgr2=0x563ae6ed1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:03.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.037+0000 7fee0914a640 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ed1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.037+0000 7fee0914a640 1 -- 172.21.15.38:0/2513406187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae6ecd9e0 con 0x563ae6ec9850 2024-04-22T00:05:03.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.037+0000 7fee0914a640 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fedf4013ea0 tx=0x7fedf401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b849563ecd83ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:03.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.038+0000 7fedef7fe640 1 -- 172.21.15.38:0/2513406187 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf401c860 con 0x563ae6ec9850 2024-04-22T00:05:03.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.038+0000 7fedef7fe640 1 -- 172.21.15.38:0/2513406187 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf401ca00 con 0x563ae6ec9850 2024-04-22T00:05:03.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.039+0000 7fee0e793740 1 -- 172.21.15.38:0/2513406187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 msgr2=0x563ae6ede210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.039+0000 7fee0e793740 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fedf4013ea0 tx=0x7fedf401f000 comp rx=0 tx=0).stop 2024-04-22T00:05:03.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.041+0000 7fee0e793740 1 -- 172.21.15.38:0/2513406187 shutdown_connections 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.041+0000 7fee0e793740 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ed2430 0x563ae6ecfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.041+0000 7fee0e793740 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ede210 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.041+0000 7fee0e793740 1 --2- 172.21.15.38:0/2513406187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ec8f30 0x563ae6ed1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.041+0000 7fee0e793740 1 -- 172.21.15.38:0/2513406187 >> 172.21.15.38:0/2513406187 conn(0x563ae6ec3660 msgr2=0x563ae6ecdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.042+0000 7fee0e793740 1 -- 172.21.15.38:0/2513406187 shutdown_connections 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.042+0000 7fee0e793740 1 -- 172.21.15.38:0/2513406187 wait complete. 2024-04-22T00:05:03.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.042+0000 7fee0e793740 1 -- start start 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.042+0000 7fee0e793740 1 Processor -- start 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ed2430 0x563ae6ed6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fede8011270 con 0x563ae6ec9850 2024-04-22T00:05:03.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fede8010ba0 con 0x563ae6ed2430 2024-04-22T00:05:03.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fede800dd20 con 0x563ae6ec8f30 2024-04-22T00:05:03.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.043+0000 7fee0e793740 10 client.0 _ll_register_callbacks cb 0x7fee00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51312/0 (socket says 172.21.15.38:51312) 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 -- 172.21.15.38:0/3153169511 learned_addr learned my addr 172.21.15.38:0/3153169511 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0a14c640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ed2430 0x563ae6ed6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0914a640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 -- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ed2430 msgr2=0x563ae6ed6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ed2430 0x563ae6ed6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 -- 172.21.15.38:0/3153169511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 msgr2=0x563ae6ed5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 -- 172.21.15.38:0/3153169511 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae6f46d00 con 0x563ae6ec8f30 2024-04-22T00:05:03.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.044+0000 7fee0994b640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedfc09ae30 tx=0x7fedfc00e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b808fa554ab4e72e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:03.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.045+0000 7feded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:05:03.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.045+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc014440 con 0x563ae6ec8f30 2024-04-22T00:05:03.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.045+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ae6e3aaa0 con 0x563ae6ec8f30 2024-04-22T00:05:03.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.046+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fedfc097510 con 0x563ae6ec8f30 2024-04-22T00:05:03.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.046+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fedfc0970e0 con 0x563ae6ec8f30 2024-04-22T00:05:03.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.046+0000 7fee0e793740 10 client.25996 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:03.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.046+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ae6f48050 con 0x563ae6ec8f30 2024-04-22T00:05:03.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.047+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563ae6e3aaa0 con 0x563ae6ec8f30 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fedfc0a35d0 con 0x563ae6ec8f30 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 10 client.25996 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 10 client.25996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 20 client.25996 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6f488b0 0x563ae6f4acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 4 client.25996 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:05:03.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ae6e34df0 con 0x563ae6f488b0 2024-04-22T00:05:03.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.050+0000 7fee0914a640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6f488b0 0x563ae6f4acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.051+0000 7fee0914a640 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6f488b0 0x563ae6f4acd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d4fd8bb7536bbc server_cookie=54e4fea8b8837588 in_seq=0 out_seq=0 2024-04-22T00:05:03.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.052+0000 7feded7fa640 10 client.25996 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:03.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.053+0000 7feded7fa640 1 -- 172.21.15.38:0/3153169511 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ae6e34df0 con 0x563ae6f488b0 2024-04-22T00:05:03.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.053+0000 7feded7fa640 10 client.25996 handle_command_reply: tid=0 2024-04-22T00:05:03.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.053+0000 7fee0e793740 1 client.25996 shutdown 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 msgr2=0x563ae6ed56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedfc09ae30 tx=0x7fedfc00e040 comp rx=0 tx=0).stop 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 shutdown_connections 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ae6ed2430 0x563ae6ed6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563ae6f488b0 0x563ae6f4acd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563ae6ec9850 0x563ae6ed5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 --2- 172.21.15.38:0/3153169511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae6ec8f30 0x563ae6ed56e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 >> 172.21.15.38:0/3153169511 conn(0x563ae6ec3660 msgr2=0x563ae6ec4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7feded7fa640 0 client.25996 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 shutdown_connections 2024-04-22T00:05:03.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 1 -- 172.21.15.38:0/3153169511 wait complete. 2024-04-22T00:05:03.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.054+0000 7fee0e793740 20 client.25996 trim_cache size 0 max 16384 2024-04-22T00:05:03.049 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:03.049 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:05:03.049 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:05:03.049 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 msgr2=0x7fedf00a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 0x7fedf00a5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee0002c290 tx=0x7fee0002c1a0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 msgr2=0x7fee041a6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00ba50 tx=0x7fedfc00bf10 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 shutdown_connections 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fedf00a2e40 0x7fedf00a5300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee0411af20 0x7fee041ab780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee0410ff90 0x7fee041a7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 --2- 172.21.15.38:0/2391536673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee0410dfc0 0x7fee041a6ec0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.056+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 >> 172.21.15.38:0/2391536673 conn(0x7fee0406c850 msgr2=0x7fee04072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.058+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 shutdown_connections 2024-04-22T00:05:03.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.058+0000 7fee0bbaf640 1 -- 172.21.15.38:0/2391536673 wait complete. 2024-04-22T00:05:03.071 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:05:03.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:05:03.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 -- 172.21.15.38:0/3384077163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 msgr2=0x7f26100a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 --2- 172.21.15.38:0/3384077163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 0x7f26100a2b80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f260c009900 tx=0x7f260c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 -- 172.21.15.38:0/3384077163 shutdown_connections 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 --2- 172.21.15.38:0/3384077163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 0x7f26100a2b80 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 --2- 172.21.15.38:0/3384077163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f2610004bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 --2- 172.21.15.38:0/3384077163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f261009c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.265+0000 7f261cc09640 1 -- 172.21.15.38:0/3384077163 >> 172.21.15.38:0/3384077163 conn(0x7f261000a730 msgr2=0x7f261000ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.266+0000 7f261cc09640 1 -- 172.21.15.38:0/3384077163 shutdown_connections 2024-04-22T00:05:03.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.266+0000 7f261cc09640 1 -- 172.21.15.38:0/3384077163 wait complete. 2024-04-22T00:05:03.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.267+0000 7f261cc09640 1 Processor -- start 2024-04-22T00:05:03.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.267+0000 7f261cc09640 1 -- start start 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f261013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 0x7f261013fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f2610143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26100a8370 con 0x7f26100050f0 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f26100a81f0 con 0x7f2610004730 2024-04-22T00:05:03.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.268+0000 7f261cc09640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f26100a84f0 con 0x7f261009be00 2024-04-22T00:05:03.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.269+0000 7f2617fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f2610143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.269+0000 7f26177fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f261013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.269+0000 7f2617fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f2610143fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51330/0 (socket says 172.21.15.38:51330) 2024-04-22T00:05:03.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.269+0000 7f2617fff640 1 -- 172.21.15.38:0/3884812127 learned_addr learned my addr 172.21.15.38:0/3884812127 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.270+0000 7f26177fe640 1 -- 172.21.15.38:0/3884812127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 msgr2=0x7f2610143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.270+0000 7f26177fe640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f2610143fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.270+0000 7f26177fe640 1 -- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 msgr2=0x7f261013fc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.270+0000 7f26177fe640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 0x7f261013fc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.270+0000 7f26177fe640 1 -- 172.21.15.38:0/3884812127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2610144710 con 0x7f2610004730 2024-04-22T00:05:03.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.271+0000 7f26177fe640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f261013f720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2618000b60 tx=0x7f261806d430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:03.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.271+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618079030 con 0x7f2610004730 2024-04-22T00:05:03.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.272+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f261806dd20 con 0x7f2610004730 2024-04-22T00:05:03.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.272+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2610144a00 con 0x7f2610004730 2024-04-22T00:05:03.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.272+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26100a3240 con 0x7f2610004730 2024-04-22T00:05:03.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.274+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618079030 con 0x7f2610004730 2024-04-22T00:05:03.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.277+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2604005290 con 0x7f2610004730 2024-04-22T00:05:03.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.281+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f261807c070 con 0x7f2610004730 2024-04-22T00:05:03.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.281+0000 7f2614ff9640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 0x7f26000a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.282+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2618132000 con 0x7f2610004730 2024-04-22T00:05:03.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.282+0000 7f2616ffd640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 0x7f26000a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.284+0000 7f2616ffd640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 0x7f26000a5440 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f261009b310 tx=0x7f2608005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:03.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.289+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f26180fe990 con 0x7f2610004730 2024-04-22T00:05:03.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.761+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2604005580 con 0x7f2610004730 2024-04-22T00:05:03.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.763+0000 7f2614ff9640 1 -- 172.21.15.38:0/3884812127 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11949 (secure 0 0 0) 0x7f26180fe290 con 0x7f2610004730 2024-04-22T00:05:03.757 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:05:03.757 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:03:41.555018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24448,24455,24460,24487,24500]},"id":1}]} 2024-04-22T00:05:03.757 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 32 2024-04-22T00:05:03.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 msgr2=0x7f26000a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 0x7f26000a5440 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f261009b310 tx=0x7f2608005f90 comp rx=0 tx=0).stop 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 msgr2=0x7f261013f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f261013f720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2618000b60 tx=0x7f261806d430 comp rx=0 tx=0).stop 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 shutdown_connections 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f26000a2f80 0x7f26000a5440 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f261009b310 tx=0x7f2608005f90 comp rx=0 tx=0).stop 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f261009be00 0x7f2610143fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26100050f0 0x7f261013fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.768+0000 7f261cc09640 1 --2- 172.21.15.38:0/3884812127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2610004730 0x7f261013f720 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2618000b60 tx=0x7f261806d430 comp rx=0 tx=0).stop 2024-04-22T00:05:03.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.769+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 >> 172.21.15.38:0/3884812127 conn(0x7f261000a730 msgr2=0x7f2610093850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.770+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 shutdown_connections 2024-04-22T00:05:03.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.770+0000 7f261cc09640 1 -- 172.21.15.38:0/3884812127 wait complete. 2024-04-22T00:05:03.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:05:03.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.980+0000 7fcb25147640 1 -- 172.21.15.38:0/3102449960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb20105ad0 msgr2=0x7fcb20105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.980+0000 7fcb25147640 1 --2- 172.21.15.38:0/3102449960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb20105ad0 0x7fcb20105f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009ca0 tx=0x7fcb1802c230 comp rx=0 tx=0).stop 2024-04-22T00:05:03.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.985+0000 7fcb25147640 1 -- 172.21.15.38:0/3102449960 shutdown_connections 2024-04-22T00:05:03.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.985+0000 7fcb25147640 1 --2- 172.21.15.38:0/3102449960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb20105ad0 0x7fcb20105f60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.985+0000 7fcb25147640 1 --2- 172.21.15.38:0/3102449960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb201010d0 0x7fcb20105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.985+0000 7fcb25147640 1 --2- 172.21.15.38:0/3102449960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb201007b0 0x7fcb20100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.985+0000 7fcb25147640 1 -- 172.21.15.38:0/3102449960 >> 172.21.15.38:0/3102449960 conn(0x7fcb20077740 msgr2=0x7fcb20077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:03.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.986+0000 7fcb25147640 1 -- 172.21.15.38:0/3102449960 shutdown_connections 2024-04-22T00:05:03.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.986+0000 7fcb25147640 1 -- 172.21.15.38:0/3102449960 wait complete. 2024-04-22T00:05:03.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.987+0000 7fcb25147640 1 Processor -- start 2024-04-22T00:05:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 -- start start 2024-04-22T00:05:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb201007b0 0x7fcb20072f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb201010d0 0x7fcb2006bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:03.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb201180d0 con 0x7fcb201007b0 2024-04-22T00:05:03.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb20117f50 con 0x7fcb20105ad0 2024-04-22T00:05:03.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.990+0000 7fcb25147640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb20118250 con 0x7fcb201010d0 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43520/0 (socket says 172.21.15.38:43520) 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 -- 172.21.15.38:0/709809270 learned_addr learned my addr 172.21.15.38:0/709809270 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 -- 172.21.15.38:0/709809270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb201010d0 msgr2=0x7fcb2006bf90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb201010d0 0x7fcb2006bf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 -- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb201007b0 msgr2=0x7fcb20072f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:03.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb201007b0 0x7fcb20072f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:03.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.992+0000 7fcb1f59e640 1 -- 172.21.15.38:0/709809270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2006ca90 con 0x7fcb20105ad0 2024-04-22T00:05:03.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.993+0000 7fcb1f59e640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb18037040 tx=0x7fcb18035cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:03.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.994+0000 7fcb07fff640 1 -- 172.21.15.38:0/709809270 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18039040 con 0x7fcb20105ad0 2024-04-22T00:05:03.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.994+0000 7fcb07fff640 1 -- 172.21.15.38:0/709809270 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb18044070 con 0x7fcb20105ad0 2024-04-22T00:05:03.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.994+0000 7fcb07fff640 1 -- 172.21.15.38:0/709809270 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18004280 con 0x7fcb20105ad0 2024-04-22T00:05:03.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.994+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb2006cd20 con 0x7fcb20105ad0 2024-04-22T00:05:03.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:03.994+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb20070c70 con 0x7fcb20105ad0 2024-04-22T00:05:04.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.008+0000 7fcb27e05740 1 Processor -- start 2024-04-22T00:05:04.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.009+0000 7fcb07fff640 1 -- 172.21.15.38:0/709809270 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcb18004420 con 0x7fcb20105ad0 2024-04-22T00:05:04.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.010+0000 7fcb07fff640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 0x7fcb080a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.010+0000 7fcb07fff640 1 -- 172.21.15.38:0/709809270 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcb180f14f0 con 0x7fcb20105ad0 2024-04-22T00:05:04.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 -- start start 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f444400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f44d520 0x55e27f44f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e27f3a8530 con 0x55e27f444020 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e27f350830 con 0x55e27f44d520 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.011+0000 7fcb27e05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e27f3a44f0 con 0x55e27f444940 2024-04-22T00:05:04.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.012+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 0x7fcb080a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.012+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.012+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:38874/0 (socket says 172.21.15.38:38874) 2024-04-22T00:05:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.012+0000 7fcb1e59c640 1 -- 172.21.15.38:0/1422981791 learned_addr learned my addr 172.21.15.38:0/1422981791 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f444400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 -- 172.21.15.38:0/1422981791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f44d520 msgr2=0x55e27f44f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f44d520 0x55e27f44f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 -- 172.21.15.38:0/1422981791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 msgr2=0x55e27f444400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f444400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 -- 172.21.15.38:0/1422981791 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27f3742c0 con 0x55e27f444940 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.013+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb100148a0 tx=0x7fcb10014d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7224a429be9c7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.014+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 0x7fcb080a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcb10012760 tx=0x7fcb1001a000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.015+0000 7fcb057fa640 1 -- 172.21.15.38:0/1422981791 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000f870 con 0x55e27f444940 2024-04-22T00:05:04.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.015+0000 7fcb057fa640 1 -- 172.21.15.38:0/1422981791 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb1001f440 con 0x55e27f444940 2024-04-22T00:05:04.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.015+0000 7fcb057fa640 1 -- 172.21.15.38:0/1422981791 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1001d670 con 0x55e27f444940 2024-04-22T00:05:04.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.016+0000 7fcb27e05740 1 -- 172.21.15.38:0/1422981791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 msgr2=0x55e27f44cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.016+0000 7fcb27e05740 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb100148a0 tx=0x7fcb10014d60 comp rx=0 tx=0).stop 2024-04-22T00:05:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.017+0000 7fcb27e05740 1 -- 172.21.15.38:0/1422981791 shutdown_connections 2024-04-22T00:05:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.017+0000 7fcb27e05740 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f44d520 0x55e27f44f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.017+0000 7fcb27e05740 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f444940 0x55e27f44cfe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.017+0000 7fcb27e05740 1 --2- 172.21.15.38:0/1422981791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f444400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.017+0000 7fcb27e05740 1 -- 172.21.15.38:0/1422981791 >> 172.21.15.38:0/1422981791 conn(0x55e27f43e750 msgr2=0x55e27f43eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.018+0000 7fcb27e05740 1 -- 172.21.15.38:0/1422981791 shutdown_connections 2024-04-22T00:05:04.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.019+0000 7fcb27e05740 1 -- 172.21.15.38:0/1422981791 wait complete. 2024-04-22T00:05:04.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.020+0000 7fcb27e05740 1 -- start start 2024-04-22T00:05:04.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.020+0000 7fcb27e05740 1 Processor -- start 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f44cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f44a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f44a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e27f450250 con 0x55e27f444020 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e27f363f10 con 0x55e27f444940 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e27f4503d0 con 0x55e27f44d520 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb27e05740 10 client.0 _ll_register_callbacks cb 0x55e27f0bbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f44a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f44a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43538/0 (socket says 172.21.15.38:43538) 2024-04-22T00:05:04.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb1e59c640 1 -- 172.21.15.38:0/284731645 learned_addr learned my addr 172.21.15.38:0/284731645 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.023+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f44cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 -- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 msgr2=0x55e27f44a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f44a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 -- 172.21.15.38:0/284731645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 msgr2=0x55e27f44a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f44a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 -- 172.21.15.38:0/284731645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27f335c80 con 0x55e27f444020 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f44cf00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400e650 tx=0x7fcb1400eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b84c817e70b79 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.024+0000 7fcb02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14004220 con 0x55e27f444020 2024-04-22T00:05:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcb140043c0 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb1001f440 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb1400a890 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb27e05740 10 client.16722 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.026+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e27f0bc3f0 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.028+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb1001f440 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.028+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcb14010430 con 0x55e27f444020 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 10 client.16722 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 10 client.16722 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:05:04.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 20 client.16722 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:04.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f46e660 0x55e27f470a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 4 client.16722 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:04.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.029+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e27f3b5a70 con 0x55e27f46e660 2024-04-22T00:05:04.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.033+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f46e660 0x55e27f470a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.034+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f46e660 0x55e27f470a90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=113eeb252e843b09 server_cookie=51bc864519357962 in_seq=0 out_seq=0 2024-04-22T00:05:04.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.034+0000 7fcb02ff5640 10 client.16722 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:04.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.036+0000 7fcb02ff5640 1 -- 172.21.15.38:0/284731645 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e27f3b5a70 con 0x55e27f46e660 2024-04-22T00:05:04.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.036+0000 7fcb02ff5640 10 client.16722 handle_command_reply: tid=0 2024-04-22T00:05:04.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.036+0000 7fcb27e05740 1 client.16722 shutdown 2024-04-22T00:05:04.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.038+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 msgr2=0x55e27f44cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.038+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f44cf00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400e650 tx=0x7fcb1400eb10 comp rx=0 tx=0).stop 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 shutdown_connections 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f46e660 0x55e27f470a90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f44a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f44a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 --2- 172.21.15.38:0/284731645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f44cf00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.039+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 >> 172.21.15.38:0/284731645 conn(0x55e27f43e750 msgr2=0x55e27f4404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.041+0000 7fcb02ff5640 0 client.16722 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:04.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.041+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 shutdown_connections 2024-04-22T00:05:04.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.042+0000 7fcb27e05740 1 -- 172.21.15.38:0/284731645 wait complete. 2024-04-22T00:05:04.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.042+0000 7fcb27e05740 20 client.16722 trim_cache size 0 max 16384 2024-04-22T00:05:04.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.069+0000 7fcb27e05740 1 Processor -- start 2024-04-22T00:05:04.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 -- start start 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f4591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f43d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c00fe90 con 0x55e27f444020 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c011b50 con 0x55e27f444940 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.070+0000 7fcb27e05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c010560 con 0x55e27f44d520 2024-04-22T00:05:04.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43544/0 (socket says 172.21.15.38:43544) 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3399037367 learned_addr learned my addr 172.21.15.38:0/3399037367 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f4591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3399037367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 msgr2=0x55e27f43d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f43d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3399037367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 msgr2=0x55e27f4591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f4591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.071+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3399037367 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e27f444770 con 0x55e27f444940 2024-04-22T00:05:04.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.072+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcb10021010 tx=0x7fcb1001fe00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff24295141ce3dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.074+0000 7fcb04ff9640 1 -- 172.21.15.38:0/3399037367 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1003b070 con 0x55e27f444940 2024-04-22T00:05:04.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.074+0000 7fcb04ff9640 1 -- 172.21.15.38:0/3399037367 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb10005e90 con 0x55e27f444940 2024-04-22T00:05:04.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.074+0000 7fcb04ff9640 1 -- 172.21.15.38:0/3399037367 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb100055e0 con 0x55e27f444940 2024-04-22T00:05:04.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.076+0000 7fcb27e05740 1 -- 172.21.15.38:0/3399037367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 msgr2=0x55e27f4493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.076+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcb10021010 tx=0x7fcb1001fe00 comp rx=0 tx=0).stop 2024-04-22T00:05:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.079+0000 7fcb27e05740 1 -- 172.21.15.38:0/3399037367 shutdown_connections 2024-04-22T00:05:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.079+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f43d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.079+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f4493f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.079+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3399037367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f4591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.079+0000 7fcb27e05740 1 -- 172.21.15.38:0/3399037367 >> 172.21.15.38:0/3399037367 conn(0x55e27f43e750 msgr2=0x55e27f43cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.081+0000 7fcb27e05740 1 -- 172.21.15.38:0/3399037367 shutdown_connections 2024-04-22T00:05:04.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.081+0000 7fcb27e05740 1 -- 172.21.15.38:0/3399037367 wait complete. 2024-04-22T00:05:04.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.082+0000 7fcb27e05740 1 -- start start 2024-04-22T00:05:04.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.082+0000 7fcb27e05740 1 Processor -- start 2024-04-22T00:05:04.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.083+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f455780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.083+0000 7fcb27e05740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f450c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.083+0000 7fcb1ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f455780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.083+0000 7fcb1ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f455780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36930/0 (socket says 172.21.15.38:36930) 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb1ed9d640 1 -- 172.21.15.38:0/3817642964 learned_addr learned my addr 172.21.15.38:0/3817642964 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f450c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f4511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c0113c0 con 0x55e27f444020 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c010cf0 con 0x55e27f444940 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb1f59e640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f4511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.084+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c00de70 con 0x55e27f44d520 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 msgr2=0x55e27f4511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f4511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3817642964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 msgr2=0x55e27f455780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f455780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e27f1867d0 con 0x55e27f444940 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb27e05740 10 client.0 _ll_register_callbacks cb 0x7fcb1400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.085+0000 7fcb1e59c640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f450c90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55e27f44b6a0 tx=0x7fcb1000bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=971f4da5b3caca8b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb10005e90 con 0x55e27f444940 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb02ff5640 10 client.26132 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb02ff5640 1 -- 172.21.15.38:0/3817642964 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1003b070 con 0x55e27f444940 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb02ff5640 1 -- 172.21.15.38:0/3817642964 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fcb10005590 con 0x55e27f444940 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb02ff5640 1 -- 172.21.15.38:0/3817642964 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcb10014410 con 0x55e27f444940 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb27e05740 10 client.26132 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:04.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.086+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55e27f4d4460 con 0x55e27f444940 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb02ff5640 1 -- 172.21.15.38:0/3817642964 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fcb1000f840 con 0x55e27f444940 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 10 client.26132 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 10 client.26132 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 20 client.26132 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f4d4840 0x55e27f4d6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 4 client.26132 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.087+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e27f3afdc0 con 0x55e27f4d4840 2024-04-22T00:05:04.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.093+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f4d4840 0x55e27f4d6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.094+0000 7fcb1ed9d640 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f4d4840 0x55e27f4d6c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7262a7860c3fbc4f server_cookie=8d8934c81fadf689 in_seq=0 out_seq=0 2024-04-22T00:05:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.094+0000 7fcb02ff5640 10 client.26132 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.094+0000 7fcb02ff5640 1 -- 172.21.15.38:0/3817642964 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e27f3afdc0 con 0x55e27f4d4840 2024-04-22T00:05:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.094+0000 7fcb02ff5640 10 client.26132 handle_command_reply: tid=0 2024-04-22T00:05:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.094+0000 7fcb27e05740 1 client.26132 shutdown 2024-04-22T00:05:04.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 msgr2=0x55e27f450c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f450c90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55e27f44b6a0 tx=0x7fcb1000bd30 comp rx=0 tx=0).stop 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 shutdown_connections 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e27f4d4840 0x55e27f4d6c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e27f44d520 0x55e27f4511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e27f444940 0x55e27f450c90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 --2- 172.21.15.38:0/3817642964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e27f444020 0x55e27f455780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.095+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 >> 172.21.15.38:0/3817642964 conn(0x55e27f43e750 msgr2=0x55e27f43fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.096+0000 7fcb02ff5640 0 client.26132 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:04.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.096+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 shutdown_connections 2024-04-22T00:05:04.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.096+0000 7fcb27e05740 1 -- 172.21.15.38:0/3817642964 wait complete. 2024-04-22T00:05:04.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.097+0000 7fcb27e05740 20 client.26132 trim_cache size 0 max 16384 2024-04-22T00:05:04.096 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:04.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.104+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 msgr2=0x7fcb080a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.104+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 0x7fcb080a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcb10012760 tx=0x7fcb1001a000 comp rx=0 tx=0).stop 2024-04-22T00:05:04.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.104+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 msgr2=0x7fcb2006c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.104+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb18037040 tx=0x7fcb18035cf0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 shutdown_connections 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcb080a2ed0 0x7fcb080a5390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb20105ad0 0x7fcb2006c4d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb201010d0 0x7fcb2006bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 --2- 172.21.15.38:0/709809270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb201007b0 0x7fcb20072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 >> 172.21.15.38:0/709809270 conn(0x7fcb20077740 msgr2=0x7fcb20103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 shutdown_connections 2024-04-22T00:05:04.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.107+0000 7fcb25147640 1 -- 172.21.15.38:0/709809270 wait complete. 2024-04-22T00:05:04.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:05:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:04 smithi038 ceph-mon[27972]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 60 MiB/s wr, 9.46k op/s 2024-04-22T00:05:04.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:04 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3884812127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:04 smithi203 ceph-mon[31431]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 60 MiB/s wr, 9.46k op/s 2024-04-22T00:05:04.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:04 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3884812127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:04 smithi111 ceph-mon[31543]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 60 MiB/s wr, 9.46k op/s 2024-04-22T00:05:04.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:04 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3884812127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:04.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.642+0000 7f9c9d89d640 1 -- 172.21.15.38:0/1239320222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c981020d0 msgr2=0x7f9c9810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.642+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/1239320222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c981020d0 0x7f9c9810e5e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009cd0 tx=0x7f9c9002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 -- 172.21.15.38:0/1239320222 shutdown_connections 2024-04-22T00:05:04.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/1239320222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c981020d0 0x7f9c9810e5e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/1239320222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c98101710 0x7f9c98101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/1239320222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98107720 0x7f9c98107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 -- 172.21.15.38:0/1239320222 >> 172.21.15.38:0/1239320222 conn(0x7f9c98078b90 msgr2=0x7f9c98078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 -- 172.21.15.38:0/1239320222 shutdown_connections 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 -- 172.21.15.38:0/1239320222 wait complete. 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 Processor -- start 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 -- start start 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98101710 0x7f9c98073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.643+0000 7f9c9d89d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c981020d0 0x7f9c98073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98111b30 con 0x7f9c98107720 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981119b0 con 0x7f9c98101710 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c98111cb0 con 0x7f9c981020d0 2024-04-22T00:05:04.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36952/0 (socket says 172.21.15.38:36952) 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 -- 172.21.15.38:0/2439368477 learned_addr learned my addr 172.21.15.38:0/2439368477 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c981020d0 0x7f9c98073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98101710 0x7f9c98073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 -- 172.21.15.38:0/2439368477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c981020d0 msgr2=0x7f9c98073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c981020d0 0x7f9c98073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 -- 172.21.15.38:0/2439368477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98101710 msgr2=0x7f9c98073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98101710 0x7f9c98073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 -- 172.21.15.38:0/2439368477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c981a6510 con 0x7f9c98107720 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9c90004af0 tx=0x7f9c90032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9c89b640 1 -- 172.21.15.38:0/2439368477 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90041020 con 0x7f9c98107720 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9c89b640 1 -- 172.21.15.38:0/2439368477 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c90034cd0 con 0x7f9c98107720 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c981a67a0 con 0x7f9c98107720 2024-04-22T00:05:04.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9c89b640 1 -- 172.21.15.38:0/2439368477 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90039040 con 0x7f9c98107720 2024-04-22T00:05:04.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.644+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c981a6c90 con 0x7f9c98107720 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9c9c89b640 1 -- 172.21.15.38:0/2439368477 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9c90044020 con 0x7f9c98107720 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 Processor -- start 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 -- start start 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc020 0x556e0c8fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc940 0x556e0c904fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9c9c89b640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 0x7f9c840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556e0c860530 con 0x556e0c905520 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556e0c808830 con 0x556e0c8fc940 2024-04-22T00:05:04.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.647+0000 7f9ca0566740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556e0c85c4f0 con 0x556e0c8fc020 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 0x7f9c840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc940 0x556e0c904fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36960/0 (socket says 172.21.15.38:36960) 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 -- 172.21.15.38:0/2328468138 learned_addr learned my addr 172.21.15.38:0/2328468138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc020 0x556e0c8fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c9c89b640 1 -- 172.21.15.38:0/2439368477 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9c900f13b0 con 0x7f9c98107720 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 -- 172.21.15.38:0/2328468138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc020 msgr2=0x556e0c8fc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc020 0x556e0c8fc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 -- 172.21.15.38:0/2328468138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc940 msgr2=0x556e0c904fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc940 0x556e0c904fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 -- 172.21.15.38:0/2328468138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e0c82c2c0 con 0x556e0c905520 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9c900bd630 tx=0x7f9c90032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8329d670533c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8dffb640 1 -- 172.21.15.38:0/2328468138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c900bd470 con 0x556e0c905520 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8dffb640 1 -- 172.21.15.38:0/2328468138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c900348f0 con 0x556e0c905520 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 0x7f9c840a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c980754d0 tx=0x7f9c80009910 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.648+0000 7f9c8dffb640 1 -- 172.21.15.38:0/2328468138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c900bdda0 con 0x556e0c905520 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- 172.21.15.38:0/2328468138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 msgr2=0x556e0c907910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9c900bd630 tx=0x7f9c90032b00 comp rx=0 tx=0).stop 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- 172.21.15.38:0/2328468138 shutdown_connections 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c907910 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc940 0x556e0c904fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2328468138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc020 0x556e0c8fc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- 172.21.15.38:0/2328468138 >> 172.21.15.38:0/2328468138 conn(0x556e0c8f6750 msgr2=0x556e0c8f6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- 172.21.15.38:0/2328468138 shutdown_connections 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- 172.21.15.38:0/2328468138 wait complete. 2024-04-22T00:05:04.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.649+0000 7f9ca0566740 1 -- start start 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 Processor -- start 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 0x556e0c904f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9022c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c9028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556e0c9080d0 con 0x556e0c905520 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556e0c81bf10 con 0x556e0c8fc020 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556e0c908250 con 0x556e0c8fc940 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9ca0566740 10 client.0 _ll_register_callbacks cb 0x556e0c573be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c8ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 0x556e0c904f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c9028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c8ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:38932/0 (socket says 172.21.15.38:38932) 2024-04-22T00:05:04.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c8ffff640 1 -- 172.21.15.38:0/216309931 learned_addr learned my addr 172.21.15.38:0/216309931 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 -- 172.21.15.38:0/216309931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 msgr2=0x556e0c9022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 -- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 msgr2=0x556e0c904f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 0x556e0c904f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c977fe640 1 -- 172.21.15.38:0/216309931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e0c7edc80 con 0x556e0c905520 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.650+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 0x556e0c904f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c977fe640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c9028d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x556e0c907ee0 tx=0x7f9c9002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd2196581bdab73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c900be7d0 con 0x556e0c905520 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9c900eea40 con 0x556e0c905520 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c900348f0 con 0x556e0c905520 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9c900bd000 con 0x556e0c905520 2024-04-22T00:05:04.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 10 client.16740 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e0c5743f0 con 0x556e0c905520 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c900348f0 con 0x556e0c905520 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f9c9003c030 con 0x556e0c905520 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 10 client.16740 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 10 client.16740 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 20 client.16740 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.651+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c915170 0x556e0c917560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.652+0000 7f9ca0566740 4 client.16740 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.652+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e0c86da70 con 0x556e0c915170 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.652+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c915170 0x556e0c917560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.652+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c915170 0x556e0c917560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=76aed20744044ea7 server_cookie=e7f813a5ffc76010 in_seq=0 out_seq=0 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.652+0000 7f9c76ffd640 10 client.16740 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.653+0000 7f9c76ffd640 1 -- 172.21.15.38:0/216309931 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e0c86da70 con 0x556e0c915170 2024-04-22T00:05:04.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.653+0000 7f9c76ffd640 10 client.16740 handle_command_reply: tid=0 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.653+0000 7f9ca0566740 1 client.16740 shutdown 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 msgr2=0x556e0c9028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c9028d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x556e0c907ee0 tx=0x7f9c9002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 shutdown_connections 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c915170 0x556e0c917560 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c905520 0x556e0c9028d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9c76ffd640 0 client.16740 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 --2- 172.21.15.38:0/216309931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c8fc020 0x556e0c904f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 >> 172.21.15.38:0/216309931 conn(0x556e0c8f6750 msgr2=0x556e0c8f84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 shutdown_connections 2024-04-22T00:05:04.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 1 -- 172.21.15.38:0/216309931 wait complete. 2024-04-22T00:05:04.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.654+0000 7f9ca0566740 20 client.16740 trim_cache size 0 max 16384 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 Processor -- start 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 -- start start 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c904f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981b0c10 con 0x556e0c8fc020 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981b28d0 con 0x556e0c905520 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9ca0566740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981b12e0 con 0x556e0c8fc940 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36976/0 (socket says 172.21.15.38:36976) 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 -- 172.21.15.38:0/1231814817 learned_addr learned my addr 172.21.15.38:0/1231814817 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c977fe640 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c904f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 -- 172.21.15.38:0/1231814817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 msgr2=0x556e0c9049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 -- 172.21.15.38:0/1231814817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 msgr2=0x556e0c904f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c904f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 -- 172.21.15.38:0/1231814817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e0c8fc770 con 0x556e0c8fc020 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.669+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x556e0c917c20 tx=0x7f9c8800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a934cc18ef67e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9c8d7fa640 1 -- 172.21.15.38:0/1231814817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88004220 con 0x556e0c8fc020 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9c8d7fa640 1 -- 172.21.15.38:0/1231814817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88023030 con 0x556e0c8fc020 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9c8d7fa640 1 -- 172.21.15.38:0/1231814817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88004e30 con 0x556e0c8fc020 2024-04-22T00:05:04.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 -- 172.21.15.38:0/1231814817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 msgr2=0x556e0c910e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x556e0c917c20 tx=0x7f9c8800b410 comp rx=0 tx=0).stop 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 -- 172.21.15.38:0/1231814817 shutdown_connections 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c904f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c9049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 --2- 172.21.15.38:0/1231814817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c910e70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 -- 172.21.15.38:0/1231814817 >> 172.21.15.38:0/1231814817 conn(0x556e0c8f6750 msgr2=0x556e0c8f5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 -- 172.21.15.38:0/1231814817 shutdown_connections 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.670+0000 7f9ca0566740 1 -- 172.21.15.38:0/1231814817 wait complete. 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 -- start start 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 Processor -- start 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c909320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c909860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981b2140 con 0x556e0c8fc020 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981b1a70 con 0x556e0c905520 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c981aebf0 con 0x556e0c8fc940 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9ca0566740 10 client.0 _ll_register_callbacks cb 0x7f9c8800b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9c96ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36978/0 (socket says 172.21.15.38:36978) 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9c96ffd640 1 -- 172.21.15.38:0/2518333142 learned_addr learned my addr 172.21.15.38:0/2518333142 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.671+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c909320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c909860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 -- 172.21.15.38:0/2518333142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 msgr2=0x556e0c909320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c909320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 -- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 msgr2=0x556e0c909860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c909860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 -- 172.21.15.38:0/2518333142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e0c63e7d0 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c977fe640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c909860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c909320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c96ffd640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9c880049d0 tx=0x7f9c880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5660c5647d684f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8801f3b0 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9c8801f550 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c88023030 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9c88039050 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9ca0566740 10 client.16746 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556e0c97b650 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.672+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88023030 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f9c88030500 con 0x556e0c8fc020 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 10 client.16746 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:04.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 10 client.16746 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 20 client.16746 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c97bce0 0x556e0c97e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 4 client.16746 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e0c867dc0 con 0x556e0c97bce0 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c97bce0 0x556e0c97e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9c8ffff640 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c97bce0 0x556e0c97e100 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=955d3694dfe26ad3 server_cookie=9d5f5307450d43a3 in_seq=0 out_seq=0 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.673+0000 7f9c76ffd640 10 client.16746 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.674+0000 7f9c76ffd640 1 -- 172.21.15.38:0/2518333142 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e0c867dc0 con 0x556e0c97bce0 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.674+0000 7f9c76ffd640 10 client.16746 handle_command_reply: tid=0 2024-04-22T00:05:04.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.674+0000 7f9ca0566740 1 client.16746 shutdown 2024-04-22T00:05:04.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.675+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 msgr2=0x556e0c908de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.675+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9c880049d0 tx=0x7f9c880047c0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 shutdown_connections 2024-04-22T00:05:04.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556e0c905520 0x556e0c909860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556e0c8fc940 0x556e0c909320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x556e0c97bce0 0x556e0c97e100 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 --2- 172.21.15.38:0/2518333142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e0c8fc020 0x556e0c908de0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 >> 172.21.15.38:0/2518333142 conn(0x556e0c8f6750 msgr2=0x556e0c8fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.683+0000 7f9c76ffd640 0 client.16746 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.691+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 shutdown_connections 2024-04-22T00:05:04.687 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:04.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.692+0000 7f9ca0566740 1 -- 172.21.15.38:0/2518333142 wait complete. 2024-04-22T00:05:04.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.692+0000 7f9ca0566740 20 client.16746 trim_cache size 0 max 16384 2024-04-22T00:05:04.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.698+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 msgr2=0x7f9c840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.698+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 0x7f9c840a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c980754d0 tx=0x7f9c80009910 comp rx=0 tx=0).stop 2024-04-22T00:05:04.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.698+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 msgr2=0x7f9c980744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.698+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9c90004af0 tx=0x7f9c90032ec0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.704+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 shutdown_connections 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.704+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9c840a2d00 0x7f9c840a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.705+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c98107720 0x7f9c980744c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.705+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c981020d0 0x7f9c98073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.705+0000 7f9c9d89d640 1 --2- 172.21.15.38:0/2439368477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c98101710 0x7f9c98073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.705+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 >> 172.21.15.38:0/2439368477 conn(0x7f9c98078b90 msgr2=0x7f9c980ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.706+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 shutdown_connections 2024-04-22T00:05:04.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.706+0000 7f9c9d89d640 1 -- 172.21.15.38:0/2439368477 wait complete. 2024-04-22T00:05:04.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:05:04.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.917+0000 7f034c313640 1 -- 172.21.15.38:0/3458574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 msgr2=0x7f034410a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.917+0000 7f034c313640 1 --2- 172.21.15.38:0/3458574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034410a4c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0340009930 tx=0x7f034002bc50 comp rx=0 tx=0).stop 2024-04-22T00:05:04.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.918+0000 7f034c313640 1 -- 172.21.15.38:0/3458574101 shutdown_connections 2024-04-22T00:05:04.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.918+0000 7f034c313640 1 --2- 172.21.15.38:0/3458574101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0344113690 0x7f0344115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.918+0000 7f034c313640 1 --2- 172.21.15.38:0/3458574101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f034410aa00 0x7f0344113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.918+0000 7f034c313640 1 --2- 172.21.15.38:0/3458574101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034410a4c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.918+0000 7f034c313640 1 -- 172.21.15.38:0/3458574101 >> 172.21.15.38:0/3458574101 conn(0x7f034406c850 msgr2=0x7f034406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.920+0000 7f034c313640 1 -- 172.21.15.38:0/3458574101 shutdown_connections 2024-04-22T00:05:04.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.920+0000 7f034c313640 1 -- 172.21.15.38:0/3458574101 wait complete. 2024-04-22T00:05:04.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 Processor -- start 2024-04-22T00:05:04.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 -- start start 2024-04-22T00:05:04.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034410aa00 0x7f034419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0344119280 con 0x7f034410a0e0 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0344119100 con 0x7f0344113690 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.921+0000 7f034c313640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0344119400 con 0x7f034410aa00 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a0af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43622/0 (socket says 172.21.15.38:43622) 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 -- 172.21.15.38:0/3408963986 learned_addr learned my addr 172.21.15.38:0/3408963986 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a0af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36988/0 (socket says 172.21.15.38:36988) 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 -- 172.21.15.38:0/3408963986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034410aa00 msgr2=0x7f034419fca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034410aa00 0x7f034419fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 -- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 msgr2=0x7f034419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.922+0000 7f034a8b0640 1 -- 172.21.15.38:0/3408963986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03441a4750 con 0x7f0344113690 2024-04-22T00:05:04.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.923+0000 7f034a8b0640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f033c00eb10 tx=0x7f033c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.924+0000 7f033b7fe640 1 -- 172.21.15.38:0/3408963986 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c004b20 con 0x7f0344113690 2024-04-22T00:05:04.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.924+0000 7f033b7fe640 1 -- 172.21.15.38:0/3408963986 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c004cc0 con 0x7f0344113690 2024-04-22T00:05:04.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.924+0000 7f033b7fe640 1 -- 172.21.15.38:0/3408963986 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0107b0 con 0x7f0344113690 2024-04-22T00:05:04.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.924+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03441a4a40 con 0x7f0344113690 2024-04-22T00:05:04.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.924+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03441ac2d0 con 0x7f0344113690 2024-04-22T00:05:04.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.928+0000 7f034efde740 1 Processor -- start 2024-04-22T00:05:04.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 -- start start 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff09aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f033b7fe640 1 -- 172.21.15.38:0/3408963986 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f033c005000 con 0x7f0344113690 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff11590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56003fe61530 con 0x56003fefc1d0 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56003fe09830 con 0x56003fefcb70 2024-04-22T00:05:04.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.931+0000 7f034efde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56003fe5d4f0 con 0x56003ff096c0 2024-04-22T00:05:04.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.932+0000 7f033b7fe640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 0x7f03300a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.932+0000 7f033b7fe640 1 -- 172.21.15.38:0/3408963986 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f033c0cfbc0 con 0x7f0344113690 2024-04-22T00:05:04.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.934+0000 7f034a8b0640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 0x7f03300a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.935+0000 7f034a0af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.935+0000 7f034a0af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36990/0 (socket says 172.21.15.38:36990) 2024-04-22T00:05:04.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.935+0000 7f034a0af640 1 -- 172.21.15.38:0/1672926394 learned_addr learned my addr 172.21.15.38:0/1672926394 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a0af640 1 -- 172.21.15.38:0/1672926394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 msgr2=0x56003ff09aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f03498ae640 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff11590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a0af640 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff09aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a0af640 1 -- 172.21.15.38:0/1672926394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 msgr2=0x56003ff11590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a0af640 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff11590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a8b0640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 0x7f03300a53c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f033c0981a0 tx=0x7f033c005b40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.939+0000 7f034a0af640 1 -- 172.21.15.38:0/1672926394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003fe2d2c0 con 0x56003fefc1d0 2024-04-22T00:05:04.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.940+0000 7f034a0af640 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f034002c1e0 tx=0x7f034002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13b3170056b5d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.941+0000 7f0338bfe640 1 -- 172.21.15.38:0/1672926394 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0340045070 con 0x56003fefc1d0 2024-04-22T00:05:04.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.941+0000 7f0338bfe640 1 -- 172.21.15.38:0/1672926394 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340004040 con 0x56003fefc1d0 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.942+0000 7f034efde740 1 -- 172.21.15.38:0/1672926394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 msgr2=0x56003fefc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.942+0000 7f034efde740 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f034002c1e0 tx=0x7f034002c5f0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.942+0000 7f03498ae640 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff11590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 -- 172.21.15.38:0/1672926394 shutdown_connections 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff11590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003fefc630 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 --2- 172.21.15.38:0/1672926394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff09aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 -- 172.21.15.38:0/1672926394 >> 172.21.15.38:0/1672926394 conn(0x56003fef71c0 msgr2=0x56003fef75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 -- 172.21.15.38:0/1672926394 shutdown_connections 2024-04-22T00:05:04.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.943+0000 7f034efde740 1 -- 172.21.15.38:0/1672926394 wait complete. 2024-04-22T00:05:04.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.944+0000 7f034efde740 1 -- start start 2024-04-22T00:05:04.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.944+0000 7f034efde740 1 Processor -- start 2024-04-22T00:05:04.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff11310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff0e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56003ff11b30 con 0x56003fefc1d0 2024-04-22T00:05:04.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56003fe1cf10 con 0x56003ff096c0 2024-04-22T00:05:04.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56003ff11cb0 con 0x56003fefcb70 2024-04-22T00:05:04.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.952+0000 7f034efde740 10 client.0 _ll_register_callbacks cb 0x56003fb74be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:04.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.954+0000 7f03498ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.954+0000 7f03498ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43640/0 (socket says 172.21.15.38:43640) 2024-04-22T00:05:04.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.954+0000 7f03498ae640 1 -- 172.21.15.38:0/3559900503 learned_addr learned my addr 172.21.15.38:0/3559900503 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.954+0000 7f03498ae640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff11310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f034a0af640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff0e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 -- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 msgr2=0x56003ff0e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff0e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 -- 172.21.15.38:0/3559900503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 msgr2=0x56003ff11310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff11310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 -- 172.21.15.38:0/3559900503 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56003fdeec80 con 0x56003ff096c0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.955+0000 7f03498ae640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0334001e90 tx=0x7f033400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ff46bed501981b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.956+0000 7f032dffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.956+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334008b80 con 0x56003ff096c0 2024-04-22T00:05:04.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.956+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0334008d20 con 0x56003ff096c0 2024-04-22T00:05:04.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.957+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0340004040 con 0x56003ff096c0 2024-04-22T00:05:04.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.959+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0334007160 con 0x56003ff096c0 2024-04-22T00:05:04.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.961+0000 7f034efde740 10 client.26153 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:04.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.961+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56003fb753f0 con 0x56003ff096c0 2024-04-22T00:05:04.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.963+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0340004040 con 0x56003ff096c0 2024-04-22T00:05:04.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.963+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f0334009030 con 0x56003ff096c0 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 10 client.26153 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 10 client.26153 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 20 client.26153 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff01210 0x56003ff03630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 4 client.26153 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:04.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56003fe6ea70 con 0x56003ff01210 2024-04-22T00:05:04.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.964+0000 7f034a0af640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff01210 0x56003ff03630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.966+0000 7f032dffb640 10 client.26153 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:04.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.966+0000 7f034a0af640 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff01210 0x56003ff03630 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e03d9259d3b3d285 server_cookie=6bd11a2b2ad74844 in_seq=0 out_seq=0 2024-04-22T00:05:04.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f032dffb640 1 -- 172.21.15.38:0/3559900503 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56003fe6ea70 con 0x56003ff01210 2024-04-22T00:05:04.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f032dffb640 10 client.26153 handle_command_reply: tid=0 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 client.26153 shutdown 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 msgr2=0x56003ff0ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0334001e90 tx=0x7f033400b040 comp rx=0 tx=0).stop 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 shutdown_connections 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff01210 0x56003ff03630 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff0ebe0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff0e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f032dffb640 0 client.26153 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 --2- 172.21.15.38:0/3559900503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff11310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.967+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 >> 172.21.15.38:0/3559900503 conn(0x56003fef71c0 msgr2=0x56003ff0ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.968+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 shutdown_connections 2024-04-22T00:05:04.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.970+0000 7f034efde740 1 -- 172.21.15.38:0/3559900503 wait complete. 2024-04-22T00:05:04.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.970+0000 7f034efde740 20 client.26153 trim_cache size 0 max 16384 2024-04-22T00:05:04.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.998+0000 7f034efde740 1 Processor -- start 2024-04-22T00:05:04.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.999+0000 7f034efde740 1 -- start start 2024-04-22T00:05:04.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.999+0000 7f034efde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:04.999+0000 7f034efde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff0fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f034efde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff10300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f034efde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f032800fe80 con 0x56003fefc1d0 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f034efde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0328011b40 con 0x56003fefcb70 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f034efde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0328010550 con 0x56003ff096c0 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f03498ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f03498ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37012/0 (socket says 172.21.15.38:37012) 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f03498ae640 1 -- 172.21.15.38:0/3187535723 learned_addr learned my addr 172.21.15.38:0/3187535723 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.000+0000 7f03498ae640 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff10300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.001+0000 7f03498ae640 1 -- 172.21.15.38:0/3187535723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 msgr2=0x56003ff10300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.001+0000 7f03498ae640 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff10300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.001+0000 7f03498ae640 1 -- 172.21.15.38:0/3187535723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 msgr2=0x56003ff0fdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.001+0000 7f03498ae640 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff0fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.001+0000 7f03498ae640 1 -- 172.21.15.38:0/3187535723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003fefc000 con 0x56003fefc1d0 2024-04-22T00:05:04.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.002+0000 7f03498ae640 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0334009430 tx=0x7f03340013d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267ffd019f7791b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:04.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.003+0000 7f032ffff640 1 -- 172.21.15.38:0/3187535723 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334009af0 con 0x56003fefc1d0 2024-04-22T00:05:04.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.003+0000 7f032ffff640 1 -- 172.21.15.38:0/3187535723 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0334009620 con 0x56003fefc1d0 2024-04-22T00:05:04.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.003+0000 7f032ffff640 1 -- 172.21.15.38:0/3187535723 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03340047d0 con 0x56003fefc1d0 2024-04-22T00:05:04.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.005+0000 7f034efde740 1 -- 172.21.15.38:0/3187535723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 msgr2=0x56003ff1aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:04.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.005+0000 7f034efde740 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0334009430 tx=0x7f03340013d0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 -- 172.21.15.38:0/3187535723 shutdown_connections 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003ff096c0 0x56003ff10300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003fefcb70 0x56003ff0fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 --2- 172.21.15.38:0/3187535723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff1aba0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 -- 172.21.15.38:0/3187535723 >> 172.21.15.38:0/3187535723 conn(0x56003fef71c0 msgr2=0x56003ff08b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 -- 172.21.15.38:0/3187535723 shutdown_connections 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 -- 172.21.15.38:0/3187535723 wait complete. 2024-04-22T00:05:04.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 -- start start 2024-04-22T00:05:05.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.006+0000 7f034efde740 1 Processor -- start 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.007+0000 7f034efde740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff17580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff05230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03280113b0 con 0x56003fefc1d0 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0328010ce0 con 0x56003ff096c0 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f032800de00 con 0x56003fefcb70 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034efde740 10 client.0 _ll_register_callbacks cb 0x7f0334001470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f03498ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f03498ae640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43666/0 (socket says 172.21.15.38:43666) 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f03498ae640 1 -- 172.21.15.38:0/2564018519 learned_addr learned my addr 172.21.15.38:0/2564018519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.008+0000 7f034a0af640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff05230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 -- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 msgr2=0x56003ff05230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff05230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 -- 172.21.15.38:0/2564018519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 msgr2=0x56003ff17580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff17580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 -- 172.21.15.38:0/2564018519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56003fc3f7d0 con 0x56003ff096c0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f03498ae640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0334016350 tx=0x7f0334001a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7897638e53f2786 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f032dffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033405a020 con 0x56003ff096c0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0334009620 con 0x56003ff096c0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.009+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0334007300 con 0x56003ff096c0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.010+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0334065050 con 0x56003ff096c0 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.010+0000 7f034efde740 10 client.26156 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:05.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.010+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56003ff8dd80 con 0x56003ff096c0 2024-04-22T00:05:05.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.010+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0334009620 con 0x56003ff096c0 2024-04-22T00:05:05.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f03340047d0 con 0x56003ff096c0 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 10 client.26156 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 10 client.26156 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 20 client.26156 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff8e410 0x56003ff90830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 4 client.26156 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.013+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56003fe68dc0 con 0x56003ff8e410 2024-04-22T00:05:05.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.014+0000 7f034a0af640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff8e410 0x56003ff90830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.014+0000 7f034a0af640 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff8e410 0x56003ff90830 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e843170df834946d server_cookie=f9d3ccb798ff2796 in_seq=0 out_seq=0 2024-04-22T00:05:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.014+0000 7f032dffb640 10 client.26156 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.015+0000 7f032dffb640 1 -- 172.21.15.38:0/2564018519 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56003fe68dc0 con 0x56003ff8e410 2024-04-22T00:05:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.015+0000 7f032dffb640 10 client.26156 handle_command_reply: tid=0 2024-04-22T00:05:05.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.015+0000 7f034efde740 1 client.26156 shutdown 2024-04-22T00:05:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 msgr2=0x56003ff058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0334016350 tx=0x7f0334001a50 comp rx=0 tx=0).stop 2024-04-22T00:05:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 shutdown_connections 2024-04-22T00:05:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x56003ff8e410 0x56003ff90830 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56003ff096c0 0x56003ff058b0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0334016350 tx=0x7f0334001a50 comp rx=0 tx=0).stop 2024-04-22T00:05:05.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56003fefcb70 0x56003ff05230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 --2- 172.21.15.38:0/2564018519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56003fefc1d0 0x56003ff17580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.016+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 >> 172.21.15.38:0/2564018519 conn(0x56003fef71c0 msgr2=0x56003fef5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.017+0000 7f032dffb640 0 client.26156 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:05.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.017+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 shutdown_connections 2024-04-22T00:05:05.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.017+0000 7f034efde740 1 -- 172.21.15.38:0/2564018519 wait complete. 2024-04-22T00:05:05.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.018+0000 7f034efde740 20 client.26156 trim_cache size 0 max 16384 2024-04-22T00:05:05.012 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:05.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.018+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 msgr2=0x7f03300a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.018+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 0x7f03300a53c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f033c0981a0 tx=0x7f033c005b40 comp rx=0 tx=0).stop 2024-04-22T00:05:05.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 msgr2=0x7f03441a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f033c00eb10 tx=0x7f033c00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 shutdown_connections 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03300a2f00 0x7f03300a53c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0344113690 0x7f03441a4020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034410aa00 0x7f034419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 --2- 172.21.15.38:0/3408963986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f034410a0e0 0x7f034419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.019+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 >> 172.21.15.38:0/3408963986 conn(0x7f034406c850 msgr2=0x7f03440703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.020+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 shutdown_connections 2024-04-22T00:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.020+0000 7f034c313640 1 -- 172.21.15.38:0/3408963986 wait complete. 2024-04-22T00:05:05.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:05:05.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.222+0000 7fe94f0c2640 1 -- 172.21.15.38:0/3931288140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948105ad0 msgr2=0x7fe948105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.222+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/3931288140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948105ad0 0x7fe948105f60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe938009860 tx=0x7fe93802bcb0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.223+0000 7fe94f0c2640 1 -- 172.21.15.38:0/3931288140 shutdown_connections 2024-04-22T00:05:05.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.223+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/3931288140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948105ad0 0x7fe948105f60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.223+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/3931288140 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 0x7fe948105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.223+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/3931288140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9481007b0 0x7fe948100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.223+0000 7fe94f0c2640 1 -- 172.21.15.38:0/3931288140 >> 172.21.15.38:0/3931288140 conn(0x7fe948077740 msgr2=0x7fe948077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.225+0000 7fe94f0c2640 1 -- 172.21.15.38:0/3931288140 shutdown_connections 2024-04-22T00:05:05.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.226+0000 7fe94f0c2640 1 -- 172.21.15.38:0/3931288140 wait complete. 2024-04-22T00:05:05.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.226+0000 7fe94f0c2640 1 Processor -- start 2024-04-22T00:05:05.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.226+0000 7fe94f0c2640 1 -- start start 2024-04-22T00:05:05.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 0x7fe948110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe948105ad0 0x7fe948111050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481094b0 con 0x7fe9481010d0 2024-04-22T00:05:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe948109330 con 0x7fe948105ad0 2024-04-22T00:05:05.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.227+0000 7fe94f0c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe948109630 con 0x7fe9481007b0 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39022/0 (socket says 172.21.15.38:39022) 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 -- 172.21.15.38:0/1286233251 learned_addr learned my addr 172.21.15.38:0/1286233251 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe948105ad0 0x7fe948111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe947fff640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 0x7fe948110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 -- 172.21.15.38:0/1286233251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe948105ad0 msgr2=0x7fe948111050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe948105ad0 0x7fe948111050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 -- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 msgr2=0x7fe948110b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 0x7fe948110b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.228+0000 7fe94ce5e640 1 -- 172.21.15.38:0/1286233251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe948111780 con 0x7fe9481007b0 2024-04-22T00:05:05.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.230+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe94001fe10 tx=0x7fe94001d750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.231+0000 7fe945ffb640 1 -- 172.21.15.38:0/1286233251 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94001de60 con 0x7fe9481007b0 2024-04-22T00:05:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.232+0000 7fe945ffb640 1 -- 172.21.15.38:0/1286233251 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe940003e50 con 0x7fe9481007b0 2024-04-22T00:05:05.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.232+0000 7fe945ffb640 1 -- 172.21.15.38:0/1286233251 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94001c7e0 con 0x7fe9481007b0 2024-04-22T00:05:05.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.232+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9481a2510 con 0x7fe9481007b0 2024-04-22T00:05:05.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.233+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9481a2920 con 0x7fe9481007b0 2024-04-22T00:05:05.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.237+0000 7fe945ffb640 1 -- 172.21.15.38:0/1286233251 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe94002e020 con 0x7fe9481007b0 2024-04-22T00:05:05.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.238+0000 7fe951d84740 1 Processor -- start 2024-04-22T00:05:05.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.238+0000 7fe951d84740 1 -- start start 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6aca5520 0x562c6aca7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c6ac00530 con 0x562c6aca5520 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe945ffb640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 0x7fe9300a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c6aba8830 con 0x562c6ac9c020 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe951d84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c6abfc4f0 con 0x562c6ac9c940 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe945ffb640 1 -- 172.21.15.38:0/1286233251 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe9400e0280 con 0x7fe9481007b0 2024-04-22T00:05:05.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.239+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 0x7fe9300a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43700/0 (socket says 172.21.15.38:43700) 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 -- 172.21.15.38:0/1881494396 learned_addr learned my addr 172.21.15.38:0/1881494396 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 0x7fe9300a53c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe938009830 tx=0x7fe938009510 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe947fff640 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6aca5520 0x562c6aca7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 -- 172.21.15.38:0/1881494396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 msgr2=0x562c6aca4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 -- 172.21.15.38:0/1881494396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6aca5520 msgr2=0x562c6aca7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6aca5520 0x562c6aca7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.240+0000 7fe94d65f640 1 -- 172.21.15.38:0/1881494396 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6abcc2c0 con 0x562c6ac9c020 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.241+0000 7fe94d65f640 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe938036000 tx=0x7fe938045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c82e9464b369b060 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.241+0000 7fe92effd640 1 -- 172.21.15.38:0/1881494396 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938051070 con 0x562c6ac9c020 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.241+0000 7fe92effd640 1 -- 172.21.15.38:0/1881494396 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe938046070 con 0x562c6ac9c020 2024-04-22T00:05:05.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.241+0000 7fe92effd640 1 -- 172.21.15.38:0/1881494396 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938039040 con 0x562c6ac9c020 2024-04-22T00:05:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 -- 172.21.15.38:0/1881494396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 msgr2=0x562c6ac9c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe938036000 tx=0x7fe938045000 comp rx=0 tx=0).stop 2024-04-22T00:05:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 -- 172.21.15.38:0/1881494396 shutdown_connections 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6aca5520 0x562c6aca7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 --2- 172.21.15.38:0/1881494396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6ac9c400 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 -- 172.21.15.38:0/1881494396 >> 172.21.15.38:0/1881494396 conn(0x562c6ac96750 msgr2=0x562c6ac96b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 -- 172.21.15.38:0/1881494396 shutdown_connections 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.242+0000 7fe951d84740 1 -- 172.21.15.38:0/1881494396 wait complete. 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.243+0000 7fe951d84740 1 -- start start 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.243+0000 7fe951d84740 1 Processor -- start 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.243+0000 7fe951d84740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6aca4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.243+0000 7fe951d84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe951d84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6aca2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe951d84740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c6aca82b0 con 0x562c6ac9c020 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe951d84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c6abbbf10 con 0x562c6aca5520 2024-04-22T00:05:05.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe951d84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c6aca8430 con 0x562c6ac9c940 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe951d84740 10 client.0 _ll_register_callbacks cb 0x562c6a913be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe947fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe947fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39036/0 (socket says 172.21.15.38:39036) 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe947fff640 1 -- 172.21.15.38:0/3740221190 learned_addr learned my addr 172.21.15.38:0/3740221190 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.244+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6aca4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 -- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 msgr2=0x562c6aca2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 -- 172.21.15.38:0/3740221190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 msgr2=0x562c6aca2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6aca2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 -- 172.21.15.38:0/3740221190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c6ab8dc80 con 0x562c6ac9c020 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6aca4f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe9400dd630 tx=0x7fe9400abf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbd6e077c130a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.245+0000 7fe927fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.247+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9400c5600 con 0x562c6ac9c020 2024-04-22T00:05:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.247+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe9400a8a70 con 0x562c6ac9c020 2024-04-22T00:05:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.247+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe938046070 con 0x562c6ac9c020 2024-04-22T00:05:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.248+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe9400acb00 con 0x562c6ac9c020 2024-04-22T00:05:05.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.248+0000 7fe951d84740 10 client.16773 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:05.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.248+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562c6a9143f0 con 0x562c6ac9c020 2024-04-22T00:05:05.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe938046070 con 0x562c6ac9c020 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fe9400ca570 con 0x562c6ac9c020 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe951d84740 10 client.16773 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe951d84740 10 client.16773 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe951d84740 20 client.16773 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.249+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6acb5a10 0x562c6acb7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.250+0000 7fe951d84740 4 client.16773 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.250+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c6ac0da70 con 0x562c6acb5a10 2024-04-22T00:05:05.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.250+0000 7fe947fff640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6acb5a10 0x562c6acb7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.255+0000 7fe947fff640 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6acb5a10 0x562c6acb7e70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b118e05aec23848 server_cookie=7e0ff3519a211542 in_seq=0 out_seq=0 2024-04-22T00:05:05.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.255+0000 7fe927fff640 10 client.16773 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:05.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.256+0000 7fe927fff640 1 -- 172.21.15.38:0/3740221190 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c6ac0da70 con 0x562c6acb5a10 2024-04-22T00:05:05.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.256+0000 7fe927fff640 10 client.16773 handle_command_reply: tid=0 2024-04-22T00:05:05.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.257+0000 7fe951d84740 1 client.16773 shutdown 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 msgr2=0x562c6aca4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6aca4f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe9400dd630 tx=0x7fe9400abf40 comp rx=0 tx=0).stop 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 shutdown_connections 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6acb5a10 0x562c6acb7e70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6aca2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6aca2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 --2- 172.21.15.38:0/3740221190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6aca4f70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.258+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 >> 172.21.15.38:0/3740221190 conn(0x562c6ac96750 msgr2=0x562c6ac984a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.260+0000 7fe927fff640 0 client.16773 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:05.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.260+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 shutdown_connections 2024-04-22T00:05:05.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.263+0000 7fe951d84740 1 -- 172.21.15.38:0/3740221190 wait complete. 2024-04-22T00:05:05.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.263+0000 7fe951d84740 20 client.16773 trim_cache size 0 max 16384 2024-04-22T00:05:05.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.290+0000 7fe951d84740 1 Processor -- start 2024-04-22T00:05:05.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.290+0000 7fe951d84740 1 -- start start 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.290+0000 7fe951d84740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6acb1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe951d84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6acbd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe951d84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6acbdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe951d84740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ac9c0 con 0x562c6ac9c020 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe951d84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ae680 con 0x562c6aca5520 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6acbdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6acbdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43724/0 (socket says 172.21.15.38:43724) 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe94d65f640 1 -- 172.21.15.38:0/1802662081 learned_addr learned my addr 172.21.15.38:0/1802662081 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ad090 con 0x562c6ac9c940 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe947fff640 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6acbd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.291+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6acb1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 -- 172.21.15.38:0/1802662081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 msgr2=0x562c6acbdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6acbdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 -- 172.21.15.38:0/1802662081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 msgr2=0x562c6acb1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6acb1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 -- 172.21.15.38:0/1802662081 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c6aca2db0 con 0x562c6ac9c940 2024-04-22T00:05:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.292+0000 7fe947fff640 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6acbd780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x562c6acb86c0 tx=0x7fe93c005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=753ea9f8f51b9c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.293+0000 7fe92e7fc640 1 -- 172.21.15.38:0/1802662081 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0140d0 con 0x562c6ac9c940 2024-04-22T00:05:05.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.293+0000 7fe92e7fc640 1 -- 172.21.15.38:0/1802662081 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c01f020 con 0x562c6ac9c940 2024-04-22T00:05:05.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.294+0000 7fe92e7fc640 1 -- 172.21.15.38:0/1802662081 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c01b590 con 0x562c6ac9c940 2024-04-22T00:05:05.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 msgr2=0x562c6acbd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6acbd780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x562c6acb86c0 tx=0x7fe93c005f30 comp rx=0 tx=0).stop 2024-04-22T00:05:05.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 shutdown_connections 2024-04-22T00:05:05.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6aca5520 0x562c6acbdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6ac9c940 0x562c6acbd780 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x562c6acb86c0 tx=0x7fe93c005f30 comp rx=0 tx=0).stop 2024-04-22T00:05:05.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 --2- 172.21.15.38:0/1802662081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9c020 0x562c6acb1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.295+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 >> 172.21.15.38:0/1802662081 conn(0x562c6ac96750 msgr2=0x562c6ac94fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.296+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 shutdown_connections 2024-04-22T00:05:05.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.296+0000 7fe951d84740 1 -- 172.21.15.38:0/1802662081 wait complete. 2024-04-22T00:05:05.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.296+0000 7fe951d84740 1 -- start start 2024-04-22T00:05:05.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.296+0000 7fe951d84740 1 Processor -- start 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6acad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6aca5520 0x562c6acadbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481adef0 con 0x562c6ac9ea70 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481ad820 con 0x562c6ac9c020 2024-04-22T00:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9481aa9a0 con 0x562c6aca5520 2024-04-22T00:05:05.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe951d84740 10 client.0 _ll_register_callbacks cb 0x7fe94001af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe94d65f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37078/0 (socket says 172.21.15.38:37078) 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.298+0000 7fe94d65f640 1 -- 172.21.15.38:0/2172315953 learned_addr learned my addr 172.21.15.38:0/2172315953 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 -- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6aca5520 msgr2=0x562c6acadbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6aca5520 0x562c6acadbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 -- 172.21.15.38:0/2172315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 msgr2=0x562c6acad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6acad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 -- 172.21.15.38:0/2172315953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c6a9de7d0 con 0x562c6ac9ea70 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.299+0000 7fe94d65f640 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe938004410 tx=0x7fe93802c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cee1b3434e9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.300+0000 7fe927fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:05.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.300+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938051070 con 0x562c6ac9ea70 2024-04-22T00:05:05.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.300+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe93802c650 con 0x562c6ac9ea70 2024-04-22T00:05:05.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.301+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe93c01f020 con 0x562c6ac9ea70 2024-04-22T00:05:05.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.303+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe938039040 con 0x562c6ac9ea70 2024-04-22T00:05:05.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.303+0000 7fe951d84740 10 client.16779 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:05.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.303+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562c6ad1bdc0 con 0x562c6ac9ea70 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.304+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe93c01f020 con 0x562c6ac9ea70 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.304+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7fe938003be0 con 0x562c6ac9ea70 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 10 client.16779 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 10 client.16779 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 20 client.16779 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:05.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6ad1c050 0x562c6ad1e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 4 client.16779 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:05.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c6ac07dc0 con 0x562c6ad1c050 2024-04-22T00:05:05.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.305+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6ad1c050 0x562c6ad1e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe927fff640 10 client.16779 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:05.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe94ce5e640 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6ad1c050 0x562c6ad1e460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78071ff3576a9a40 server_cookie=13aa6428e08244ad in_seq=0 out_seq=0 2024-04-22T00:05:05.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe927fff640 1 -- 172.21.15.38:0/2172315953 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c6ac07dc0 con 0x562c6ad1c050 2024-04-22T00:05:05.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe927fff640 10 client.16779 handle_command_reply: tid=0 2024-04-22T00:05:05.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe951d84740 1 client.16779 shutdown 2024-04-22T00:05:05.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 msgr2=0x562c6ac9eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.306+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe938004410 tx=0x7fe93802c4a0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 shutdown_connections 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x562c6ad1c050 0x562c6ad1e460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c6ac9ea70 0x562c6ac9eef0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c6aca5520 0x562c6acadbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 --2- 172.21.15.38:0/2172315953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c6ac9c020 0x562c6acad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 >> 172.21.15.38:0/2172315953 conn(0x562c6ac96750 msgr2=0x562c6ac95850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe927fff640 0 client.16779 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 shutdown_connections 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 1 -- 172.21.15.38:0/2172315953 wait complete. 2024-04-22T00:05:05.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.307+0000 7fe951d84740 20 client.16779 trim_cache size 0 max 16384 2024-04-22T00:05:05.304 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.309+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 msgr2=0x7fe9300a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.309+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 0x7fe9300a53c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe938009830 tx=0x7fe938009510 comp rx=0 tx=0).stop 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.309+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 msgr2=0x7fe948114a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.309+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe94001fe10 tx=0x7fe94001d750 comp rx=0 tx=0).stop 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 shutdown_connections 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe9300a2f00 0x7fe9300a53c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe948105ad0 0x7fe948111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe9481010d0 0x7fe948110b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 --2- 172.21.15.38:0/1286233251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481007b0 0x7fe948114a80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 >> 172.21.15.38:0/1286233251 conn(0x7fe948077740 msgr2=0x7fe948103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 shutdown_connections 2024-04-22T00:05:05.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.311+0000 7fe94f0c2640 1 -- 172.21.15.38:0/1286233251 wait complete. 2024-04-22T00:05:05.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:05:05.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.521+0000 7f41ad90f640 1 -- 172.21.15.38:0/3129176624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 msgr2=0x7f41a8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.521+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3129176624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a8106a80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4190009860 tx=0x7f419002bb70 comp rx=0 tx=0).stop 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.522+0000 7f41ad90f640 1 -- 172.21.15.38:0/3129176624 shutdown_connections 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.522+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3129176624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.522+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3129176624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 0x7f41a8100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.522+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3129176624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a8106a80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.522+0000 7f41ad90f640 1 -- 172.21.15.38:0/3129176624 >> 172.21.15.38:0/3129176624 conn(0x7f41a80761c0 msgr2=0x7f41a80765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.523+0000 7f41ad90f640 1 -- 172.21.15.38:0/3129176624 shutdown_connections 2024-04-22T00:05:05.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.523+0000 7f41ad90f640 1 -- 172.21.15.38:0/3129176624 wait complete. 2024-04-22T00:05:05.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.524+0000 7f41ad90f640 1 Processor -- start 2024-04-22T00:05:05.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 -- start start 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 0x7f41a8195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41a810f8e0 con 0x7f41a8100b20 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41a810f760 con 0x7f41a81066a0 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41ad90f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41a810fa60 con 0x7f41a81014e0 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41a67fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41a67fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39074/0 (socket says 172.21.15.38:39074) 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.525+0000 7f41a67fc640 1 -- 172.21.15.38:0/3969852258 learned_addr learned my addr 172.21.15.38:0/3969852258 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 0x7f41a8195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a77fe640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a67fc640 1 -- 172.21.15.38:0/3969852258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 msgr2=0x7f41a819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a67fc640 1 -- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 msgr2=0x7f41a8195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 0x7f41a8195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.526+0000 7f41a67fc640 1 -- 172.21.15.38:0/3969852258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a819ae10 con 0x7f41a81014e0 2024-04-22T00:05:05.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f419801fa60 tx=0x7f419801ff20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41ac90d640 1 -- 172.21.15.38:0/3969852258 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419801dc50 con 0x7f41a81014e0 2024-04-22T00:05:05.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41ac90d640 1 -- 172.21.15.38:0/3969852258 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f419801ddf0 con 0x7f41a81014e0 2024-04-22T00:05:05.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41ac90d640 1 -- 172.21.15.38:0/3969852258 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980216a0 con 0x7f41a81014e0 2024-04-22T00:05:05.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a819b100 con 0x7f41a81014e0 2024-04-22T00:05:05.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.527+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a81a2990 con 0x7f41a81014e0 2024-04-22T00:05:05.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.531+0000 7f41b05cc740 1 Processor -- start 2024-04-22T00:05:05.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.531+0000 7f41b05cc740 1 -- start start 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.531+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e417400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.531+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.531+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e422910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41b05cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e37b530 con 0x55bb7e420520 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41b05cc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e323830 con 0x55bb7e417020 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41b05cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e3774f0 con 0x55bb7e417940 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41ac90d640 1 -- 172.21.15.38:0/3969852258 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f41980046c0 con 0x7f41a81014e0 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41a77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e417400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41a6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39090/0 (socket says 172.21.15.38:39090) 2024-04-22T00:05:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1601030726 learned_addr learned my addr 172.21.15.38:0/1601030726 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.532+0000 7f41a67fc640 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e422910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1601030726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 msgr2=0x55bb7e417400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e417400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1601030726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 msgr2=0x55bb7e422910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e422910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1601030726 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb7e3472c0 con 0x55bb7e417940 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.533+0000 7f41ac90d640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 0x7f41940a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.534+0000 7f41ac90d640 1 -- 172.21.15.38:0/3969852258 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41980e0b70 con 0x7f41a81014e0 2024-04-22T00:05:05.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.534+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f419002c180 tx=0x7f4190009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196765bb40f2b669 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.536+0000 7f41a77fe640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 0x7f41940a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.536+0000 7f419dffb640 1 -- 172.21.15.38:0/1601030726 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190043020 con 0x55bb7e417940 2024-04-22T00:05:05.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.536+0000 7f419dffb640 1 -- 172.21.15.38:0/1601030726 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4190034ce0 con 0x55bb7e417940 2024-04-22T00:05:05.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.536+0000 7f419dffb640 1 -- 172.21.15.38:0/1601030726 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190039040 con 0x55bb7e417940 2024-04-22T00:05:05.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.537+0000 7f41b05cc740 1 -- 172.21.15.38:0/1601030726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 msgr2=0x55bb7e41ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.537+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f419002c180 tx=0x7f4190009510 comp rx=0 tx=0).stop 2024-04-22T00:05:05.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 -- 172.21.15.38:0/1601030726 shutdown_connections 2024-04-22T00:05:05.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e422910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e41ffe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1601030726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e417400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 -- 172.21.15.38:0/1601030726 >> 172.21.15.38:0/1601030726 conn(0x55bb7e411750 msgr2=0x55bb7e411b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41a77fe640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 0x7f41940a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f41a0005fd0 tx=0x7f41a0009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 -- 172.21.15.38:0/1601030726 shutdown_connections 2024-04-22T00:05:05.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 -- 172.21.15.38:0/1601030726 wait complete. 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.539+0000 7f41b05cc740 1 -- start start 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 Processor -- start 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417020 0x55bb7e41ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e41d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e4232b0 con 0x55bb7e417940 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e336f10 con 0x55bb7e420520 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bb7e423430 con 0x55bb7e417020 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.540+0000 7f41b05cc740 10 client.0 _ll_register_callbacks cb 0x55bb7e08ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43756/0 (socket says 172.21.15.38:43756) 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 -- 172.21.15.38:0/2639897631 learned_addr learned my addr 172.21.15.38:0/2639897631 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 -- 172.21.15.38:0/2639897631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417020 msgr2=0x55bb7e41ff70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417020 0x55bb7e41ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 -- 172.21.15.38:0/2639897631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 msgr2=0x55bb7e41d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e41d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 -- 172.21.15.38:0/2639897631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb7e308c80 con 0x55bb7e420520 2024-04-22T00:05:05.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.541+0000 7f41a77fe640 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41a0012880 tx=0x7f41a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f69a7db0571f4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.543+0000 7f4186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:05.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.543+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0010bf0 con 0x55bb7e420520 2024-04-22T00:05:05.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.543+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41a0010d90 con 0x55bb7e420520 2024-04-22T00:05:05.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.543+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4190034ce0 con 0x55bb7e420520 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.544+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41a001b460 con 0x55bb7e420520 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.546+0000 7f41b05cc740 10 client.26180 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.546+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb7e08f3f0 con 0x55bb7e420520 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.546+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4190034ce0 con 0x55bb7e420520 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.547+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f41a0024070 con 0x55bb7e420520 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 10 client.26180 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 10 client.26180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 20 client.26180 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e441700 0x55bb7e443ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 4 client.26180 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb7e388a70 con 0x55bb7e441700 2024-04-22T00:05:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.548+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e441700 0x55bb7e443ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.550+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e441700 0x55bb7e443ba0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d958cc5f4466002 server_cookie=fd9953f3c44bf6cc in_seq=0 out_seq=0 2024-04-22T00:05:05.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.550+0000 7f4186ffd640 10 client.26180 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.553+0000 7f4186ffd640 1 -- 172.21.15.38:0/2639897631 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb7e388a70 con 0x55bb7e441700 2024-04-22T00:05:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.553+0000 7f4186ffd640 10 client.26180 handle_command_reply: tid=0 2024-04-22T00:05:05.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.554+0000 7f41b05cc740 1 client.26180 shutdown 2024-04-22T00:05:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.554+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 msgr2=0x55bb7e41da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.554+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41a0012880 tx=0x7f41a0004a40 comp rx=0 tx=0).stop 2024-04-22T00:05:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 shutdown_connections 2024-04-22T00:05:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e420520 0x55bb7e41da40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e441700 0x55bb7e443ba0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e41d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 --2- 172.21.15.38:0/2639897631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417020 0x55bb7e41ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 >> 172.21.15.38:0/2639897631 conn(0x55bb7e411750 msgr2=0x55bb7e4134a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f4186ffd640 0 client.26180 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 shutdown_connections 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 1 -- 172.21.15.38:0/2639897631 wait complete. 2024-04-22T00:05:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.555+0000 7f41b05cc740 20 client.26180 trim_cache size 0 max 16384 2024-04-22T00:05:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.584+0000 7f41b05cc740 1 Processor -- start 2024-04-22T00:05:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.585+0000 7f41b05cc740 1 -- start start 2024-04-22T00:05:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.586+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.586+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e449470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.586+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e4499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.586+0000 7f41b05cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41a81a4230 con 0x55bb7e420520 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43770/0 (socket says 172.21.15.38:43770) 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1646784899 learned_addr learned my addr 172.21.15.38:0/1646784899 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a67fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e449470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41a81a5ef0 con 0x55bb7e417020 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41a81a4900 con 0x55bb7e417940 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1646784899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 msgr2=0x55bb7e449470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e449470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1646784899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 msgr2=0x55bb7e4499b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e4499b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.587+0000 7f41a6ffd640 1 -- 172.21.15.38:0/1646784899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb7e422c80 con 0x55bb7e417020 2024-04-22T00:05:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.589+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4190006010 tx=0x7f419002bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c72e037a1f2ee07 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.589+0000 7f419d7fa640 1 -- 172.21.15.38:0/1646784899 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190043050 con 0x55bb7e417020 2024-04-22T00:05:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.589+0000 7f419d7fa640 1 -- 172.21.15.38:0/1646784899 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41900348f0 con 0x55bb7e417020 2024-04-22T00:05:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.589+0000 7f419d7fa640 1 -- 172.21.15.38:0/1646784899 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190039040 con 0x55bb7e417020 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 msgr2=0x55bb7e42c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4190006010 tx=0x7f419002bb70 comp rx=0 tx=0).stop 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 shutdown_connections 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e420520 0x55bb7e4499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e417940 0x55bb7e449470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 --2- 172.21.15.38:0/1646784899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e42c170 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 >> 172.21.15.38:0/1646784899 conn(0x55bb7e411750 msgr2=0x55bb7e40ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 shutdown_connections 2024-04-22T00:05:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.591+0000 7f41b05cc740 1 -- 172.21.15.38:0/1646784899 wait complete. 2024-04-22T00:05:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.592+0000 7f41b05cc740 1 -- start start 2024-04-22T00:05:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.592+0000 7f41b05cc740 1 Processor -- start 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e428a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e420520 0x55bb7e41a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41a81a5760 con 0x55bb7e417940 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f41a81a5090 con 0x55bb7e417020 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41a8072fb0 con 0x55bb7e420520 2024-04-22T00:05:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41b05cc740 10 client.0 _ll_register_callbacks cb 0x7f41a0003f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41a67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41a67fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37122/0 (socket says 172.21.15.38:37122) 2024-04-22T00:05:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.594+0000 7f41a67fc640 1 -- 172.21.15.38:0/3851655350 learned_addr learned my addr 172.21.15.38:0/3851655350 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.595+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e428a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 -- 172.21.15.38:0/3851655350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e420520 msgr2=0x55bb7e41a190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e420520 0x55bb7e41a190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 -- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 msgr2=0x55bb7e428a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e428a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 -- 172.21.15.38:0/3851655350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb7e1597d0 con 0x55bb7e417940 2024-04-22T00:05:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f41a67fc640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f41980a9150 tx=0x7f41980ac650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2672b34dd204f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.596+0000 7f4186ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.597+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980accf0 con 0x55bb7e417940 2024-04-22T00:05:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.597+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb7e382dc0 con 0x55bb7e417940 2024-04-22T00:05:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.597+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f41980de1a0 con 0x55bb7e417940 2024-04-22T00:05:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.598+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41980de3a0 con 0x55bb7e417940 2024-04-22T00:05:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.600+0000 7f41b05cc740 10 client.16791 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.600+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb7e4a7a70 con 0x55bb7e417940 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.602+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bb7e382dc0 con 0x55bb7e417940 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f41980ac140 con 0x55bb7e417940 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 10 client.16791 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 20 client.16791 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e4a8380 0x55bb7e4aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 4 client.16791 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.603+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb7e3006e0 con 0x55bb7e4a8380 2024-04-22T00:05:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.604+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e4a8380 0x55bb7e4aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:05.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f4186ffd640 10 client.16791 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:05.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41a6ffd640 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e4a8380 0x55bb7e4aa7a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d807a0605951a1e server_cookie=925d371f82296f48 in_seq=0 out_seq=0 2024-04-22T00:05:05.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f4186ffd640 1 -- 172.21.15.38:0/3851655350 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb7e3006e0 con 0x55bb7e4a8380 2024-04-22T00:05:05.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f4186ffd640 10 client.16791 handle_command_reply: tid=0 2024-04-22T00:05:05.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 client.16791 shutdown 2024-04-22T00:05:05.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 msgr2=0x55bb7e419c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f41980a9150 tx=0x7f41980ac650 comp rx=0 tx=0).stop 2024-04-22T00:05:05.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 shutdown_connections 2024-04-22T00:05:05.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bb7e420520 0x55bb7e41a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55bb7e4a8380 0x55bb7e4aa7a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb7e417940 0x55bb7e419c50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 --2- 172.21.15.38:0/3851655350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bb7e417020 0x55bb7e428a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f4186ffd640 0 client.16791 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.607+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 >> 172.21.15.38:0/3851655350 conn(0x55bb7e411750 msgr2=0x55bb7e410850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.608+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 shutdown_connections 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.608+0000 7f41b05cc740 1 -- 172.21.15.38:0/3851655350 wait complete. 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.608+0000 7f41b05cc740 20 client.16791 trim_cache size 0 max 16384 2024-04-22T00:05:05.602 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:05.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.610+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 msgr2=0x7f41940a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.610+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 0x7f41940a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f41a0005fd0 tx=0x7f41a0009500 comp rx=0 tx=0).stop 2024-04-22T00:05:05.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.610+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 msgr2=0x7f41a8196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:05.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.610+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f419801fa60 tx=0x7f419801ff20 comp rx=0 tx=0).stop 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 shutdown_connections 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f41940a2d80 0x7f41940a5240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41a81066a0 0x7f41a819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41a81014e0 0x7f41a8196360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 --2- 172.21.15.38:0/3969852258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41a8100b20 0x7f41a8195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.611+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 >> 172.21.15.38:0/3969852258 conn(0x7f41a80761c0 msgr2=0x7f41a80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.612+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 shutdown_connections 2024-04-22T00:05:05.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:05.612+0000 7f41ad90f640 1 -- 172.21.15.38:0/3969852258 wait complete. 2024-04-22T00:05:06.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:06 smithi203 ceph-mon[31431]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 MiB/s wr, 7.78k op/s 2024-04-22T00:05:06.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:06 smithi111 ceph-mon[31543]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 MiB/s wr, 7.78k op/s 2024-04-22T00:05:06.623 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:05:06.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:05:06.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:05:06.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:06 smithi038 ceph-mon[27972]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 MiB/s wr, 7.78k op/s 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 -- 172.21.15.38:0/230645236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 msgr2=0x7f5cdc10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/230645236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10e5f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009ca0 tx=0x7f5cd402c100 comp rx=0 tx=0).stop 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 -- 172.21.15.38:0/230645236 shutdown_connections 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/230645236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10e5f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/230645236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc069b50 0x7f5cdc069f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/230645236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc107730 0x7f5cdc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 -- 172.21.15.38:0/230645236 >> 172.21.15.38:0/230645236 conn(0x7f5cdc0776e0 msgr2=0x7f5cdc077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 -- 172.21.15.38:0/230645236 shutdown_connections 2024-04-22T00:05:06.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.795+0000 7f5ce0d85640 1 -- 172.21.15.38:0/230645236 wait complete. 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.796+0000 7f5ce0d85640 1 Processor -- start 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.796+0000 7f5ce0d85640 1 -- start start 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc069b50 0x7f5cdc10ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc107730 0x7f5cdc10f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc113d60 con 0x7f5cdc107730 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc113be0 con 0x7f5cdc069b50 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5ce0d85640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc113ee0 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5cd9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5cdad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc107730 0x7f5cdc10f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5cd9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39098/0 (socket says 172.21.15.38:39098) 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5cd9d9b640 1 -- 172.21.15.38:0/1226840705 learned_addr learned my addr 172.21.15.38:0/1226840705 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.797+0000 7f5cda59c640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc069b50 0x7f5cdc10ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.798+0000 7f5cd9d9b640 1 -- 172.21.15.38:0/1226840705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc069b50 msgr2=0x7f5cdc10ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.798+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc069b50 0x7f5cdc10ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.798+0000 7f5cd9d9b640 1 -- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc107730 msgr2=0x7f5cdc10f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.798+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc107730 0x7f5cdc10f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.798+0000 7f5cd9d9b640 1 -- 172.21.15.38:0/1226840705 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc1aa8e0 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.799+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000b4d0 tx=0x7f5cd000b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.800+0000 7f5ccb7fe640 1 -- 172.21.15.38:0/1226840705 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0014070 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.800+0000 7f5ccb7fe640 1 -- 172.21.15.38:0/1226840705 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd0010d50 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.800+0000 7f5ccb7fe640 1 -- 172.21.15.38:0/1226840705 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0017360 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.800+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc1aabd0 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.800+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc1ab0d0 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.804+0000 7f5ccb7fe640 1 -- 172.21.15.38:0/1226840705 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5cd0017070 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 Processor -- start 2024-04-22T00:05:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 -- start start 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9187470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9187ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea91931a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561ea90eb170 con 0x561ea9190db0 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561ea90937c0 con 0x561ea9187a40 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.805+0000 7f5ce396b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ea90e7130 con 0x561ea9188720 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cd9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea91931a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9187ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cd9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea91931a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37146/0 (socket says 172.21.15.38:37146) 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cd9d9b640 1 -- 172.21.15.38:0/3463733796 learned_addr learned my addr 172.21.15.38:0/3463733796 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:06.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9187ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43804/0 (socket says 172.21.15.38:43804) 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5ccb7fe640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 0x7f5cc40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9187470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.806+0000 7f5ccb7fe640 1 -- 172.21.15.38:0/1226840705 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5cd00cec10 con 0x7f5cdc06a4d0 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3463733796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 msgr2=0x561ea9187ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9187ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3463733796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 msgr2=0x561ea91931a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea91931a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3463733796 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea90a4e60 con 0x561ea9188720 2024-04-22T00:05:06.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.807+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 0x7f5cc40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.808+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9187470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009bf0 tx=0x7f5cd4032470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbfc4a94f11a9c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.808+0000 7f5cc8ff9640 1 -- 172.21.15.38:0/3463733796 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4039040 con 0x561ea9188720 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.808+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 0x7f5cc40a53c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4006010 tx=0x7f5cd4045000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5cc8ff9640 1 -- 172.21.15.38:0/3463733796 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd4003c30 con 0x561ea9188720 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5cc8ff9640 1 -- 172.21.15.38:0/3463733796 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4034880 con 0x561ea9188720 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 -- 172.21.15.38:0/3463733796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 msgr2=0x561ea9187470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9187470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009bf0 tx=0x7f5cd4032470 comp rx=0 tx=0).stop 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 -- 172.21.15.38:0/3463733796 shutdown_connections 2024-04-22T00:05:06.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea91931a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9187ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3463733796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9187470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.809+0000 7f5ce396b740 1 -- 172.21.15.38:0/3463733796 >> 172.21.15.38:0/3463733796 conn(0x561ea914e4f0 msgr2=0x561ea9182490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.810+0000 7f5ce396b740 1 -- 172.21.15.38:0/3463733796 shutdown_connections 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.810+0000 7f5ce396b740 1 -- 172.21.15.38:0/3463733796 wait complete. 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 -- start start 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 Processor -- start 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9188720 0x561ea918e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9190db0 0x561ea918e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561ea9193c20 con 0x561ea9187a40 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561ea90a6d40 con 0x561ea9188720 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ea9193da0 con 0x561ea9190db0 2024-04-22T00:05:06.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5ce396b740 10 client.0 _ll_register_callbacks cb 0x561ea90e2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37158/0 (socket says 172.21.15.38:37158) 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.811+0000 7f5cda59c640 1 -- 172.21.15.38:0/3559646404 learned_addr learned my addr 172.21.15.38:0/3559646404 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9190db0 0x561ea918e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9188720 0x561ea918e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 -- 172.21.15.38:0/3559646404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9190db0 msgr2=0x561ea918e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9190db0 0x561ea918e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 -- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9188720 msgr2=0x561ea918e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9188720 0x561ea918e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 -- 172.21.15.38:0/3559646404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea8ec77e0 con 0x561ea9187a40 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc010820 tx=0x7f5ccc010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd30768fbcb16900 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00fb10 con 0x561ea9187a40 2024-04-22T00:05:06.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5ccc00fcb0 con 0x561ea9187a40 2024-04-22T00:05:06.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.812+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cd4003c30 con 0x561ea9187a40 2024-04-22T00:05:06.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.813+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5ccc024050 con 0x561ea9187a40 2024-04-22T00:05:06.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.813+0000 7f5ce396b740 10 client.16800 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:06.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.813+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561ea8f27f70 con 0x561ea9187a40 2024-04-22T00:05:06.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.814+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd4003c30 con 0x561ea9187a40 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f5ccc00e540 con 0x561ea9187a40 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 10 client.16800 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 10 client.16800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 20 client.16800 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea91a1050 0x561ea91a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 4 client.16800 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ea90bf450 con 0x561ea91a1050 2024-04-22T00:05:06.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.815+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea91a1050 0x561ea91a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.816+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea91a1050 0x561ea91a34b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e380f746c50bf9 server_cookie=2aac30eef2d12128 in_seq=0 out_seq=0 2024-04-22T00:05:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.816+0000 7f5cc1ffb640 10 client.16800 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.817+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3559646404 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ea90bf450 con 0x561ea91a1050 2024-04-22T00:05:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.817+0000 7f5cc1ffb640 10 client.16800 handle_command_reply: tid=0 2024-04-22T00:05:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.817+0000 7f5ce396b740 1 client.16800 shutdown 2024-04-22T00:05:06.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 msgr2=0x561ea918de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc010820 tx=0x7f5ccc010ce0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 shutdown_connections 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9190db0 0x561ea918e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9188720 0x561ea918e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea91a1050 0x561ea91a34b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3559646404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea918de00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 >> 172.21.15.38:0/3559646404 conn(0x561ea914e4f0 msgr2=0x561ea91841e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5cc1ffb640 0 client.16800 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 shutdown_connections 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 1 -- 172.21.15.38:0/3559646404 wait complete. 2024-04-22T00:05:06.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.818+0000 7f5ce396b740 20 client.16800 trim_cache size 0 max 16384 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.850+0000 7f5ce396b740 1 Processor -- start 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 -- start start 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea919cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b5050 con 0x561ea9190db0 2024-04-22T00:05:06.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b6d10 con 0x561ea9187a40 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5ce396b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b5720 con 0x561ea9188720 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43820/0 (socket says 172.21.15.38:43820) 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cda59c640 1 -- 172.21.15.38:0/3633914885 learned_addr learned my addr 172.21.15.38:0/3633914885 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37174/0 (socket says 172.21.15.38:37174) 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea919cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3633914885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 msgr2=0x561ea919cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea919cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3633914885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 msgr2=0x561ea9190530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.851+0000 7f5cdad9d640 1 -- 172.21.15.38:0/3633914885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea8dbf170 con 0x561ea9190db0 2024-04-22T00:05:06.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.852+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.852+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4032bc0 tx=0x7f5cd4032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a74587c2ef6fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.853+0000 7f5cc3fff640 1 -- 172.21.15.38:0/3633914885 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4059070 con 0x561ea9190db0 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.853+0000 7f5cc3fff640 1 -- 172.21.15.38:0/3633914885 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd4003c30 con 0x561ea9190db0 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.853+0000 7f5cc3fff640 1 -- 172.21.15.38:0/3633914885 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4039070 con 0x561ea9190db0 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.854+0000 7f5ce396b740 1 -- 172.21.15.38:0/3633914885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 msgr2=0x561ea9181370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.854+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4032bc0 tx=0x7f5cd4032590 comp rx=0 tx=0).stop 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.854+0000 7f5ce396b740 1 -- 172.21.15.38:0/3633914885 shutdown_connections 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.854+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9190db0 0x561ea9181370 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.854+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea919cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.855+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3633914885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9187a40 0x561ea9190530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.855+0000 7f5ce396b740 1 -- 172.21.15.38:0/3633914885 >> 172.21.15.38:0/3633914885 conn(0x561ea914e4f0 msgr2=0x561ea9180c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.856+0000 7f5ce396b740 1 -- 172.21.15.38:0/3633914885 shutdown_connections 2024-04-22T00:05:06.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.856+0000 7f5ce396b740 1 -- 172.21.15.38:0/3633914885 wait complete. 2024-04-22T00:05:06.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.856+0000 7f5ce396b740 1 -- start start 2024-04-22T00:05:06.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.857+0000 7f5ce396b740 1 Processor -- start 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9199570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9190db0 0x561ea918a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b6580 con 0x561ea9187a40 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b5eb0 con 0x561ea9190db0 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1b3030 con 0x561ea9188720 2024-04-22T00:05:06.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5ce396b740 10 client.0 _ll_register_callbacks cb 0x7f5ccc00a6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:06.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37186/0 (socket says 172.21.15.38:37186) 2024-04-22T00:05:06.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 -- 172.21.15.38:0/3194018646 learned_addr learned my addr 172.21.15.38:0/3194018646 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:06.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cdad9d640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9190db0 0x561ea918a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 -- 172.21.15.38:0/3194018646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 msgr2=0x561ea9199570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:06.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9199570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 -- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9190db0 msgr2=0x561ea918a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:06.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9190db0 0x561ea918a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.858+0000 7f5cda59c640 1 -- 172.21.15.38:0/3194018646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea918a750 con 0x561ea9187a40 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.859+0000 7f5cda59c640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00eb50 tx=0x7f5ccc0104d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d079df1ca20824 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.859+0000 7f5cc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.859+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00fb10 con 0x561ea9187a40 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.859+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cd4003c30 con 0x561ea9187a40 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.859+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5ccc00fcb0 con 0x561ea9187a40 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.860+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5ccc03a050 con 0x561ea9187a40 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.860+0000 7f5ce396b740 10 client.16806 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.860+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561ea918ba50 con 0x561ea9187a40 2024-04-22T00:05:06.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc02c070 con 0x561ea9187a40 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f5ccc031bb0 con 0x561ea9187a40 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 10 client.16806 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 10 client.16806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 20 client.16806 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea92087b0 0x561ea920abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 4 client.16806 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:05:06.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.861+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f5cd4003c30 con 0x561ea92087b0 2024-04-22T00:05:06.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.863+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea92087b0 0x561ea920abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:06.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.863+0000 7f5cd9d9b640 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea92087b0 0x561ea920abd0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7aff0cd21185275 server_cookie=b8a51bec5d06e274 in_seq=0 out_seq=0 2024-04-22T00:05:06.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.863+0000 7f5cc1ffb640 10 client.16806 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.866+0000 7f5cc1ffb640 1 -- 172.21.15.38:0/3194018646 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f5cd4003c30 con 0x561ea92087b0 2024-04-22T00:05:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.866+0000 7f5cc1ffb640 10 client.16806 handle_command_reply: tid=0 2024-04-22T00:05:06.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.867+0000 7f5ce396b740 1 client.16806 shutdown 2024-04-22T00:05:06.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.873+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 msgr2=0x561ea9199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.873+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00eb50 tx=0x7f5ccc0104d0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.874+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 shutdown_connections 2024-04-22T00:05:06.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.874+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ea9190db0 0x561ea918a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.874+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ea9188720 0x561ea9199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.874+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561ea92087b0 0x561ea920abd0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5ce396b740 1 --2- 172.21.15.38:0/3194018646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea9187a40 0x561ea9199030 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 >> 172.21.15.38:0/3194018646 conn(0x561ea914e4f0 msgr2=0x561ea91814f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5cc1ffb640 0 client.16806 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 shutdown_connections 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5ce396b740 1 -- 172.21.15.38:0/3194018646 wait complete. 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.875+0000 7f5ce396b740 20 client.16806 trim_cache size 0 max 16384 2024-04-22T00:05:06.870 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:06.871 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:05:06.871 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "706d9366-363b-4653-9d23-cb01cbc290e3", 2024-04-22T00:05:06.871 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:05:06.871 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:06.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.876+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 msgr2=0x7f5cc40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.877+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 0x7f5cc40a53c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4006010 tx=0x7f5cd4045000 comp rx=0 tx=0).stop 2024-04-22T00:05:06.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.877+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 msgr2=0x7f5cdc10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.877+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000b4d0 tx=0x7f5cd000b990 comp rx=0 tx=0).stop 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 shutdown_connections 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5cc40a2f00 0x7f5cc40a53c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5cdc107730 0x7f5cdc10f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5cdc06a4d0 0x7f5cdc10f2c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 --2- 172.21.15.38:0/1226840705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cdc069b50 0x7f5cdc10ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.878+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 >> 172.21.15.38:0/1226840705 conn(0x7f5cdc0776e0 msgr2=0x7f5cdc0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:06.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.879+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 shutdown_connections 2024-04-22T00:05:06.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:06.879+0000 7f5ce0d85640 1 -- 172.21.15.38:0/1226840705 wait complete. 2024-04-22T00:05:06.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:05:06.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:05:07.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.055+0000 7f14d251e640 1 -- 172.21.15.38:0/3196497834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc100610 msgr2=0x7f14cc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.055+0000 7f14d251e640 1 --2- 172.21.15.38:0/3196497834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc100610 0x7f14cc100a90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009900 tx=0x7f14b402bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 -- 172.21.15.38:0/3196497834 shutdown_connections 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 --2- 172.21.15.38:0/3196497834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100fd0 0x7f14cc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 --2- 172.21.15.38:0/3196497834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc100610 0x7f14cc100a90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 --2- 172.21.15.38:0/3196497834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc1066b0 0x7f14cc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 -- 172.21.15.38:0/3196497834 >> 172.21.15.38:0/3196497834 conn(0x7f14cc077740 msgr2=0x7f14cc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 -- 172.21.15.38:0/3196497834 shutdown_connections 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 -- 172.21.15.38:0/3196497834 wait complete. 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 Processor -- start 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.056+0000 7f14d251e640 1 -- start start 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 0x7f14cc19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1109f0 con 0x7f14cc100610 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc110870 con 0x7f14cc1066b0 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d251e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc110b70 con 0x7f14cc100fd0 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37206/0 (socket says 172.21.15.38:37206) 2024-04-22T00:05:07.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39156/0 (socket says 172.21.15.38:39156) 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 -- 172.21.15.38:0/3899531746 learned_addr learned my addr 172.21.15.38:0/3899531746 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 0x7f14cc19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 -- 172.21.15.38:0/3899531746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 msgr2=0x7f14cc19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 -- 172.21.15.38:0/3899531746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 msgr2=0x7f14cc19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 0x7f14cc19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 -- 172.21.15.38:0/3899531746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc19f190 con 0x7f14cc100610 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 0x7f14cc19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.057+0000 7f14cbfff640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e9c0 tx=0x7f14bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.058+0000 7f14c97fa640 1 -- 172.21.15.38:0/3899531746 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc00cd80 con 0x7f14cc100610 2024-04-22T00:05:07.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.058+0000 7f14c97fa640 1 -- 172.21.15.38:0/3899531746 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14bc0035f0 con 0x7f14cc100610 2024-04-22T00:05:07.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.058+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc19f480 con 0x7f14cc100610 2024-04-22T00:05:07.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.058+0000 7f14c97fa640 1 -- 172.21.15.38:0/3899531746 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc010690 con 0x7f14cc100610 2024-04-22T00:05:07.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.059+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a6d10 con 0x7f14cc100610 2024-04-22T00:05:07.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.063+0000 7f14d51e1740 1 Processor -- start 2024-04-22T00:05:07.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.064+0000 7f14c97fa640 1 -- 172.21.15.38:0/3899531746 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f14bc0037a0 con 0x7f14cc100610 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.064+0000 7f14d51e1740 1 -- start start 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c85649310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 0x555c85651ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85654820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c855ad560 con 0x555c85649850 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c85555730 con 0x555c85648f30 2024-04-22T00:05:07.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d51e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c855a9520 con 0x555c85652430 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d0abb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c85649310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 0x555c85651ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d0abb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c85649310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43850/0 (socket says 172.21.15.38:43850) 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.065+0000 7f14d0abb640 1 -- 172.21.15.38:0/100312402 learned_addr learned my addr 172.21.15.38:0/100312402 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.066+0000 7f14cbfff640 1 -- 172.21.15.38:0/100312402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 msgr2=0x555c85654820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.066+0000 7f14cbfff640 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85654820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.066+0000 7f14cbfff640 1 -- 172.21.15.38:0/100312402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 msgr2=0x555c85649310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.066+0000 7f14cbfff640 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c85649310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.066+0000 7f14cbfff640 1 -- 172.21.15.38:0/100312402 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c8538b7d0 con 0x555c85649850 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.068+0000 7f14cbfff640 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 0x555c85651ef0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0cc830 tx=0x7f14bc0d1900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e727f756453600 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.068+0000 7f14c97fa640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 0x7f14b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.068+0000 7f14c97fa640 1 -- 172.21.15.38:0/3899531746 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f14bc0d3060 con 0x7f14cc100610 2024-04-22T00:05:07.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.069+0000 7f14c27fc640 1 -- 172.21.15.38:0/100312402 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0978a0 con 0x555c85649850 2024-04-22T00:05:07.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.069+0000 7f14c27fc640 1 -- 172.21.15.38:0/100312402 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14bc0d2020 con 0x555c85649850 2024-04-22T00:05:07.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.070+0000 7f14c27fc640 1 -- 172.21.15.38:0/100312402 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc09af60 con 0x555c85649850 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.070+0000 7f14d51e1740 1 -- 172.21.15.38:0/100312402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 msgr2=0x555c85651ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.070+0000 7f14d51e1740 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 0x555c85651ef0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0cc830 tx=0x7f14bc0d1900 comp rx=0 tx=0).stop 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.070+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 0x7f14b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 -- 172.21.15.38:0/100312402 shutdown_connections 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85654820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85649850 0x555c85651ef0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 --2- 172.21.15.38:0/100312402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c85649310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 -- 172.21.15.38:0/100312402 >> 172.21.15.38:0/100312402 conn(0x555c85643660 msgr2=0x555c85643a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 -- 172.21.15.38:0/100312402 shutdown_connections 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 0x7f14b80a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009fd0 tx=0x7f14c4009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.071+0000 7f14d51e1740 1 -- 172.21.15.38:0/100312402 wait complete. 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.072+0000 7f14d51e1740 1 -- start start 2024-04-22T00:05:07.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.072+0000 7f14d51e1740 1 Processor -- start 2024-04-22T00:05:07.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.072+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 0x555c8564f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.072+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8564f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.072+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 0x555c8564f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 0x555c8564f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39172/0 (socket says 172.21.15.38:39172) 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14cbfff640 1 -- 172.21.15.38:0/2266327733 learned_addr learned my addr 172.21.15.38:0/2266327733 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c856550b0 con 0x555c85652430 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c85568e00 con 0x555c85649850 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c85655230 con 0x555c85648f30 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14d51e1740 10 client.0 _ll_register_callbacks cb 0x555c855a3eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14d0abb640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.073+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8564f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14cb7fe640 1 -- 172.21.15.38:0/2266327733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 msgr2=0x555c8564f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 0x555c8564f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14cb7fe640 1 -- 172.21.15.38:0/2266327733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 msgr2=0x555c8564fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14cb7fe640 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c852c13f0 con 0x555c85649850 2024-04-22T00:05:07.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.074+0000 7f14d0abb640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:07.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8564f930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b402c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d803e6c670559d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:07.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4042070 con 0x555c85649850 2024-04-22T00:05:07.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f14b4004450 con 0x555c85649850 2024-04-22T00:05:07.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14bc0d2020 con 0x555c85649850 2024-04-22T00:05:07.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.075+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f14b4050050 con 0x555c85649850 2024-04-22T00:05:07.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.077+0000 7f14d51e1740 10 client.26213 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:07.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.077+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555c852810f0 con 0x555c85649850 2024-04-22T00:05:07.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.077+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4004f20 con 0x555c85649850 2024-04-22T00:05:07.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.077+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f14b4003790 con 0x555c85649850 2024-04-22T00:05:07.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 10 client.26213 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:07.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:07.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 20 client.26213 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:07.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c85662700 0x555c85664b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 4 client.26213 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:07.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.079+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f14bc0d2020 con 0x555c85662700 2024-04-22T00:05:07.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.080+0000 7f14cbfff640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c85662700 0x555c85664b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.081+0000 7f14cbfff640 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c85662700 0x555c85664b20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d7a72bf4e43cfd server_cookie=a8390a60d0cf1b23 in_seq=0 out_seq=0 2024-04-22T00:05:07.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.081+0000 7f14ab7fe640 10 client.26213 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:07.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.083+0000 7f14ab7fe640 1 -- 172.21.15.38:0/2266327733 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f14bc0d2020 con 0x555c85662700 2024-04-22T00:05:07.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.083+0000 7f14ab7fe640 10 client.26213 handle_command_reply: tid=0 2024-04-22T00:05:07.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 client.26213 shutdown 2024-04-22T00:05:07.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 msgr2=0x555c8564f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8564f930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b402c800 comp rx=0 tx=0).stop 2024-04-22T00:05:07.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 shutdown_connections 2024-04-22T00:05:07.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c85662700 0x555c85664b20 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14ab7fe640 0 client.26213 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8564f930 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 --2- 172.21.15.38:0/2266327733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85648f30 0x555c8564f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 >> 172.21.15.38:0/2266327733 conn(0x555c85643660 msgr2=0x555c856453b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 shutdown_connections 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 1 -- 172.21.15.38:0/2266327733 wait complete. 2024-04-22T00:05:07.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.084+0000 7f14d51e1740 20 client.26213 trim_cache size 0 max 16384 2024-04-22T00:05:07.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 Processor -- start 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 -- start start 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85648f30 0x555c85651b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8565dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85643a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1a84c0 con 0x555c85648f30 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1aa180 con 0x555c85649850 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.119+0000 7f14d51e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1a8b90 con 0x555c85652430 2024-04-22T00:05:07.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.120+0000 7f14cb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8565dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14cb7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8565dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43858/0 (socket says 172.21.15.38:43858) 2024-04-22T00:05:07.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14cb7fe640 1 -- 172.21.15.38:0/3920747595 learned_addr learned my addr 172.21.15.38:0/3920747595 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:07.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85643a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14cbfff640 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85648f30 0x555c85651b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14d0abb640 1 -- 172.21.15.38:0/3920747595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 msgr2=0x555c8565dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.121+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8565dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.122+0000 7f14d0abb640 1 -- 172.21.15.38:0/3920747595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85648f30 msgr2=0x555c85651b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.122+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85648f30 0x555c85651b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.122+0000 7f14d0abb640 1 -- 172.21.15.38:0/3920747595 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c85643490 con 0x555c85652430 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.122+0000 7f14d0abb640 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85643a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14c40048a0 tx=0x7f14c4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deabd2194a313cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.122+0000 7f14c1ffb640 1 -- 172.21.15.38:0/3920747595 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c402c070 con 0x555c85652430 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.123+0000 7f14c1ffb640 1 -- 172.21.15.38:0/3920747595 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c401ca70 con 0x555c85652430 2024-04-22T00:05:07.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.123+0000 7f14c1ffb640 1 -- 172.21.15.38:0/3920747595 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4027c00 con 0x555c85652430 2024-04-22T00:05:07.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.124+0000 7f14d51e1740 1 -- 172.21.15.38:0/3920747595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 msgr2=0x555c85643a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.124+0000 7f14d51e1740 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85643a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14c40048a0 tx=0x7f14c4016000 comp rx=0 tx=0).stop 2024-04-22T00:05:07.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.126+0000 7f14d51e1740 1 -- 172.21.15.38:0/3920747595 shutdown_connections 2024-04-22T00:05:07.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.126+0000 7f14d51e1740 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85652430 0x555c85643a40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.126+0000 7f14d51e1740 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85649850 0x555c8565dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.126+0000 7f14d51e1740 1 --2- 172.21.15.38:0/3920747595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85648f30 0x555c85651b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.126+0000 7f14d51e1740 1 -- 172.21.15.38:0/3920747595 >> 172.21.15.38:0/3920747595 conn(0x555c85643660 msgr2=0x555c856426e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.127+0000 7f14d51e1740 1 -- 172.21.15.38:0/3920747595 shutdown_connections 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.127+0000 7f14d51e1740 1 -- 172.21.15.38:0/3920747595 wait complete. 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.127+0000 7f14d51e1740 1 -- start start 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 Processor -- start 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c8565a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 0x555c8564bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1a99f0 con 0x555c85652430 2024-04-22T00:05:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1a9320 con 0x555c85648f30 2024-04-22T00:05:07.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc19b4f0 con 0x555c85649850 2024-04-22T00:05:07.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.128+0000 7f14d51e1740 10 client.0 _ll_register_callbacks cb 0x7f14bc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:07.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37240/0 (socket says 172.21.15.38:37240) 2024-04-22T00:05:07.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 -- 172.21.15.38:0/1595484495 learned_addr learned my addr 172.21.15.38:0/1595484495 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:07.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14cbfff640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c8565a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 0x555c8564bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 -- 172.21.15.38:0/1595484495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 msgr2=0x555c8564bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 0x555c8564bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 -- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 msgr2=0x555c8565a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c8565a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.129+0000 7f14d0abb640 1 -- 172.21.15.38:0/1595484495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c85654820 con 0x555c85652430 2024-04-22T00:05:07.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.130+0000 7f14d0abb640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f14c4011de0 tx=0x7f14c40273f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37005479115311b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.130+0000 7f14ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.131+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c402c040 con 0x555c85652430 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.131+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f14c401c790 con 0x555c85652430 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.131+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14c401ca70 con 0x555c85652430 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.131+0000 7f14cbfff640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c8565a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:07.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.131+0000 7f14cb7fe640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 0x555c8564bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:07.128 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.132+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f14c4004a40 con 0x555c85652430 2024-04-22T00:05:07.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.132+0000 7f14d51e1740 10 client.16824 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:07.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.132+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555c856c9670 con 0x555c85652430 2024-04-22T00:05:07.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.134+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c401ca70 con 0x555c85652430 2024-04-22T00:05:07.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.134+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f14c4038030 con 0x555c85652430 2024-04-22T00:05:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 10 client.16824 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 10 client.16824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 20 client.16824 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:07.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c856c9d20 0x555c856cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 4 client.16824 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555c855baaa0 con 0x555c856c9d20 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.135+0000 7f14cbfff640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c856c9d20 0x555c856cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.136+0000 7f14cbfff640 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c856c9d20 0x555c856cc140 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fafd5095ad79e0f3 server_cookie=484901b140f960e8 in_seq=0 out_seq=0 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.136+0000 7f14ab7fe640 10 client.16824 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.136+0000 7f14ab7fe640 1 -- 172.21.15.38:0/1595484495 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x555c855baaa0 con 0x555c856c9d20 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.136+0000 7f14ab7fe640 10 client.16824 handle_command_reply: tid=0 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.136+0000 7f14d51e1740 1 client.16824 shutdown 2024-04-22T00:05:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 msgr2=0x555c8564c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f14c4011de0 tx=0x7f14c40273f0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 shutdown_connections 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555c856c9d20 0x555c856cc140 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c85652430 0x555c8564c260 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c85649850 0x555c8564bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 --2- 172.21.15.38:0/1595484495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c85648f30 0x555c8565a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 >> 172.21.15.38:0/1595484495 conn(0x555c85643660 msgr2=0x555c85644a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.137+0000 7f14ab7fe640 0 client.16824 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:07.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.140+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 shutdown_connections 2024-04-22T00:05:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.141+0000 7f14d51e1740 1 -- 172.21.15.38:0/1595484495 wait complete. 2024-04-22T00:05:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.141+0000 7f14d51e1740 20 client.16824 trim_cache size 0 max 16384 2024-04-22T00:05:07.136 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (216 inodes in the stack)", 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "706d9366-363b-4653-9d23-cb01cbc290e3": { 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "706d9366-363b-4653-9d23-cb01cbc290e3", 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:05:07.137 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:07.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.144+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 msgr2=0x7f14b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.144+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 0x7f14b80a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009fd0 tx=0x7f14c4009500 comp rx=0 tx=0).stop 2024-04-22T00:05:07.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.144+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 msgr2=0x7f14cc19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:07.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.144+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e9c0 tx=0x7f14bc00ee80 comp rx=0 tx=0).stop 2024-04-22T00:05:07.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 shutdown_connections 2024-04-22T00:05:07.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f14b80a2d00 0x7f14b80a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14cc1066b0 0x7f14cc19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc100fd0 0x7f14cc19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 --2- 172.21.15.38:0/3899531746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14cc100610 0x7f14cc19a1a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 >> 172.21.15.38:0/3899531746 conn(0x7f14cc077740 msgr2=0x7f14cc0688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 shutdown_connections 2024-04-22T00:05:07.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:07.145+0000 7f14d251e640 1 -- 172.21.15.38:0/3899531746 wait complete. 2024-04-22T00:05:07.157 INFO:tasks.cephfs.filesystem:scrub status for tag:706d9366-363b-4653-9d23-cb01cbc290e3 - {'path': '/', 'tag': '706d9366-363b-4653-9d23-cb01cbc290e3', 'options': 'recursive,force'} 2024-04-22T00:05:08.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:08 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:05:08.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:08 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:08.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:08 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:08.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:08 smithi203 ceph-mon[31431]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 77 MiB/s wr, 11.99k op/s 2024-04-22T00:05:08.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:08 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:05:08.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:08 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:08.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:08 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:08.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:08 smithi111 ceph-mon[31543]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 77 MiB/s wr, 11.99k op/s 2024-04-22T00:05:08.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:08 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:05:08.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:08 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:08.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:08 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:08.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:08 smithi038 ceph-mon[27972]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 77 MiB/s wr, 11.99k op/s 2024-04-22T00:05:10.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:10 smithi203 ceph-mon[31431]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 MiB/s wr, 8.98k op/s 2024-04-22T00:05:10.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:10 smithi111 ceph-mon[31543]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 MiB/s wr, 8.98k op/s 2024-04-22T00:05:10.577 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:05:10 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:05:10.157+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v198>,<0x100000001fb/client.0 v189>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v193>,<0x10000000001/sv_1 v215>,<0x10000000000/qa v194>,<0x1/volumes v186>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:05:10.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:10 smithi038 ceph-mon[27972]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 MiB/s wr, 8.98k op/s 2024-04-22T00:05:11.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.293+0000 7f4e73888640 1 -- 172.21.15.38:0/1295924746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10e610 msgr2=0x7f4e6c10c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:11.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.293+0000 7f4e73888640 1 --2- 172.21.15.38:0/1295924746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10e610 0x7f4e6c10c970 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0074e0 tx=0x7f4e5c02c8f0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 -- 172.21.15.38:0/1295924746 shutdown_connections 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 --2- 172.21.15.38:0/1295924746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e6c10d0a0 0x7f4e6c079a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 --2- 172.21.15.38:0/1295924746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10e610 0x7f4e6c10c970 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 --2- 172.21.15.38:0/1295924746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c10dc60 0x7f4e6c10e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 -- 172.21.15.38:0/1295924746 >> 172.21.15.38:0/1295924746 conn(0x7f4e6c06c810 msgr2=0x7f4e6c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 -- 172.21.15.38:0/1295924746 shutdown_connections 2024-04-22T00:05:11.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.294+0000 7f4e73888640 1 -- 172.21.15.38:0/1295924746 wait complete. 2024-04-22T00:05:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.295+0000 7f4e73888640 1 Processor -- start 2024-04-22T00:05:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.295+0000 7f4e73888640 1 -- start start 2024-04-22T00:05:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10d0a0 0x7f4e6c082a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e6c10dc60 0x7f4e6c082f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c07cfb0 con 0x7f4e6c10dc60 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c07ce30 con 0x7f4e6c10d0a0 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e73888640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c07d130 con 0x7f4e6c0834b0 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e71e25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e71e25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39196/0 (socket says 172.21.15.38:39196) 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.296+0000 7f4e71e25640 1 -- 172.21.15.38:0/2093862345 learned_addr learned my addr 172.21.15.38:0/2093862345 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.297+0000 7f4e71e25640 1 -- 172.21.15.38:0/2093862345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10d0a0 msgr2=0x7f4e6c082a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:05:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.297+0000 7f4e71e25640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10d0a0 0x7f4e6c082a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.297+0000 7f4e71e25640 1 -- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e6c10dc60 msgr2=0x7f4e6c082f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:05:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.297+0000 7f4e71e25640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e6c10dc60 0x7f4e6c082f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.297+0000 7f4e71e25640 1 -- 172.21.15.38:0/2093862345 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c1b5880 con 0x7f4e6c0834b0 2024-04-22T00:05:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.298+0000 7f4e71e25640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800b3f0 tx=0x7f4e6800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.299+0000 7f4e627fc640 1 -- 172.21.15.38:0/2093862345 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68015020 con 0x7f4e6c0834b0 2024-04-22T00:05:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.299+0000 7f4e627fc640 1 -- 172.21.15.38:0/2093862345 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e68011d10 con 0x7f4e6c0834b0 2024-04-22T00:05:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.299+0000 7f4e627fc640 1 -- 172.21.15.38:0/2093862345 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800f2e0 con 0x7f4e6c0834b0 2024-04-22T00:05:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.299+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e6c1b5b70 con 0x7f4e6c0834b0 2024-04-22T00:05:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.299+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e6c1b6070 con 0x7f4e6c0834b0 2024-04-22T00:05:11.298 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.305+0000 7f4e627fc640 1 -- 172.21.15.38:0/2093862345 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4e68018020 con 0x7f4e6c0834b0 2024-04-22T00:05:11.298 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.305+0000 7f4e76474740 1 Processor -- start 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.306+0000 7f4e627fc640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 0x7f4e580a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e627fc640 1 -- 172.21.15.38:0/2093862345 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4e680cef30 con 0x7f4e6c0834b0 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e70e23640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 0x7f4e580a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e76474740 1 -- start start 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e76474740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 0x5637eaac3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e76474740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaacccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.307+0000 7f4e76474740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaacf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e76474740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5637eaa27b70 con 0x5637eaac3b20 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e76474740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637ea9d0470 con 0x5637eaacd210 2024-04-22T00:05:11.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e76474740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5637eaa23b30 con 0x5637eaac4440 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaacccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaacccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39202/0 (socket says 172.21.15.38:39202) 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e70e23640 1 -- 172.21.15.38:0/3760586087 learned_addr learned my addr 172.21.15.38:0/3760586087 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaacf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71624640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 0x5637eaac3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e70e23640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 0x7f4e580a5440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0841f0 tx=0x7f4e5c000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 -- 172.21.15.38:0/3760586087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 msgr2=0x5637eaacccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaacccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 -- 172.21.15.38:0/3760586087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 msgr2=0x5637eaac3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 0x5637eaac3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71e25640 1 -- 172.21.15.38:0/3760586087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637ea73c210 con 0x5637eaacd210 2024-04-22T00:05:11.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.308+0000 7f4e71624640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 0x5637eaac3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.309+0000 7f4e71e25640 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaacf600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e680cc3f0 tx=0x7f4e6809ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b0fd06caa5ce1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.309+0000 7f4e577fe640 1 -- 172.21.15.38:0/3760586087 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680b4070 con 0x5637eaacd210 2024-04-22T00:05:11.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.310+0000 7f4e577fe640 1 -- 172.21.15.38:0/3760586087 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e68010380 con 0x5637eaacd210 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.310+0000 7f4e577fe640 1 -- 172.21.15.38:0/3760586087 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680cc560 con 0x5637eaacd210 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.311+0000 7f4e76474740 1 -- 172.21.15.38:0/3760586087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 msgr2=0x5637eaacf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.311+0000 7f4e76474740 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaacf600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e680cc3f0 tx=0x7f4e6809ba00 comp rx=0 tx=0).stop 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.311+0000 7f4e76474740 1 -- 172.21.15.38:0/3760586087 shutdown_connections 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.311+0000 7f4e76474740 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaacf600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.311+0000 7f4e76474740 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaacccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.312+0000 7f4e76474740 1 --2- 172.21.15.38:0/3760586087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac3b20 0x5637eaac3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.312+0000 7f4e76474740 1 -- 172.21.15.38:0/3760586087 >> 172.21.15.38:0/3760586087 conn(0x5637eaabe440 msgr2=0x5637eaabe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.314+0000 7f4e76474740 1 -- 172.21.15.38:0/3760586087 shutdown_connections 2024-04-22T00:05:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.314+0000 7f4e76474740 1 -- 172.21.15.38:0/3760586087 wait complete. 2024-04-22T00:05:11.307 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.314+0000 7f4e76474740 1 -- start start 2024-04-22T00:05:11.308 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.314+0000 7f4e76474740 1 Processor -- start 2024-04-22T00:05:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.315+0000 7f4e76474740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaaca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.315+0000 7f4e76474740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.309 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e76474740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 0x5637eaacadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:39208/0 (socket says 172.21.15.38:39208) 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e70e23640 1 -- 172.21.15.38:0/1380785369 learned_addr learned my addr 172.21.15.38:0/1380785369 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5637eaacfe40 con 0x5637eaacd210 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5637ea9e3a10 con 0x5637eaac3b20 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.316+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5637eaacffc0 con 0x5637eaac4440 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e71e25640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 0x5637eaacadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e71624640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaaca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e76474740 10 client.0 _ll_register_callbacks cb 0x5637eaa1eb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e70e23640 1 -- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 msgr2=0x5637eaaca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e70e23640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaaca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.317+0000 7f4e70e23640 1 -- 172.21.15.38:0/1380785369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 msgr2=0x5637eaacadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.318+0000 7f4e70e23640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 0x5637eaacadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.318+0000 7f4e70e23640 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637ea808620 con 0x5637eaac4440 2024-04-22T00:05:11.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.318+0000 7f4e71e25640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 0x5637eaacadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.320+0000 7f4e70e23640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c10ea60 tx=0x7f4e5c02c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d076f4c421a672fa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.322+0000 7f4e54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:05:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.322+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c02e040 con 0x5637eaac4440 2024-04-22T00:05:11.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.322+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4e5c0035f0 con 0x5637eaac4440 2024-04-22T00:05:11.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.322+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e68010380 con 0x5637eaac4440 2024-04-22T00:05:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.327+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e5c0037f0 con 0x5637eaac4440 2024-04-22T00:05:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.327+0000 7f4e76474740 10 client.26095 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:11.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.328+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637eaadc0c0 con 0x5637eaac4440 2024-04-22T00:05:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.328+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e68010380 con 0x5637eaac4440 2024-04-22T00:05:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f4e5c005520 con 0x5637eaac4440 2024-04-22T00:05:11.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 10 client.26095 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 10 client.26095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:11.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 20 client.26095 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eaadd9f0 0x5637eaadfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 4 client.26095 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.329+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637eaa350b0 con 0x5637eaadd9f0 2024-04-22T00:05:11.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.330+0000 7f4e71624640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eaadd9f0 0x5637eaadfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.331+0000 7f4e71624640 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eaadd9f0 0x5637eaadfeb0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27183b08b371cee server_cookie=eb0fae5a78c478e9 in_seq=0 out_seq=0 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.331+0000 7f4e54ff9640 10 client.26095 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.332+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1380785369 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5637eaa350b0 con 0x5637eaadd9f0 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.332+0000 7f4e54ff9640 10 client.26095 handle_command_reply: tid=0 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.332+0000 7f4e76474740 1 client.26095 shutdown 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 msgr2=0x5637eaaca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:11.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c10ea60 tx=0x7f4e5c02c8f0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 shutdown_connections 2024-04-22T00:05:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eaadd9f0 0x5637eaadfeb0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaacd210 0x5637eaacadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac4440 0x5637eaaca880 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 --2- 172.21.15.38:0/1380785369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaaca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 >> 172.21.15.38:0/1380785369 conn(0x5637eaabe440 msgr2=0x5637eaac0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e54ff9640 0 client.26095 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.333+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 shutdown_connections 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.334+0000 7f4e76474740 1 -- 172.21.15.38:0/1380785369 wait complete. 2024-04-22T00:05:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.334+0000 7f4e76474740 20 client.26095 trim_cache size 0 max 16384 2024-04-22T00:05:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.367+0000 7f4e76474740 1 Processor -- start 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.367+0000 7f4e76474740 1 -- start start 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac3b20 0x5637eaacca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaae5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c1d7230 con 0x5637eaac4440 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c1d8ef0 con 0x5637eaacd210 2024-04-22T00:05:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e76474740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c1d7900 con 0x5637eaac3b20 2024-04-22T00:05:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e70e23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37272/0 (socket says 172.21.15.38:37272) 2024-04-22T00:05:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e70e23640 1 -- 172.21.15.38:0/148991619 learned_addr learned my addr 172.21.15.38:0/148991619 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.368+0000 7f4e71e25640 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaae5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 -- 172.21.15.38:0/148991619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac3b20 msgr2=0x5637eaacca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac3b20 0x5637eaacca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 -- 172.21.15.38:0/148991619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 msgr2=0x5637eaad8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 -- 172.21.15.38:0/148991619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637eaacff50 con 0x5637eaacd210 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.369+0000 7f4e71e25640 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaae5780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e680cc420 tx=0x7f4e68010970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea71c5b47d12464b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.370+0000 7f4e56ffd640 1 -- 172.21.15.38:0/148991619 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680b9070 con 0x5637eaacd210 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.370+0000 7f4e56ffd640 1 -- 172.21.15.38:0/148991619 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e680994c0 con 0x5637eaacd210 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.370+0000 7f4e56ffd640 1 -- 172.21.15.38:0/148991619 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680b4040 con 0x5637eaacd210 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 -- 172.21.15.38:0/148991619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 msgr2=0x5637eaae5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaae5780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e680cc420 tx=0x7f4e68010970 comp rx=0 tx=0).stop 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 -- 172.21.15.38:0/148991619 shutdown_connections 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaacd210 0x5637eaae5780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.371+0000 7f4e76474740 1 --2- 172.21.15.38:0/148991619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaac3b20 0x5637eaacca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.372+0000 7f4e76474740 1 -- 172.21.15.38:0/148991619 >> 172.21.15.38:0/148991619 conn(0x5637eaabe440 msgr2=0x5637eaac5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.373+0000 7f4e76474740 1 -- 172.21.15.38:0/148991619 shutdown_connections 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.373+0000 7f4e76474740 1 -- 172.21.15.38:0/148991619 wait complete. 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.373+0000 7f4e76474740 1 -- start start 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.373+0000 7f4e76474740 1 Processor -- start 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.374+0000 7f4e76474740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.374+0000 7f4e76474740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e71624640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e71624640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43910/0 (socket says 172.21.15.38:43910) 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e71624640 1 -- 172.21.15.38:0/1504054869 learned_addr learned my addr 172.21.15.38:0/1504054869 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaacd210 0x5637eaac69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c1d8760 con 0x5637eaac4440 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c1d8090 con 0x5637eaac3b20 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e71e25640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaacd210 0x5637eaac69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c04f500 con 0x5637eaacd210 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.375+0000 7f4e76474740 10 client.0 _ll_register_callbacks cb 0x7f4e5c0073e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.376+0000 7f4e71624640 1 -- 172.21.15.38:0/1504054869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaacd210 msgr2=0x5637eaac69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.376+0000 7f4e71624640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaacd210 0x5637eaac69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.376+0000 7f4e71624640 1 -- 172.21.15.38:0/1504054869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 msgr2=0x5637eaad5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.376+0000 7f4e71624640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.376+0000 7f4e71624640 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637eaac71f0 con 0x5637eaac3b20 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.377+0000 7f4e71624640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400f850 tx=0x7f4e6400f940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0732739ee5ec6f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.378+0000 7f4e54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.378+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6400d560 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.378+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4e6400d700 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.378+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e680994c0 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.379+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e64027050 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.379+0000 7f4e76474740 10 client.26231 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.379+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637eab43ae0 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.383+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e6400dde0 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f4e6400c520 con 0x5637eaac3b20 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 10 client.26231 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 20 client.26231 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eab44300 0x5637eab46720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 4 client.26231 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.384+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4e680994c0 con 0x5637eab44300 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.386+0000 7f4e70e23640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eab44300 0x5637eab46720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.389+0000 7f4e70e23640 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eab44300 0x5637eab46720 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cff3e1f769245dd7 server_cookie=3138ce624c08c5c3 in_seq=0 out_seq=0 2024-04-22T00:05:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:11.389+0000 7f4e54ff9640 10 client.26231 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:11.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:11 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:11.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:11.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:11.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:11.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:11 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:11.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:11 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:12.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:12 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:12.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:12 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:12.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:12 smithi203 ceph-mon[31431]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 55 MiB/s wr, 8.61k op/s 2024-04-22T00:05:12.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:12 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:12.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:12 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:12.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:12 smithi111 ceph-mon[31543]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 55 MiB/s wr, 8.61k op/s 2024-04-22T00:05:12.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:12 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:12.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:12 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:12.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:12 smithi038 ceph-mon[27972]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 55 MiB/s wr, 8.61k op/s 2024-04-22T00:05:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e54ff9640 1 -- 172.21.15.38:0/1504054869 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4e680994c0 con 0x5637eab44300 2024-04-22T00:05:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e54ff9640 10 client.26231 handle_command_reply: tid=0 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 client.26231 shutdown 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 msgr2=0x5637eaad5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400f850 tx=0x7f4e6400f940 comp rx=0 tx=0).stop 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 shutdown_connections 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5637eaacd210 0x5637eaac69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5637eab44300 0x5637eab46720 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5637eaac4440 0x5637eaad5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 --2- 172.21.15.38:0/1504054869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5637eaac3b20 0x5637eaad5210 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 >> 172.21.15.38:0/1504054869 conn(0x5637eaabe440 msgr2=0x5637eaabf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.479+0000 7f4e54ff9640 0 client.26231 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.480+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 shutdown_connections 2024-04-22T00:05:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.480+0000 7f4e76474740 1 -- 172.21.15.38:0/1504054869 wait complete. 2024-04-22T00:05:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.480+0000 7f4e76474740 20 client.26231 trim_cache size 0 max 16384 2024-04-22T00:05:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:05:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:05:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:05:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 msgr2=0x7f4e580a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 0x7f4e580a5440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0841f0 tx=0x7f4e5c000bc0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 msgr2=0x7f4e6c1c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800b3f0 tx=0x7f4e6800b8b0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 shutdown_connections 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e580a2f80 0x7f4e580a5440 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e6c0834b0 0x7f4e6c1c40c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e6c10dc60 0x7f4e6c082f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 --2- 172.21.15.38:0/2093862345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c10d0a0 0x7f4e6c082a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 >> 172.21.15.38:0/2093862345 conn(0x7f4e6c06c810 msgr2=0x7f4e6c071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:13.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 shutdown_connections 2024-04-22T00:05:13.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:13.481+0000 7f4e73888640 1 -- 172.21.15.38:0/2093862345 wait complete. 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:13.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:13 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:13.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:13 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:05:13.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:13.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:13.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:13.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:13.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:13 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:05:14.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:14 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:14.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:14 smithi203 ceph-mon[31431]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 76 MiB/s wr, 11.78k op/s 2024-04-22T00:05:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:14 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:14.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:14 smithi111 ceph-mon[31543]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 76 MiB/s wr, 11.78k op/s 2024-04-22T00:05:14.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:14 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:14.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:14 smithi038 ceph-mon[27972]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 76 MiB/s wr, 11.78k op/s 2024-04-22T00:05:15.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:15 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:15.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:15 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:15 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:15.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:15 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:15 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:15.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:15 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:16.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:16 smithi203 ceph-mon[31431]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 60 MiB/s wr, 9.14k op/s 2024-04-22T00:05:16.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:16 smithi111 ceph-mon[31543]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 60 MiB/s wr, 9.14k op/s 2024-04-22T00:05:16.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:16 smithi038 ceph-mon[27972]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 60 MiB/s wr, 9.14k op/s 2024-04-22T00:05:18.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:18 smithi203 ceph-mon[31431]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 MiB/s wr, 11.10k op/s 2024-04-22T00:05:18.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:18 smithi111 ceph-mon[31543]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 MiB/s wr, 11.10k op/s 2024-04-22T00:05:18.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:18 smithi038 ceph-mon[27972]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 MiB/s wr, 11.10k op/s 2024-04-22T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:20 smithi203 ceph-mon[31431]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 48 MiB/s wr, 7.52k op/s 2024-04-22T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:20 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:20 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:20 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:05:20.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:20 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:05:20.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:20 smithi111 ceph-mon[31543]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 48 MiB/s wr, 7.52k op/s 2024-04-22T00:05:20.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:20 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:20.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:20 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:05:20.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:20 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:05:20.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:20 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:05:20.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:20 smithi038 ceph-mon[27972]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 48 MiB/s wr, 7.52k op/s 2024-04-22T00:05:20.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:20 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:20.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:20 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:05:20.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:20 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:05:20.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:20 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:05:21.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:21 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:21.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:21 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:21.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:21 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:22.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:22 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:22.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:22 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:22.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:22 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:22.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:22 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:22.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:22 smithi111 ceph-mon[31543]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-22T00:05:22.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:22 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:22.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:22 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:22.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:22 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:22.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:22 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:22.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:22 smithi038 ceph-mon[27972]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-22T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:22 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:22 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:22 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:22 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:22 smithi203 ceph-mon[31431]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-22T00:05:24.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:24 smithi111 ceph-mon[31543]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 54 MiB/s wr, 8.81k op/s 2024-04-22T00:05:24.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:24 smithi038 ceph-mon[27972]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 54 MiB/s wr, 8.81k op/s 2024-04-22T00:05:24.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:24 smithi203 ceph-mon[31431]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 54 MiB/s wr, 8.81k op/s 2024-04-22T00:05:26.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:26 smithi111 ceph-mon[31543]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 35 MiB/s wr, 5.91k op/s 2024-04-22T00:05:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.630+0000 7f08ec07d640 1 -- 172.21.15.38:0/2775273100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e4110310 msgr2=0x7f08e4108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.630+0000 7f08ec07d640 1 --2- 172.21.15.38:0/2775273100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e4110310 0x7f08e4108c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009860 tx=0x7f08e002bc10 comp rx=0 tx=0).stop 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.631+0000 7f08ec07d640 1 -- 172.21.15.38:0/2775273100 shutdown_connections 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.631+0000 7f08ec07d640 1 --2- 172.21.15.38:0/2775273100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e411af20 0x7f08e411d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.631+0000 7f08ec07d640 1 --2- 172.21.15.38:0/2775273100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e4110310 0x7f08e4108c80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.631+0000 7f08ec07d640 1 --2- 172.21.15.38:0/2775273100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e410ff30 0x7f08e4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.631+0000 7f08ec07d640 1 -- 172.21.15.38:0/2775273100 >> 172.21.15.38:0/2775273100 conn(0x7f08e406c810 msgr2=0x7f08e406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.632+0000 7f08ec07d640 1 -- 172.21.15.38:0/2775273100 shutdown_connections 2024-04-22T00:05:26.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.632+0000 7f08ec07d640 1 -- 172.21.15.38:0/2775273100 wait complete. 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 Processor -- start 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 -- start start 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e410ff30 0x7f08e41184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e4118af0 0x7f08e4111b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4120920 con 0x7f08e411af20 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41207a0 con 0x7f08e4118af0 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08ec07d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4120aa0 con 0x7f08e410ff30 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08e9618640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08e9618640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60804/0 (socket says 172.21.15.38:60804) 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.633+0000 7f08e9618640 1 -- 172.21.15.38:0/1225416096 learned_addr learned my addr 172.21.15.38:0/1225416096 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9e19640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e410ff30 0x7f08e41184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08ea61a640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e4118af0 0x7f08e4111b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9618640 1 -- 172.21.15.38:0/1225416096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e410ff30 msgr2=0x7f08e41184a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9618640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e410ff30 0x7f08e41184a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9618640 1 -- 172.21.15.38:0/1225416096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e4118af0 msgr2=0x7f08e4111b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9618640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e4118af0 0x7f08e4111b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.634+0000 7f08e9618640 1 -- 172.21.15.38:0/1225416096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e4115fc0 con 0x7f08e411af20 2024-04-22T00:05:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.635+0000 7f08e9618640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f08e0000c00 tx=0x7f08e002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.636+0000 7f08daffd640 1 -- 172.21.15.38:0/1225416096 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0041020 con 0x7f08e411af20 2024-04-22T00:05:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.637+0000 7f08daffd640 1 -- 172.21.15.38:0/1225416096 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e0034920 con 0x7f08e411af20 2024-04-22T00:05:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.637+0000 7f08daffd640 1 -- 172.21.15.38:0/1225416096 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0039040 con 0x7f08e411af20 2024-04-22T00:05:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.638+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e4116250 con 0x7f08e411af20 2024-04-22T00:05:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.638+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41166c0 con 0x7f08e411af20 2024-04-22T00:05:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.643+0000 7f08daffd640 1 -- 172.21.15.38:0/1225416096 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f08e0044020 con 0x7f08e411af20 2024-04-22T00:05:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.644+0000 7f08daffd640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 0x7f08d00a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.645+0000 7f08daffd640 1 -- 172.21.15.38:0/1225416096 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08e00f0850 con 0x7f08e411af20 2024-04-22T00:05:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.646+0000 7f08eec66740 1 Processor -- start 2024-04-22T00:05:26.640 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 -- start start 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08e9e19640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 0x7f08d00a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9db9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9d1db70 con 0x55e9d9db9b20 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9cc6470 con 0x55e9d9dba440 2024-04-22T00:05:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.647+0000 7f08eec66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9d19b30 con 0x55e9d9dc3210 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08e9618640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08e9618640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:57220/0 (socket says 172.21.15.38:57220) 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08e9618640 1 -- 172.21.15.38:0/1207671935 learned_addr learned my addr 172.21.15.38:0/1207671935 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08ea61a640 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08e9618640 1 -- 172.21.15.38:0/1207671935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 msgr2=0x55e9d9dc5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.648+0000 7f08e9618640 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08e9618640 1 -- 172.21.15.38:0/1207671935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 msgr2=0x55e9d9db9f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:26.643 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08e9618640 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9db9f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08e9618640 1 -- 172.21.15.38:0/1207671935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9d9a32210 con 0x55e9d9dba440 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08ea61a640 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08e9618640 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f08e00bcaf0 tx=0x7f08e0003ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286ecf49641b3485 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.649+0000 7f08e9e19640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 0x7f08d00a5440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f08dc005fd0 tx=0x7f08dc0095c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.651+0000 7f08cffff640 1 -- 172.21.15.38:0/1207671935 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e002cc60 con 0x55e9d9dba440 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.651+0000 7f08cffff640 1 -- 172.21.15.38:0/1207671935 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e00bcb60 con 0x55e9d9dba440 2024-04-22T00:05:26.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.651+0000 7f08cffff640 1 -- 172.21.15.38:0/1207671935 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e00ede80 con 0x55e9d9dba440 2024-04-22T00:05:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 -- 172.21.15.38:0/1207671935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 msgr2=0x55e9d9dc2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f08e00bcaf0 tx=0x7f08e0003ab0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 -- 172.21.15.38:0/1207671935 shutdown_connections 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dba440 0x55e9d9dc2cd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 --2- 172.21.15.38:0/1207671935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9db9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 -- 172.21.15.38:0/1207671935 >> 172.21.15.38:0/1207671935 conn(0x55e9d9db4440 msgr2=0x55e9d9db4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 -- 172.21.15.38:0/1207671935 shutdown_connections 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.652+0000 7f08eec66740 1 -- 172.21.15.38:0/1207671935 wait complete. 2024-04-22T00:05:26.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.653+0000 7f08eec66740 1 -- start start 2024-04-22T00:05:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.653+0000 7f08eec66740 1 Processor -- start 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dc07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08e9e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08e9e19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60816/0 (socket says 172.21.15.38:60816) 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9dc5c00 con 0x55e9d9db9b20 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9cd9a10 con 0x55e9d9dc3210 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08eec66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e9d9dc5d80 con 0x55e9d9dba440 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.654+0000 7f08e9e19640 1 -- 172.21.15.38:0/4285348725 learned_addr learned my addr 172.21.15.38:0/4285348725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08eec66740 10 client.0 _ll_register_callbacks cb 0x55e9d9d14b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08ea61a640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08e9e19640 1 -- 172.21.15.38:0/4285348725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 msgr2=0x55e9d9dc07f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08e9e19640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dc07f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08e9e19640 1 -- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dc3210 msgr2=0x55e9d9dc0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08e9e19640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.655+0000 7f08e9e19640 1 -- 172.21.15.38:0/4285348725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9d9afe620 con 0x55e9d9db9b20 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.656+0000 7f08e9e19640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f08dc01bc60 tx=0x7f08dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38509dc6080aaa4c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.656+0000 7f08cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.656+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc02a060 con 0x55e9d9db9b20 2024-04-22T00:05:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.656+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08dc015030 con 0x55e9d9db9b20 2024-04-22T00:05:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.657+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08e00bcb60 con 0x55e9d9db9b20 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.657+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08dc031050 con 0x55e9d9db9b20 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.657+0000 7f08eec66740 10 client.16839 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.657+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9d9dd1e00 con 0x55e9d9db9b20 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.657+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08dc01b260 con 0x55e9d9db9b20 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f08dc00b040 con 0x55e9d9db9b20 2024-04-22T00:05:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 10 client.16839 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 20 client.16839 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9dd3380 0x55e9d9dd57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 4 client.16839 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f08e00bcb60 con 0x55e9d9dd3380 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.658+0000 7f08e9618640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9dd3380 0x55e9d9dd57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.659+0000 7f08cd7fa640 10 client.16839 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.659+0000 7f08e9618640 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9dd3380 0x55e9d9dd57e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b602649ed6313ad9 server_cookie=23a6bafb1fe1256c in_seq=0 out_seq=0 2024-04-22T00:05:26.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.660+0000 7f08cd7fa640 1 -- 172.21.15.38:0/4285348725 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f08e00bcb60 con 0x55e9d9dd3380 2024-04-22T00:05:26.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.660+0000 7f08cd7fa640 10 client.16839 handle_command_reply: tid=0 2024-04-22T00:05:26.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.660+0000 7f08eec66740 1 client.16839 shutdown 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 msgr2=0x55e9d9dc02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f08dc01bc60 tx=0x7f08dc004a40 comp rx=0 tx=0).stop 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 shutdown_connections 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dc07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.663+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9dd3380 0x55e9d9dd57e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.664+0000 7f08eec66740 1 --2- 172.21.15.38:0/4285348725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9db9b20 0x55e9d9dc02b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.664+0000 7f08cd7fa640 0 client.16839 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.664+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 >> 172.21.15.38:0/4285348725 conn(0x55e9d9db4440 msgr2=0x55e9d9db6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.665+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 shutdown_connections 2024-04-22T00:05:26.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.665+0000 7f08eec66740 1 -- 172.21.15.38:0/4285348725 wait complete. 2024-04-22T00:05:26.659 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.665+0000 7f08eec66740 20 client.16839 trim_cache size 0 max 16384 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.693+0000 7f08eec66740 1 Processor -- start 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 -- start start 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c60c0 con 0x55e9d9dc3210 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c7d80 con 0x55e9d9db9b20 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08eec66740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c6790 con 0x55e9d9dba440 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08ea61a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08ea61a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60828/0 (socket says 172.21.15.38:60828) 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08ea61a640 1 -- 172.21.15.38:0/105824887 learned_addr learned my addr 172.21.15.38:0/105824887 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08e9618640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.694+0000 7f08e9e19640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 -- 172.21.15.38:0/105824887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 msgr2=0x55e9d9dbf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 -- 172.21.15.38:0/105824887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 msgr2=0x55e9d9dcea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 -- 172.21.15.38:0/105824887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9d9db9400 con 0x55e9d9dc3210 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08e9e19640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08ea61a640 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f08d4005fd0 tx=0x7f08d400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b920aafd3b0dd76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08cf7fe640 1 -- 172.21.15.38:0/105824887 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4011d40 con 0x55e9d9dc3210 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08cf7fe640 1 -- 172.21.15.38:0/105824887 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08d4005500 con 0x55e9d9dc3210 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.695+0000 7f08cf7fe640 1 -- 172.21.15.38:0/105824887 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d401b770 con 0x55e9d9dc3210 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 -- 172.21.15.38:0/105824887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 msgr2=0x55e9d9dc0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f08d4005fd0 tx=0x7f08d400bf40 comp rx=0 tx=0).stop 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 -- 172.21.15.38:0/105824887 shutdown_connections 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dc0b50 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 --2- 172.21.15.38:0/105824887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.696+0000 7f08eec66740 1 -- 172.21.15.38:0/105824887 >> 172.21.15.38:0/105824887 conn(0x55e9d9db4440 msgr2=0x55e9d9db5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.697+0000 7f08eec66740 1 -- 172.21.15.38:0/105824887 shutdown_connections 2024-04-22T00:05:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.697+0000 7f08eec66740 1 -- 172.21.15.38:0/105824887 wait complete. 2024-04-22T00:05:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.697+0000 7f08eec66740 1 -- start start 2024-04-22T00:05:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.697+0000 7f08eec66740 1 Processor -- start 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08e9e19640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08e9e19640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:57236/0 (socket says 172.21.15.38:57236) 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dbd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08e9e19640 1 -- 172.21.15.38:0/2324184323 learned_addr learned my addr 172.21.15.38:0/2324184323 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c75f0 con 0x55e9d9dc3210 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c6f20 con 0x55e9d9db9b20 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41c40a0 con 0x55e9d9dba440 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.698+0000 7f08e9618640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.695 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08ea61a640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dbd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08eec66740 10 client.0 _ll_register_callbacks cb 0x7f08dc009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 -- 172.21.15.38:0/2324184323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 msgr2=0x55e9d9dbcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 -- 172.21.15.38:0/2324184323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 msgr2=0x55e9d9dbd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dbd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9d9dbdb50 con 0x55e9d9db9b20 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08ea61a640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dbd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08e9e19640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08dc01bc60 tx=0x7f08dc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e36fbeb3ecf69c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc039070 con 0x55e9d9db9b20 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f08dc004ab0 con 0x55e9d9db9b20 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.699+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d4005500 con 0x55e9d9db9b20 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08dc045050 con 0x55e9d9db9b20 2024-04-22T00:05:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 10 client.26246 fetch_fsmap learned FSMap version 32 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9d9e399a0 con 0x55e9d9db9b20 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08dc004cb0 con 0x55e9d9db9b20 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) ==== 8394+0+0 (secure 0 0 0) 0x7f08dc00b050 con 0x55e9d9db9b20 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 10 client.26246 fetch_fsmap finished waiting for FSMap version 32 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 10 client.26246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 20 client.26246 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9e3a210 0x55e9d9e3c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.700+0000 7f08eec66740 4 client.26246 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.701+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f08d4005500 con 0x55e9d9e3a210 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.701+0000 7f08e9618640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9e3a210 0x55e9d9e3c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.701+0000 7f08e9618640 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9e3a210 0x55e9d9e3c630 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533db28ea235e4ac server_cookie=c30a04568120dd0e in_seq=0 out_seq=0 2024-04-22T00:05:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:26.703+0000 7f08cd7fa640 10 client.26246 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:26.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:26 smithi038 ceph-mon[27972]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 35 MiB/s wr, 5.91k op/s 2024-04-22T00:05:26.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:26 smithi203 ceph-mon[31431]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 35 MiB/s wr, 5.91k op/s 2024-04-22T00:05:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.253+0000 7f08cd7fa640 1 -- 172.21.15.38:0/2324184323 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f08d4005500 con 0x55e9d9e3a210 2024-04-22T00:05:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.253+0000 7f08cd7fa640 10 client.26246 handle_command_reply: tid=0 2024-04-22T00:05:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.253+0000 7f08eec66740 1 client.26246 shutdown 2024-04-22T00:05:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 msgr2=0x55e9d9dcb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08dc01bc60 tx=0x7f08dc014000 comp rx=0 tx=0).stop 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 shutdown_connections 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e9d9e3a210 0x55e9d9e3c630 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9d9dc3210 0x55e9d9dbd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e9d9dba440 0x55e9d9dbcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 --2- 172.21.15.38:0/2324184323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e9d9db9b20 0x55e9d9dcb3e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08cd7fa640 0 client.26246 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.254+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 >> 172.21.15.38:0/2324184323 conn(0x55e9d9db4440 msgr2=0x55e9d9db60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.255+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 shutdown_connections 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.255+0000 7f08eec66740 1 -- 172.21.15.38:0/2324184323 wait complete. 2024-04-22T00:05:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.255+0000 7f08eec66740 20 client.26246 trim_cache size 0 max 16384 2024-04-22T00:05:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:05:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:05:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:05:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:05:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 msgr2=0x7f08d00a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 0x7f08d00a5440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f08dc005fd0 tx=0x7f08dc0095c0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 msgr2=0x7f08e4111520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f08e0000c00 tx=0x7f08e002c7e0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 shutdown_connections 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f08d00a2f80 0x7f08d00a5440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e4118af0 0x7f08e4111b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.256+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08e411af20 0x7f08e4111520 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.257+0000 7f08ec07d640 1 --2- 172.21.15.38:0/1225416096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08e410ff30 0x7f08e41184a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.257+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 >> 172.21.15.38:0/1225416096 conn(0x7f08e406c810 msgr2=0x7f08e410dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.257+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 shutdown_connections 2024-04-22T00:05:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:27.257+0000 7f08ec07d640 1 -- 172.21.15.38:0/1225416096 wait complete. 2024-04-22T00:05:28.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:28 smithi111 ceph-mon[31543]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 41 MiB/s wr, 7.42k op/s 2024-04-22T00:05:28.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:28 smithi038 ceph-mon[27972]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 41 MiB/s wr, 7.42k op/s 2024-04-22T00:05:28.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:28 smithi203 ceph-mon[31431]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 41 MiB/s wr, 7.42k op/s 2024-04-22T00:05:30.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:30 smithi111 ceph-mon[31543]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 MiB/s wr, 5.19k op/s 2024-04-22T00:05:30.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:30 smithi038 ceph-mon[27972]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 MiB/s wr, 5.19k op/s 2024-04-22T00:05:30.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:30 smithi203 ceph-mon[31431]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 MiB/s wr, 5.19k op/s 2024-04-22T00:05:32.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:32 smithi111 ceph-mon[31543]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 4.83k op/s 2024-04-22T00:05:32.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:32 smithi038 ceph-mon[27972]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 4.83k op/s 2024-04-22T00:05:32.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:32 smithi203 ceph-mon[31431]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 4.83k op/s 2024-04-22T00:05:33.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:33 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:33 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:33.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:33 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:34.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:34 smithi038 ceph-mon[27972]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 29 MiB/s wr, 5.76k op/s 2024-04-22T00:05:34.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:34 smithi203 ceph-mon[31431]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 29 MiB/s wr, 5.76k op/s 2024-04-22T00:05:34.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:34 smithi111 ceph-mon[31543]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 29 MiB/s wr, 5.76k op/s 2024-04-22T00:05:36.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:36 smithi038 ceph-mon[27972]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 3.65k op/s 2024-04-22T00:05:36.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:36 smithi203 ceph-mon[31431]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 3.65k op/s 2024-04-22T00:05:36.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:36 smithi111 ceph-mon[31543]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 3.65k op/s 2024-04-22T00:05:37.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:05:37.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:05:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2148328777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec1032f0 msgr2=0x7fdaec10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2148328777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec1032f0 0x7fdaec10b7d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009900 tx=0x7fdadc02bd60 comp rx=0 tx=0).stop 2024-04-22T00:05:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2148328777 shutdown_connections 2024-04-22T00:05:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2148328777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec10bd10 0x7fdaec10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2148328777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec1032f0 0x7fdaec10b7d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2148328777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1029d0 0x7fdaec102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2148328777 >> 172.21.15.38:0/2148328777 conn(0x7fdaec0769c0 msgr2=0x7fdaec076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2148328777 shutdown_connections 2024-04-22T00:05:37.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.645+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2148328777 wait complete. 2024-04-22T00:05:37.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 Processor -- start 2024-04-22T00:05:37.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 -- start start 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec1029d0 0x7fdaec10b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1032f0 0x7fdaec105340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10f950 con 0x7fdaec1029d0 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10f7d0 con 0x7fdaec10bd10 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.646+0000 7fdaf1d12640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec10fad0 con 0x7fdaec1032f0 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43432/0 (socket says 172.21.15.38:43432) 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaeb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec1029d0 0x7fdaec10b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 -- 172.21.15.38:0/2560476441 learned_addr learned my addr 172.21.15.38:0/2560476441 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1032f0 0x7fdaec105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 -- 172.21.15.38:0/2560476441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1032f0 msgr2=0x7fdaec105340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1032f0 0x7fdaec105340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 -- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec1029d0 msgr2=0x7fdaec10b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec1029d0 0x7fdaec10b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.647+0000 7fdaebfff640 1 -- 172.21.15.38:0/2560476441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec105e40 con 0x7fdaec10bd10 2024-04-22T00:05:37.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.648+0000 7fdaebfff640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdae400eb10 tx=0x7fdae400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.649+0000 7fdae8ff9640 1 -- 172.21.15.38:0/2560476441 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4004b20 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.649+0000 7fdae8ff9640 1 -- 172.21.15.38:0/2560476441 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae4004cc0 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.649+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaec106130 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.649+0000 7fdae8ff9640 1 -- 172.21.15.38:0/2560476441 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4010760 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.649+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaec071490 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdae8ff9640 1 -- 172.21.15.38:0/2560476441 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdae4005020 con 0x7fdaec10bd10 2024-04-22T00:05:37.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 Processor -- start 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 -- start start 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55772522d560 con 0x5577252c9850 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5577251d5730 con 0x5577252c8f30 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557725229520 con 0x5577252d2430 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.652+0000 7fdae8ff9640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 0x7fdad40a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdae8ff9640 1 -- 172.21.15.38:0/2560476441 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fdae40cfba0 con 0x7fdaec10bd10 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43442/0 (socket says 172.21.15.38:43442) 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1868177790 learned_addr learned my addr 172.21.15.38:0/1868177790 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1868177790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 msgr2=0x5577252d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252d4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1868177790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 msgr2=0x5577252d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaebfff640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 0x7fdad40a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.653+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1868177790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55772500b7d0 con 0x5577252c8f30 2024-04-22T00:05:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.654+0000 7fdaebfff640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 0x7fdad40a5350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdae40cd060 tx=0x7fdae4005b40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.654+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdadc005c40 tx=0x7fdadc005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c449d6e08650583a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.654+0000 7fdad1ffb640 1 -- 172.21.15.38:0/1868177790 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc049070 con 0x5577252c8f30 2024-04-22T00:05:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.654+0000 7fdad1ffb640 1 -- 172.21.15.38:0/1868177790 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdadc004320 con 0x5577252c8f30 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.654+0000 7fdad1ffb640 1 -- 172.21.15.38:0/1868177790 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc004ed0 con 0x5577252c8f30 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.655+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1868177790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 msgr2=0x5577252c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.655+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdadc005c40 tx=0x7fdadc005b50 comp rx=0 tx=0).stop 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1868177790 shutdown_connections 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252d4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1868177790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252c9310 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1868177790 >> 172.21.15.38:0/1868177790 conn(0x5577252c3660 msgr2=0x5577252c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.656+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1868177790 shutdown_connections 2024-04-22T00:05:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.657+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1868177790 wait complete. 2024-04-22T00:05:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.657+0000 7fdaf49ce740 1 -- start start 2024-04-22T00:05:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.657+0000 7fdaf49ce740 1 Processor -- start 2024-04-22T00:05:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43456/0 (socket says 172.21.15.38:43456) 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5577252d51c0 con 0x5577252c9850 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.658+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5577251e8e00 con 0x5577252c8f30 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5577252d5340 con 0x5577252d2430 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaf49ce740 10 client.0 _ll_register_callbacks cb 0x557725223eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 -- 172.21.15.38:0/900535420 learned_addr learned my addr 172.21.15.38:0/900535420 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 -- 172.21.15.38:0/900535420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 msgr2=0x5577252cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 -- 172.21.15.38:0/900535420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 msgr2=0x5577252cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.659+0000 7fdaeaffd640 1 -- 172.21.15.38:0/900535420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557724f413f0 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009a20 tx=0x7fdadc02c280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35da0fdc2764f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdacaffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc049070 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fdadc02cdc0 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdadc004320 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdadc057050 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdaf49ce740 10 client.26258 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557724f010f0 con 0x5577252c8f30 2024-04-22T00:05:37.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.660+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdadc004320 con 0x5577252c8f30 2024-04-22T00:05:37.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fdadc0038f0 con 0x5577252c8f30 2024-04-22T00:05:37.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 10 client.26258 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 10 client.26258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 20 client.26258 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5577252e2670 0x5577252e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 4 client.26258 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.661+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55772523aaa0 con 0x5577252e2670 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.662+0000 7fdaeb7fe640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5577252e2670 0x5577252e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.662+0000 7fdaeb7fe640 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5577252e2670 0x5577252e4a60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49580736bf244b15 server_cookie=4f330e8505319948 in_seq=0 out_seq=0 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.662+0000 7fdacaffd640 10 client.26258 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdacaffd640 1 -- 172.21.15.38:0/900535420 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55772523aaa0 con 0x5577252e2670 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdacaffd640 10 client.26258 handle_command_reply: tid=0 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 client.26258 shutdown 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 msgr2=0x5577252cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009a20 tx=0x7fdadc02c280 comp rx=0 tx=0).stop 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 shutdown_connections 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5577252e2670 0x5577252e4a60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c9850 0x5577252cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/900535420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c8f30 0x5577252cf490 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 >> 172.21.15.38:0/900535420 conn(0x5577252c3660 msgr2=0x5577252c53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdacaffd640 0 client.26258 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 shutdown_connections 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 1 -- 172.21.15.38:0/900535420 wait complete. 2024-04-22T00:05:37.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.663+0000 7fdaf49ce740 20 client.26258 trim_cache size 0 max 16384 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.691+0000 7fdaf49ce740 1 Processor -- start 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.691+0000 7fdaf49ce740 1 -- start start 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c9850 0x5577252de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c5f80 con 0x5577252c8f30 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c7bb0 con 0x5577252c9850 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c6650 con 0x5577252d2430 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34126/0 (socket says 172.21.15.38:34126) 2024-04-22T00:05:37.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 -- 172.21.15.38:0/808939214 learned_addr learned my addr 172.21.15.38:0/808939214 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 -- 172.21.15.38:0/808939214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 msgr2=0x5577252ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 -- 172.21.15.38:0/808939214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c9850 msgr2=0x5577252de0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c9850 0x5577252de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.692+0000 7fdaeaffd640 1 -- 172.21.15.38:0/808939214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577252c3fa0 con 0x5577252c8f30 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.693+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdadc001eb0 tx=0x7fdadc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a1494eb1235c40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.694+0000 7fdad17fa640 1 -- 172.21.15.38:0/808939214 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc062020 con 0x5577252c8f30 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.694+0000 7fdad17fa640 1 -- 172.21.15.38:0/808939214 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdadc003ad0 con 0x5577252c8f30 2024-04-22T00:05:37.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.695+0000 7fdaf49ce740 1 -- 172.21.15.38:0/808939214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 msgr2=0x5577252d1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.695+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdadc001eb0 tx=0x7fdadc005fd0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 -- 172.21.15.38:0/808939214 shutdown_connections 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252d2430 0x5577252ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252c9850 0x5577252de0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/808939214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d1bc0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 -- 172.21.15.38:0/808939214 >> 172.21.15.38:0/808939214 conn(0x5577252c3660 msgr2=0x5577252cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 -- 172.21.15.38:0/808939214 shutdown_connections 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 -- 172.21.15.38:0/808939214 wait complete. 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.696+0000 7fdaf49ce740 1 -- start start 2024-04-22T00:05:37.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 Processor -- start 2024-04-22T00:05:37.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252c9850 0x5577252d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252d2430 0x5577252d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c7420 con 0x5577252c8f30 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c6d50 con 0x5577252d2430 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1c3f60 con 0x5577252c9850 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaf49ce740 10 client.0 _ll_register_callbacks cb 0x7fdadc047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaeaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34134/0 (socket says 172.21.15.38:34134) 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1830069302 learned_addr learned my addr 172.21.15.38:0/1830069302 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaeb7fe640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252c9850 0x5577252d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.697+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252d2430 0x5577252d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1830069302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252c9850 msgr2=0x5577252d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252c9850 0x5577252d64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252d2430 msgr2=0x5577252d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252d2430 0x5577252d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 -- 172.21.15.38:0/1830069302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557725347870 con 0x5577252c8f30 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdaeaffd640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdadc003e60 tx=0x7fdadc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4f2ff8de73ac10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.698+0000 7fdacaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.699+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc062030 con 0x5577252c8f30 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.699+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fdadc0037f0 con 0x5577252c8f30 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.699+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdadc003ad0 con 0x5577252c8f30 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.700+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fdadc004ed0 con 0x5577252c8f30 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.700+0000 7fdaf49ce740 10 client.16860 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:37.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.700+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x557725348ae0 con 0x5577252c8f30 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.701+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdadc003ad0 con 0x5577252c8f30 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.701+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fdadc067040 con 0x5577252c8f30 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 10 client.16860 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 10 client.16860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 20 client.16860 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557725349130 0x55772534b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 4 client.16860 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:05:37.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557725234df0 con 0x557725349130 2024-04-22T00:05:37.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.702+0000 7fdaeb7fe640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557725349130 0x55772534b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.706+0000 7fdaeb7fe640 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557725349130 0x55772534b550 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffed089845ee5e3c server_cookie=8558c45878245f68 in_seq=0 out_seq=0 2024-04-22T00:05:37.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.707+0000 7fdacaffd640 10 client.16860 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:37.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.707+0000 7fdacaffd640 1 -- 172.21.15.38:0/1830069302 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557725234df0 con 0x557725349130 2024-04-22T00:05:37.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.707+0000 7fdacaffd640 10 client.16860 handle_command_reply: tid=0 2024-04-22T00:05:37.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.708+0000 7fdaf49ce740 1 client.16860 shutdown 2024-04-22T00:05:37.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 msgr2=0x5577252d5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdadc003e60 tx=0x7fdadc000bc0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 shutdown_connections 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5577252d2430 0x5577252d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5577252c9850 0x5577252d64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557725349130 0x55772534b550 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 --2- 172.21.15.38:0/1830069302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577252c8f30 0x5577252d5f90 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 >> 172.21.15.38:0/1830069302 conn(0x5577252c3660 msgr2=0x5577252ce470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdacaffd640 0 client.16860 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 shutdown_connections 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.709+0000 7fdaf49ce740 1 -- 172.21.15.38:0/1830069302 wait complete. 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.710+0000 7fdaf49ce740 20 client.16860 trim_cache size 0 max 16384 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:05:37.704 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:37.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.711+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 msgr2=0x7fdad40a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.711+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 0x7fdad40a5350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdae40cd060 tx=0x7fdae4005b40 comp rx=0 tx=0).stop 2024-04-22T00:05:37.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.711+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 msgr2=0x7fdaec105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.711+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdae400eb10 tx=0x7fdae400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 shutdown_connections 2024-04-22T00:05:37.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdad40a2e90 0x7fdad40a5350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdaec10bd10 0x7fdaec105880 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaec1032f0 0x7fdaec105340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 --2- 172.21.15.38:0/2560476441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaec1029d0 0x7fdaec10b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.713+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 >> 172.21.15.38:0/2560476441 conn(0x7fdaec0769c0 msgr2=0x7fdaec0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.714+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 shutdown_connections 2024-04-22T00:05:37.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.714+0000 7fdaf1d12640 1 -- 172.21.15.38:0/2560476441 wait complete. 2024-04-22T00:05:37.733 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:05:37.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:05:37.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/4241376820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 msgr2=0x7f4e9410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/4241376820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e9410d4f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009cd0 tx=0x7f4e8c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/4241376820 shutdown_connections 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/4241376820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e9410d4f0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/4241376820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94100620 0x7f4e94100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/4241376820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94106630 0x7f4e94106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.873+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/4241376820 >> 172.21.15.38:0/4241376820 conn(0x7f4e94077740 msgr2=0x7f4e94077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.874+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/4241376820 shutdown_connections 2024-04-22T00:05:37.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.874+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/4241376820 wait complete. 2024-04-22T00:05:37.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.874+0000 7f4e9c1bf640 1 Processor -- start 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 -- start start 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e94103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 0x7f4e941036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94110a00 con 0x7f4e94100fe0 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94110880 con 0x7f4e94106630 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9c1bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e94110b80 con 0x7f4e94100620 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e99f5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e99f5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48770/0 (socket says 172.21.15.38:48770) 2024-04-22T00:05:37.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e99f5b640 1 -- 172.21.15.38:0/724578588 learned_addr learned my addr 172.21.15.38:0/724578588 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9975a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e94103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.876+0000 7f4e9a75c640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 0x7f4e941036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 -- 172.21.15.38:0/724578588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 msgr2=0x7f4e941036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 0x7f4e941036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 -- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 msgr2=0x7f4e94103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e94103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 -- 172.21.15.38:0/724578588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e941a64a0 con 0x7f4e94100620 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e9a75c640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 0x7f4e941036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e99f5b640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000cd80 con 0x7f4e94100620 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e941a6790 con 0x7f4e94100620 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e90010910 con 0x7f4e94100620 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e941a6d30 con 0x7f4e94100620 2024-04-22T00:05:37.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.877+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900188f0 con 0x7f4e94100620 2024-04-22T00:05:37.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.878+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e78005290 con 0x7f4e94100620 2024-04-22T00:05:37.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.880+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4e90018bd0 con 0x7f4e94100620 2024-04-22T00:05:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.880+0000 7f4e8affd640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 0x7f4e800a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.880+0000 7f4e9975a640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 0x7f4e800a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.880+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4e900cf920 con 0x7f4e94100620 2024-04-22T00:05:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.881+0000 7f4e9975a640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 0x7f4e800a5240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4e941046d0 tx=0x7f4e84005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:37.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:37.885+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4e9009bbb0 con 0x7f4e94100620 2024-04-22T00:05:38.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.212+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e78005850 con 0x7f4e94100620 2024-04-22T00:05:38.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.215+0000 7f4e8affd640 1 -- 172.21.15.38:0/724578588 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11943 (secure 0 0 0) 0x7f4e9009feb0 con 0x7f4e94100620 2024-04-22T00:05:38.209 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:05:38.209 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:05:31.559778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24487,24460,24448,24455]},"id":1}]} 2024-04-22T00:05:38.209 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 33 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 msgr2=0x7f4e800a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 0x7f4e800a5240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4e941046d0 tx=0x7f4e84005f90 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 msgr2=0x7f4e94102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 shutdown_connections 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4e800a2d80 0x7f4e800a5240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94106630 0x7f4e941036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4e94100fe0 0x7f4e94103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 --2- 172.21.15.38:0/724578588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e94100620 0x7f4e94102c40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 >> 172.21.15.38:0/724578588 conn(0x7f4e94077740 msgr2=0x7f4e94068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 shutdown_connections 2024-04-22T00:05:38.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.219+0000 7f4e9c1bf640 1 -- 172.21.15.38:0/724578588 wait complete. 2024-04-22T00:05:38.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 -- 172.21.15.38:0/1178973067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344109990 msgr2=0x7f7344115460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 --2- 172.21.15.38:0/1178973067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344109990 0x7f7344115460 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f733c009c40 tx=0x7f733c02bef0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 -- 172.21.15.38:0/1178973067 shutdown_connections 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 --2- 172.21.15.38:0/1178973067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344109990 0x7f7344115460 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 --2- 172.21.15.38:0/1178973067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344108fd0 0x7f7344109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 --2- 172.21.15.38:0/1178973067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 0x7f734410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 -- 172.21.15.38:0/1178973067 >> 172.21.15.38:0/1178973067 conn(0x7f734406c850 msgr2=0x7f734406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 -- 172.21.15.38:0/1178973067 shutdown_connections 2024-04-22T00:05:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.401+0000 7f7349d5b640 1 -- 172.21.15.38:0/1178973067 wait complete. 2024-04-22T00:05:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.402+0000 7f7349d5b640 1 Processor -- start 2024-04-22T00:05:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 -- start start 2024-04-22T00:05:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344109990 0x7f73441a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 0x7f73441a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7344117c20 con 0x7f734410e6e0 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7344117aa0 con 0x7f7344109990 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7349d5b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7344117da0 con 0x7f7344108fd0 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f73437fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344109990 0x7f73441a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f73437fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48786/0 (socket says 172.21.15.38:48786) 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f73437fe640 1 -- 172.21.15.38:0/2897714793 learned_addr learned my addr 172.21.15.38:0/2897714793 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.403+0000 7f7343fff640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 0x7f73441a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 -- 172.21.15.38:0/2897714793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344109990 msgr2=0x7f73441a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344109990 0x7f73441a7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 -- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 msgr2=0x7f73441a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 0x7f73441a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 -- 172.21.15.38:0/2897714793 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73441a3660 con 0x7f7344108fd0 2024-04-22T00:05:38.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.404+0000 7f73437fe640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f733800e9f0 tx=0x7f733800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.405+0000 7f7340ff9640 1 -- 172.21.15.38:0/2897714793 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733800ccc0 con 0x7f7344108fd0 2024-04-22T00:05:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.405+0000 7f7340ff9640 1 -- 172.21.15.38:0/2897714793 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73380035f0 con 0x7f7344108fd0 2024-04-22T00:05:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.405+0000 7f7340ff9640 1 -- 172.21.15.38:0/2897714793 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733800ce60 con 0x7f7344108fd0 2024-04-22T00:05:38.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.406+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73441a3950 con 0x7f7344108fd0 2024-04-22T00:05:38.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.406+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73441af580 con 0x7f7344108fd0 2024-04-22T00:05:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.409+0000 7f734ca23740 1 Processor -- start 2024-04-22T00:05:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.410+0000 7f7340ff9640 1 -- 172.21.15.38:0/2897714793 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f73380037a0 con 0x7f7344108fd0 2024-04-22T00:05:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.411+0000 7f7340ff9640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 0x7f732c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.411+0000 7f7340ff9640 1 -- 172.21.15.38:0/2897714793 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7338099ba0 con 0x7f7344108fd0 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 -- start start 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 0x55664c2f4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55664c244530 con 0x55664c2e9150 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55664c1ec830 con 0x55664c2f1d30 2024-04-22T00:05:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.412+0000 7f734ca23740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55664c2404f0 con 0x55664c2e8830 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34170/0 (socket says 172.21.15.38:34170) 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f7342ffd640 1 -- 172.21.15.38:0/1575626003 learned_addr learned my addr 172.21.15.38:0/1575626003 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f73437fe640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f7343fff640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 0x55664c2f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f7343fff640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 0x7f732c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f73437fe640 1 -- 172.21.15.38:0/1575626003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 msgr2=0x55664c2f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.413+0000 7f73437fe640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 0x55664c2f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.414+0000 7f73437fe640 1 -- 172.21.15.38:0/1575626003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 msgr2=0x55664c2f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.414+0000 7f73437fe640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.414+0000 7f73437fe640 1 -- 172.21.15.38:0/1575626003 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55664c2102c0 con 0x55664c2e8830 2024-04-22T00:05:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.414+0000 7f7342ffd640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.414+0000 7f7343fff640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 0x55664c2f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.415+0000 7f73437fe640 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2e8c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73380ccd30 tx=0x7f73380a0fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32517759bba044ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.415+0000 7f7329ffb640 1 -- 172.21.15.38:0/1575626003 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7338098530 con 0x55664c2e8830 2024-04-22T00:05:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.415+0000 7f7343fff640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 0x7f732c0a5310 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55664c2e40f0 tx=0x7f733c005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.416+0000 7f7329ffb640 1 -- 172.21.15.38:0/1575626003 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733801ee50 con 0x55664c2e8830 2024-04-22T00:05:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.416+0000 7f7329ffb640 1 -- 172.21.15.38:0/1575626003 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73380986d0 con 0x55664c2e8830 2024-04-22T00:05:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.417+0000 7f734ca23740 1 -- 172.21.15.38:0/1575626003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 msgr2=0x55664c2e8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.417+0000 7f734ca23740 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2e8c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73380ccd30 tx=0x7f73380a0fd0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.419+0000 7f734ca23740 1 -- 172.21.15.38:0/1575626003 shutdown_connections 2024-04-22T00:05:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.419+0000 7f734ca23740 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2f1d30 0x55664c2f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.419+0000 7f734ca23740 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e9150 0x55664c2f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.419+0000 7f734ca23740 1 --2- 172.21.15.38:0/1575626003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2e8c10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.419+0000 7f734ca23740 1 -- 172.21.15.38:0/1575626003 >> 172.21.15.38:0/1575626003 conn(0x55664c2e2f60 msgr2=0x55664c2e3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.421+0000 7f734ca23740 1 -- 172.21.15.38:0/1575626003 shutdown_connections 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.421+0000 7f734ca23740 1 -- 172.21.15.38:0/1575626003 wait complete. 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.421+0000 7f734ca23740 1 -- start start 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.422+0000 7f734ca23740 1 Processor -- start 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.422+0000 7f734ca23740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2dfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55664c2da3c0 con 0x55664c2f1d30 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55664c1fff10 con 0x55664c2e9150 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55664c2da540 con 0x55664c2e8830 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43540/0 (socket says 172.21.15.38:43540) 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f734ca23740 10 client.0 _ll_register_callbacks cb 0x55664bf57be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.423+0000 7f7342ffd640 1 -- 172.21.15.38:0/310261458 learned_addr learned my addr 172.21.15.38:0/310261458 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f73437fe640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2dfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f7342ffd640 1 -- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 msgr2=0x55664c2dfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f7342ffd640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2dfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f7342ffd640 1 -- 172.21.15.38:0/310261458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 msgr2=0x55664c2f07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f7342ffd640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.424+0000 7f7342ffd640 1 -- 172.21.15.38:0/310261458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55664c1d1c80 con 0x55664c2e9150 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.425+0000 7f7342ffd640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f733400e780 tx=0x7f733400ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891d064e7ca05661 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.425+0000 7f7322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.426+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733401d060 con 0x55664c2e9150 2024-04-22T00:05:38.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.426+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f733801ee50 con 0x55664c2e9150 2024-04-22T00:05:38.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.427+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7334004230 con 0x55664c2e9150 2024-04-22T00:05:38.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.427+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7334024050 con 0x55664c2e9150 2024-04-22T00:05:38.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.428+0000 7f734ca23740 10 client.26279 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.428+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55664bf583f0 con 0x55664c2e9150 2024-04-22T00:05:38.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.429+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733801ee50 con 0x55664c2e9150 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.429+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f733400b040 con 0x55664c2e9150 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 10 client.26279 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 10 client.26279 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 20 client.26279 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c30a6b0 0x55664c30cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 4 client.26279 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:38.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.431+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55664c251a70 con 0x55664c30a6b0 2024-04-22T00:05:38.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.432+0000 7f73437fe640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c30a6b0 0x55664c30cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.433+0000 7f7322ffd640 10 client.26279 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:38.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.433+0000 7f73437fe640 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c30a6b0 0x55664c30cac0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b519e0f697b65b8 server_cookie=2130fb61a8cc3407 in_seq=0 out_seq=0 2024-04-22T00:05:38.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.435+0000 7f7322ffd640 1 -- 172.21.15.38:0/310261458 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55664c251a70 con 0x55664c30a6b0 2024-04-22T00:05:38.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.435+0000 7f7322ffd640 10 client.26279 handle_command_reply: tid=0 2024-04-22T00:05:38.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 client.26279 shutdown 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 msgr2=0x55664c2e01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f733400e780 tx=0x7f733400ec40 comp rx=0 tx=0).stop 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 shutdown_connections 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c30a6b0 0x55664c30cac0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e9150 0x55664c2e01a0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f733400e780 tx=0x7f733400ec40 comp rx=0 tx=0).stop 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 --2- 172.21.15.38:0/310261458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e8830 0x55664c2dfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 >> 172.21.15.38:0/310261458 conn(0x55664c2e2f60 msgr2=0x55664c2e4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f7322ffd640 0 client.26279 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 shutdown_connections 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.436+0000 7f734ca23740 1 -- 172.21.15.38:0/310261458 wait complete. 2024-04-22T00:05:38.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.437+0000 7f734ca23740 20 client.26279 trim_cache size 0 max 16384 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 Processor -- start 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 -- start start 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2e2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b51b0 con 0x55664c2f1d30 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b6de0 con 0x55664c2e8830 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f734ca23740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b5880 con 0x55664c2e0290 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f7343fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f7343fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48818/0 (socket says 172.21.15.38:48818) 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f7343fff640 1 -- 172.21.15.38:0/127188230 learned_addr learned my addr 172.21.15.38:0/127188230 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.462+0000 7f73437fe640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7342ffd640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7343fff640 1 -- 172.21.15.38:0/127188230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 msgr2=0x55664c2e2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7343fff640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2e2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7343fff640 1 -- 172.21.15.38:0/127188230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 msgr2=0x55664c2f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7343fff640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f7343fff640 1 -- 172.21.15.38:0/127188230 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55664c0227d0 con 0x55664c2e0290 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.463+0000 7f73437fe640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.464+0000 7f7343fff640 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f733c035000 tx=0x7f733c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7352ed98458f761f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.464+0000 7f73297fa640 1 -- 172.21.15.38:0/127188230 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c052070 con 0x55664c2e0290 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.464+0000 7f73297fa640 1 -- 172.21.15.38:0/127188230 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c02cd70 con 0x55664c2e0290 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.464+0000 7f73297fa640 1 -- 172.21.15.38:0/127188230 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c042470 con 0x55664c2e0290 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- 172.21.15.38:0/127188230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 msgr2=0x55664c2d96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f733c035000 tx=0x7f733c009500 comp rx=0 tx=0).stop 2024-04-22T00:05:38.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- 172.21.15.38:0/127188230 shutdown_connections 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2e0290 0x55664c2d96d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2f1d30 0x55664c2f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 --2- 172.21.15.38:0/127188230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2e2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- 172.21.15.38:0/127188230 >> 172.21.15.38:0/127188230 conn(0x55664c2e2f60 msgr2=0x55664c2d8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- 172.21.15.38:0/127188230 shutdown_connections 2024-04-22T00:05:38.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- 172.21.15.38:0/127188230 wait complete. 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.465+0000 7f734ca23740 1 -- start start 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.466+0000 7f734ca23740 1 Processor -- start 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.466+0000 7f734ca23740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e0290 0x55664c2f0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.466+0000 7f734ca23740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2eb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f734ca23740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 0x55664c2ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2eb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f7342ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2eb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43570/0 (socket says 172.21.15.38:43570) 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f7342ffd640 1 -- 172.21.15.38:0/1568821386 learned_addr learned my addr 172.21.15.38:0/1568821386 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f7343fff640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 0x55664c2ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b6650 con 0x55664c2e0290 2024-04-22T00:05:38.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b5f80 con 0x55664c2e8830 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.467+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73441b31d0 con 0x55664c2f1d30 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f734ca23740 10 client.0 _ll_register_callbacks cb 0x7f733400a810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 -- 172.21.15.38:0/1568821386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 msgr2=0x55664c2eb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2eb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 -- 172.21.15.38:0/1568821386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e0290 msgr2=0x55664c2f0570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e0290 0x55664c2f0570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55664c2ec670 con 0x55664c2f1d30 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7343fff640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 0x55664c2ebe70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f733c050040 tx=0x7f733c033b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=429345ca1b49d547 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7322ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:05:38.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7322ffd640 1 -- 172.21.15.38:0/1568821386 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c052040 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.468+0000 7f7322ffd640 1 -- 172.21.15.38:0/1568821386 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f733c004720 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.469+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f733c02cd70 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.469+0000 7f7322ffd640 1 -- 172.21.15.38:0/1568821386 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f733c02c890 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 10 client.26152 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55664c370a00 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f7322ffd640 1 -- 172.21.15.38:0/1568821386 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f733c060030 con 0x55664c2f1d30 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 10 client.26152 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 10 client.26152 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 20 client.26152 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c370f10 0x55664c373300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 4 client.26152 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:38.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.470+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f733c02cd70 con 0x55664c370f10 2024-04-22T00:05:38.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.472+0000 7f73437fe640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c370f10 0x55664c373300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.474+0000 7f73437fe640 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c370f10 0x55664c373300 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1fcb473e3637f5d5 server_cookie=c7e921db95ba797f in_seq=0 out_seq=0 2024-04-22T00:05:38.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.474+0000 7f7322ffd640 10 client.26152 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:38.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.474+0000 7f7322ffd640 1 -- 172.21.15.38:0/1568821386 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f733c02cd70 con 0x55664c370f10 2024-04-22T00:05:38.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.474+0000 7f7322ffd640 10 client.26152 handle_command_reply: tid=0 2024-04-22T00:05:38.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.474+0000 7f734ca23740 1 client.26152 shutdown 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 msgr2=0x55664c2ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 0x55664c2ebe70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f733c050040 tx=0x7f733c033b80 comp rx=0 tx=0).stop 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 shutdown_connections 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55664c370f10 0x55664c373300 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55664c2f1d30 0x55664c2ebe70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55664c2e8830 0x55664c2eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 --2- 172.21.15.38:0/1568821386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55664c2e0290 0x55664c2f0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.475+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 >> 172.21.15.38:0/1568821386 conn(0x55664c2e2f60 msgr2=0x55664c2de0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.476+0000 7f7322ffd640 0 client.26152 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:05:38.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.476+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 shutdown_connections 2024-04-22T00:05:38.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.476+0000 7f734ca23740 1 -- 172.21.15.38:0/1568821386 wait complete. 2024-04-22T00:05:38.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.477+0000 7f734ca23740 20 client.26152 trim_cache size 0 max 16384 2024-04-22T00:05:38.473 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:38.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.478+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 msgr2=0x7f732c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.478+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 0x7f732c0a5310 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55664c2e40f0 tx=0x7f733c005fd0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.478+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 msgr2=0x7f73441a7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.478+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f733800e9f0 tx=0x7f733800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.479+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 shutdown_connections 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.479+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f732c0a2e50 0x7f732c0a5310 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.479+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f734410e6e0 0x7f73441a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.479+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7344109990 0x7f73441a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.479+0000 7f7349d5b640 1 --2- 172.21.15.38:0/2897714793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7344108fd0 0x7f73441a7570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.480+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 >> 172.21.15.38:0/2897714793 conn(0x7f734406c850 msgr2=0x7f7344070770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.480+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 shutdown_connections 2024-04-22T00:05:38.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.480+0000 7f7349d5b640 1 -- 172.21.15.38:0/2897714793 wait complete. 2024-04-22T00:05:38.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:38 smithi038 ceph-mon[27972]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 5.47k op/s 2024-04-22T00:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:38 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/724578588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.673+0000 7f9677337640 1 -- 172.21.15.38:0/4066959006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 msgr2=0x7f9670069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.673+0000 7f9677337640 1 --2- 172.21.15.38:0/4066959006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f9670069b50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f9668009c10 tx=0x7f966802bf00 comp rx=0 tx=0).stop 2024-04-22T00:05:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.679+0000 7f9677337640 1 -- 172.21.15.38:0/4066959006 shutdown_connections 2024-04-22T00:05:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.679+0000 7f9677337640 1 --2- 172.21.15.38:0/4066959006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9670104160 0x7f96701045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.679+0000 7f9677337640 1 --2- 172.21.15.38:0/4066959006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967006a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.679+0000 7f9677337640 1 --2- 172.21.15.38:0/4066959006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f9670069b50 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.679+0000 7f9677337640 1 -- 172.21.15.38:0/4066959006 >> 172.21.15.38:0/4066959006 conn(0x7f96700776e0 msgr2=0x7f9670077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.682+0000 7f9677337640 1 -- 172.21.15.38:0/4066959006 shutdown_connections 2024-04-22T00:05:38.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.684+0000 7f9677337640 1 -- 172.21.15.38:0/4066959006 wait complete. 2024-04-22T00:05:38.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 Processor -- start 2024-04-22T00:05:38.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 -- start start 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9670104160 0x7f967019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f96701a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96701180f0 con 0x7f96701066a0 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9670117f70 con 0x7f967006a090 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.685+0000 7f9677337640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9670118270 con 0x7f9670104160 2024-04-22T00:05:38.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.686+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.686+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43592/0 (socket says 172.21.15.38:43592) 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.686+0000 7f96750d3640 1 -- 172.21.15.38:0/3851264581 learned_addr learned my addr 172.21.15.38:0/3851264581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.686+0000 7f96758d4640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f96701a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.687+0000 7f96750d3640 1 -- 172.21.15.38:0/3851264581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9670104160 msgr2=0x7f967019e970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.687+0000 7f96750d3640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9670104160 0x7f967019e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.687+0000 7f96750d3640 1 -- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 msgr2=0x7f96701a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.687+0000 7f96750d3640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f96701a2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.687+0000 7f96750d3640 1 -- 172.21.15.38:0/3851264581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96701a3420 con 0x7f967006a090 2024-04-22T00:05:38.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.688+0000 7f96750d3640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9668009560 tx=0x7f96680336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.689+0000 7f96667fc640 1 -- 172.21.15.38:0/3851264581 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668043020 con 0x7f967006a090 2024-04-22T00:05:38.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.690+0000 7f96667fc640 1 -- 172.21.15.38:0/3851264581 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9668034b20 con 0x7f967006a090 2024-04-22T00:05:38.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.691+0000 7f96667fc640 1 -- 172.21.15.38:0/3851264581 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966803ec50 con 0x7f967006a090 2024-04-22T00:05:38.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.691+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96701a36b0 con 0x7f967006a090 2024-04-22T00:05:38.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.692+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96701aafa0 con 0x7f967006a090 2024-04-22T00:05:38.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.695+0000 7f9679ff7740 1 Processor -- start 2024-04-22T00:05:38.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f96667fc640 1 -- 172.21.15.38:0/3851264581 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9668046020 con 0x7f967006a090 2024-04-22T00:05:38.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 -- start start 2024-04-22T00:05:38.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 0x55f7a43f2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a43fd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7a4356530 con 0x55f7a43fb520 2024-04-22T00:05:38.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f7a42fe830 con 0x55f7a43f2940 2024-04-22T00:05:38.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.697+0000 7f9679ff7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f7a43524f0 con 0x55f7a43f2020 2024-04-22T00:05:38.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.698+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.698+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43604/0 (socket says 172.21.15.38:43604) 2024-04-22T00:05:38.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.698+0000 7f96750d3640 1 -- 172.21.15.38:0/2206182871 learned_addr learned my addr 172.21.15.38:0/2206182871 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.698+0000 7f96758d4640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 0x55f7a43f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.698+0000 7f96748d2640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a43fd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96750d3640 1 -- 172.21.15.38:0/2206182871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 msgr2=0x55f7a43f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96750d3640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 0x55f7a43f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96750d3640 1 -- 172.21.15.38:0/2206182871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 msgr2=0x55f7a43fd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96750d3640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a43fd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96750d3640 1 -- 172.21.15.38:0/2206182871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7a43222c0 con 0x55f7a43f2940 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96748d2640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a43fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.699+0000 7f96758d4640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 0x55f7a43f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.700+0000 7f96667fc640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 0x7f965c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.700+0000 7f96667fc640 1 -- 172.21.15.38:0/3851264581 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96680f1960 con 0x7f967006a090 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.701+0000 7f96750d3640 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f96680eed60 tx=0x7f96680f7ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4788c5392a767d73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.701+0000 7f965b7fe640 1 -- 172.21.15.38:0/2206182871 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680b9e90 con 0x55f7a43f2940 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.701+0000 7f965b7fe640 1 -- 172.21.15.38:0/2206182871 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966803e450 con 0x55f7a43f2940 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.701+0000 7f965b7fe640 1 -- 172.21.15.38:0/2206182871 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680bd550 con 0x55f7a43f2940 2024-04-22T00:05:38.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.701+0000 7f96758d4640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 0x7f965c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.702+0000 7f9679ff7740 1 -- 172.21.15.38:0/2206182871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 msgr2=0x55f7a43fafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.702+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f96680eed60 tx=0x7f96680f7ef0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.702+0000 7f96758d4640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 0x7f965c0a5160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9660005fd0 tx=0x7f9660005cf0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 -- 172.21.15.38:0/2206182871 shutdown_connections 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a43fd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2940 0x55f7a43fafe0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f96680eed60 tx=0x7f96680f7ef0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2206182871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f2020 0x55f7a43f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 -- 172.21.15.38:0/2206182871 >> 172.21.15.38:0/2206182871 conn(0x55f7a43ec750 msgr2=0x55f7a43ecb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 -- 172.21.15.38:0/2206182871 shutdown_connections 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.703+0000 7f9679ff7740 1 -- 172.21.15.38:0/2206182871 wait complete. 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 -- start start 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 Processor -- start 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43fb520 0x55f7a43f88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 0x55f7a4407330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f7a42fe830 con 0x55f7a43f8e30 2024-04-22T00:05:38.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f7a4311f10 con 0x55f7a43f2020 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f7a43fe0d0 con 0x55f7a43fb520 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.704+0000 7f9679ff7740 10 client.0 _ll_register_callbacks cb 0x55f7a4069be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43614/0 (socket says 172.21.15.38:43614) 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96748d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43fb520 0x55f7a43f88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 -- 172.21.15.38:0/1137063601 learned_addr learned my addr 172.21.15.38:0/1137063601 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96758d4640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 0x55f7a4407330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 -- 172.21.15.38:0/1137063601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43fb520 msgr2=0x55f7a43f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43fb520 0x55f7a43f88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 -- 172.21.15.38:0/1137063601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 msgr2=0x55f7a4407330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 0x55f7a4407330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 -- 172.21.15.38:0/1137063601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7a42e3c80 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.705+0000 7f96750d3640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f966802c4e0 tx=0x7f966803f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdda13e3793b00c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.706+0000 7f9658ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680bd730 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f96758d4640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 0x55f7a4407330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96680eef90 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7a4363a70 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f96680bd9c0 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9679ff7740 10 client.26297 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f7a406a3f0 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f7a4363a70 con 0x55f7a43f2020 2024-04-22T00:05:38.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.707+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f9668046310 con 0x55f7a43f2020 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 10 client.26297 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 10 client.26297 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 20 client.26297 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a440b6a0 0x55f7a440dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 4 client.26297 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7a435ddc0 con 0x55f7a440b6a0 2024-04-22T00:05:38.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.708+0000 7f96748d2640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a440b6a0 0x55f7a440dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.709+0000 7f96748d2640 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a440b6a0 0x55f7a440dab0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fc6a88f1fae34101 server_cookie=e55f1960777d74cf in_seq=0 out_seq=0 2024-04-22T00:05:38.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.709+0000 7f9658ff9640 10 client.26297 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:38.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.710+0000 7f9658ff9640 1 -- 172.21.15.38:0/1137063601 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7a435ddc0 con 0x55f7a440b6a0 2024-04-22T00:05:38.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.710+0000 7f9658ff9640 10 client.26297 handle_command_reply: tid=0 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.710+0000 7f9679ff7740 1 client.26297 shutdown 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 msgr2=0x55f7a43f83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f966802c4e0 tx=0x7f966803f000 comp rx=0 tx=0).stop 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 shutdown_connections 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a440b6a0 0x55f7a440dab0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43f8e30 0x55f7a4407330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43fb520 0x55f7a43f88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 --2- 172.21.15.38:0/1137063601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a43f83b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 >> 172.21.15.38:0/1137063601 conn(0x55f7a43ec750 msgr2=0x55f7a43ee4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9658ff9640 0 client.26297 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 shutdown_connections 2024-04-22T00:05:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 1 -- 172.21.15.38:0/1137063601 wait complete. 2024-04-22T00:05:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.711+0000 7f9679ff7740 20 client.26297 trim_cache size 0 max 16384 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.736+0000 7f9679ff7740 1 Processor -- start 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.736+0000 7f9679ff7740 1 -- start start 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f8e30 0x55f7a44138c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4406d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43618/0 (socket says 172.21.15.38:43618) 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 -- 172.21.15.38:0/2171911707 learned_addr learned my addr 172.21.15.38:0/2171911707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ac540 con 0x55f7a43fb520 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ae170 con 0x55f7a43f2020 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96758d4640 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4406d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96748d2640 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f8e30 0x55f7a44138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96701acc10 con 0x55f7a43f8e30 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 -- 172.21.15.38:0/2171911707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f8e30 msgr2=0x55f7a44138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f8e30 0x55f7a44138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 -- 172.21.15.38:0/2171911707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 msgr2=0x55f7a4406d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4406d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.737+0000 7f96750d3640 1 -- 172.21.15.38:0/2171911707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7a41347d0 con 0x55f7a43f2020 2024-04-22T00:05:38.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.738+0000 7f96750d3640 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96680f79b0 tx=0x7f9668005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4b387f95aca13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.738+0000 7f965affd640 1 -- 172.21.15.38:0/2171911707 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680bd730 con 0x55f7a43f2020 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.738+0000 7f965affd640 1 -- 172.21.15.38:0/2171911707 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96680f9730 con 0x55f7a43f2020 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 msgr2=0x55f7a4413380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96680f79b0 tx=0x7f9668005e30 comp rx=0 tx=0).stop 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 shutdown_connections 2024-04-22T00:05:38.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4406d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a43f8e30 0x55f7a44138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 --2- 172.21.15.38:0/2171911707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f2020 0x55f7a4413380 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 >> 172.21.15.38:0/2171911707 conn(0x55f7a43ec750 msgr2=0x55f7a43eb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 shutdown_connections 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.739+0000 7f9679ff7740 1 -- 172.21.15.38:0/2171911707 wait complete. 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 -- start start 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 Processor -- start 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4403110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a4403650 0x55f7a43f4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ad9e0 con 0x55f7a43fb520 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96701ad310 con 0x55f7a43f8e30 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9670100c80 con 0x55f7a4403650 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.740+0000 7f9679ff7740 10 client.0 _ll_register_callbacks cb 0x7f96680f78e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96748d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4403110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43630/0 (socket says 172.21.15.38:43630) 2024-04-22T00:05:38.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96748d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4403110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34258/0 (socket says 172.21.15.38:34258) 2024-04-22T00:05:38.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 -- 172.21.15.38:0/3893000824 learned_addr learned my addr 172.21.15.38:0/3893000824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96758d4640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a4403650 0x55f7a43f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 -- 172.21.15.38:0/3893000824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a4403650 msgr2=0x55f7a43f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a4403650 0x55f7a43f4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 -- 172.21.15.38:0/3893000824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 msgr2=0x55f7a4403110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4403110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 -- 172.21.15.38:0/3893000824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7a43f4bf0 con 0x55f7a43f8e30 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f96750d3640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f966802c4e0 tx=0x7f9668005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca09b5db5f1d871 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f9658ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:38.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680bd550 con 0x55f7a43f8e30 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96680bde50 con 0x55f7a43f8e30 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.741+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7a42db6e0 con 0x55f7a43f8e30 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f96680c1ff0 con 0x55f7a43f8e30 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 10 client.26303 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f7a43f4bf0 con 0x55f7a43f8e30 2024-04-22T00:05:38.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f7a42db6e0 con 0x55f7a43f8e30 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f96680eef00 con 0x55f7a43f8e30 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 10 client.26303 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 10 client.26303 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 20 client.26303 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a44717f0 0x55f7a4473c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 4 client.26303 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.742+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7a4327bd0 con 0x55f7a44717f0 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.743+0000 7f96748d2640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a44717f0 0x55f7a4473c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9658ff9640 10 client.26303 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:38.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f96748d2640 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a44717f0 0x55f7a4473c10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=326cecbfd0a36686 server_cookie=642c6c86b061e160 in_seq=0 out_seq=0 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9658ff9640 1 -- 172.21.15.38:0/3893000824 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7a4327bd0 con 0x55f7a44717f0 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9658ff9640 10 client.26303 handle_command_reply: tid=0 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 client.26303 shutdown 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 msgr2=0x55f7a43f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f966802c4e0 tx=0x7f9668005ea0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 shutdown_connections 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7a4403650 0x55f7a43f4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55f7a44717f0 0x55f7a4473c10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f7a43fb520 0x55f7a4403110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 --2- 172.21.15.38:0/3893000824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f7a43f8e30 0x55f7a43f6000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 >> 172.21.15.38:0/3893000824 conn(0x55f7a43ec750 msgr2=0x55f7a43edaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.744+0000 7f9658ff9640 0 client.26303 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 shutdown_connections 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9679ff7740 1 -- 172.21.15.38:0/3893000824 wait complete. 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9679ff7740 20 client.26303 trim_cache size 0 max 16384 2024-04-22T00:05:38.746 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 msgr2=0x7f965c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 0x7f965c0a5160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9660005fd0 tx=0x7f9660005cf0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 msgr2=0x7f967019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.745+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9668009560 tx=0x7f96680336b0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 shutdown_connections 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f965c0a2ca0 0x7f965c0a5160 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96701066a0 0x7f96701a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9670104160 0x7f967019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 --2- 172.21.15.38:0/3851264581 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f967006a090 0x7f967019e430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.746+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 >> 172.21.15.38:0/3851264581 conn(0x7f96700776e0 msgr2=0x7f96700fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.748+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 shutdown_connections 2024-04-22T00:05:38.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.748+0000 7f9677337640 1 -- 172.21.15.38:0/3851264581 wait complete. 2024-04-22T00:05:38.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:05:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:38 smithi203 ceph-mon[31431]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 5.47k op/s 2024-04-22T00:05:38.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:38 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/724578588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:38 smithi111 ceph-mon[31543]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 5.47k op/s 2024-04-22T00:05:38.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:38 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/724578588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:05:38.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1601752625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e8100fd0 msgr2=0x7f02e810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1601752625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e8100fd0 0x7f02e810d4e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009930 tx=0x7f02e402bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:38.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1601752625 shutdown_connections 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1601752625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e8100fd0 0x7f02e810d4e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1601752625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100610 0x7f02e8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1601752625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e81066b0 0x7f02e8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.881+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1601752625 >> 172.21.15.38:0/1601752625 conn(0x7f02e8077740 msgr2=0x7f02e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1601752625 shutdown_connections 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1601752625 wait complete. 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 Processor -- start 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- start start 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100fd0 0x7f02e81973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e81066b0 0x7f02e819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8110a30 con 0x7f02e81066b0 2024-04-22T00:05:38.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81108b0 con 0x7f02e8100610 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ef4c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8110bb0 con 0x7f02e8100fd0 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02eda5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e81066b0 0x7f02e819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.882+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43644/0 (socket says 172.21.15.38:43644) 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 -- 172.21.15.38:0/1147314437 learned_addr learned my addr 172.21.15.38:0/1147314437 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02eca5d640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100fd0 0x7f02e81973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 -- 172.21.15.38:0/1147314437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100fd0 msgr2=0x7f02e81973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100fd0 0x7f02e81973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 -- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e81066b0 msgr2=0x7f02e819b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e81066b0 0x7f02e819b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 -- 172.21.15.38:0/1147314437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e819be80 con 0x7f02e8100610 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ed25e640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f02e000ebc0 tx=0x7f02e000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02de7fc640 1 -- 172.21.15.38:0/1147314437 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0003b70 con 0x7f02e8100610 2024-04-22T00:05:38.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e819c170 con 0x7f02e8100610 2024-04-22T00:05:38.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02de7fc640 1 -- 172.21.15.38:0/1147314437 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e0003d10 con 0x7f02e8100610 2024-04-22T00:05:38.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e81a3a00 con 0x7f02e8100610 2024-04-22T00:05:38.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.883+0000 7f02de7fc640 1 -- 172.21.15.38:0/1147314437 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0019980 con 0x7f02e8100610 2024-04-22T00:05:38.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 Processor -- start 2024-04-22T00:05:38.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 -- start start 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02de7fc640 1 -- 172.21.15.38:0/1147314437 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f02e0019b20 con 0x7f02e8100610 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78c9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78d4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5571b782d530 con 0x5571b78c9940 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5571b77d5830 con 0x5571b78d2520 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02f2185740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571b78294f0 con 0x5571b78c9020 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34284/0 (socket says 172.21.15.38:34284) 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 -- 172.21.15.38:0/4061330100 learned_addr learned my addr 172.21.15.38:0/4061330100 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02eda5f640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02eca5d640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78d4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 -- 172.21.15.38:0/4061330100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 msgr2=0x5571b78c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 -- 172.21.15.38:0/4061330100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 msgr2=0x5571b78d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78d4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02ed25e640 1 -- 172.21.15.38:0/4061330100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b77f92c0 con 0x5571b78c9940 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02de7fc640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 0x7f02d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.886+0000 7f02eca5d640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02eda5f640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 0x7f02d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02ed25e640 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f02e00cd380 tx=0x7f02e00d8320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbeee79c33c2c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02de7fc640 1 -- 172.21.15.38:0/1147314437 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f02e00d0090 con 0x7f02e8100610 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02d37fe640 1 -- 172.21.15.38:0/4061330100 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e009bfe0 con 0x5571b78c9940 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02d37fe640 1 -- 172.21.15.38:0/4061330100 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e00bae50 con 0x5571b78c9940 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.887+0000 7f02d37fe640 1 -- 172.21.15.38:0/4061330100 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e009bb70 con 0x5571b78c9940 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- 172.21.15.38:0/4061330100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 msgr2=0x5571b78d1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f02e00cd380 tx=0x7f02e00d8320 comp rx=0 tx=0).stop 2024-04-22T00:05:38.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02eda5f640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 0x7f02d40a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005940 tx=0x7f02e402e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- 172.21.15.38:0/4061330100 shutdown_connections 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9940 0x5571b78d1fe0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- 172.21.15.38:0/4061330100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- 172.21.15.38:0/4061330100 >> 172.21.15.38:0/4061330100 conn(0x5571b78c3750 msgr2=0x5571b78c3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- 172.21.15.38:0/4061330100 shutdown_connections 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- 172.21.15.38:0/4061330100 wait complete. 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 -- start start 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 Processor -- start 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78da2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.888+0000 7f02f2185740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78da820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5571b78d5310 con 0x5571b78d2520 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5571b77e8f10 con 0x5571b78c9940 2024-04-22T00:05:38.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571b78d5490 con 0x5571b78c9020 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 10 client.0 _ll_register_callbacks cb 0x5571b7540be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eda5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eda5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34294/0 (socket says 172.21.15.38:34294) 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eda5f640 1 -- 172.21.15.38:0/991384240 learned_addr learned my addr 172.21.15.38:0/991384240 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02ed25e640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78da2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 -- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 msgr2=0x5571b78da2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78da2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 -- 172.21.15.38:0/991384240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 msgr2=0x5571b78d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 -- 172.21.15.38:0/991384240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b77bac80 con 0x5571b78c9940 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eda5f640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:38.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02ed25e640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78da2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02eca5d640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78da820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02e81053c0 tx=0x7f02d8012950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ff641b6e0dba0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8004030 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e00bae50 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f02d80041d0 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f02d8025050 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.889+0000 7f02f2185740 10 client.26315 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5571b75413f0 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e00bae50 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f02d80039f0 con 0x5571b78c9940 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 10 client.26315 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 10 client.26315 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 20 client.26315 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b78e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 4 client.26315 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571b783aa70 con 0x5571b78e4910 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.890+0000 7f02ed25e640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b78e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02ed25e640 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b78e6d00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10c4db86953d9b3b server_cookie=3a53b65a3283c4ca in_seq=0 out_seq=0 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02d0ff9640 10 client.26315 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02d0ff9640 1 -- 172.21.15.38:0/991384240 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571b783aa70 con 0x5571b78e4910 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02d0ff9640 10 client.26315 handle_command_reply: tid=0 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 client.26315 shutdown 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 msgr2=0x5571b78da820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78da820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f02e81053c0 tx=0x7f02d8012950 comp rx=0 tx=0).stop 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 shutdown_connections 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b78e6d00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78d2520 0x5571b78d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02d0ff9640 0 client.26315 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:38.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78da820 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 --2- 172.21.15.38:0/991384240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9020 0x5571b78da2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.891+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 >> 172.21.15.38:0/991384240 conn(0x5571b78c3750 msgr2=0x5571b78c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.892+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 shutdown_connections 2024-04-22T00:05:38.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.892+0000 7f02f2185740 1 -- 172.21.15.38:0/991384240 wait complete. 2024-04-22T00:05:38.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.892+0000 7f02f2185740 20 client.26315 trim_cache size 0 max 16384 2024-04-22T00:05:38.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 Processor -- start 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 -- start start 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78d2520 0x5571b78ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81ac8f0 con 0x5571b78c9020 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81ae520 con 0x5571b78c9940 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.905+0000 7f02f2185740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81acfc0 con 0x5571b78d2520 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34310/0 (socket says 172.21.15.38:34310) 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 -- 172.21.15.38:0/2142760844 learned_addr learned my addr 172.21.15.38:0/2142760844 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02eca5d640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02eda5f640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78d2520 0x5571b78ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 -- 172.21.15.38:0/2142760844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78d2520 msgr2=0x5571b78ecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78d2520 0x5571b78ecb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 -- 172.21.15.38:0/2142760844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 msgr2=0x5571b78ec5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78ec5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 -- 172.21.15.38:0/2142760844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b760b7d0 con 0x5571b78c9020 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02eca5d640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02ed25e640 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f02e000eb90 tx=0x7f02e00d3b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9926d59f24c6bbd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02d2ffd640 1 -- 172.21.15.38:0/2142760844 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e009bd50 con 0x5571b78c9020 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02d2ffd640 1 -- 172.21.15.38:0/2142760844 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e00bae50 con 0x5571b78c9020 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.906+0000 7f02d2ffd640 1 -- 172.21.15.38:0/2142760844 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e00a05c0 con 0x5571b78c9020 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- 172.21.15.38:0/2142760844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 msgr2=0x5571b78ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f02e000eb90 tx=0x7f02e00d3b40 comp rx=0 tx=0).stop 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- 172.21.15.38:0/2142760844 shutdown_connections 2024-04-22T00:05:38.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78d2520 0x5571b78ecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78c9940 0x5571b78ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 --2- 172.21.15.38:0/2142760844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78ca690 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- 172.21.15.38:0/2142760844 >> 172.21.15.38:0/2142760844 conn(0x5571b78c3750 msgr2=0x5571b78c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- 172.21.15.38:0/2142760844 shutdown_connections 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- 172.21.15.38:0/2142760844 wait complete. 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 -- start start 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.907+0000 7f02f2185740 1 Processor -- start 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 0x5571b78ddf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81add90 con 0x5571b78c9020 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81ad6c0 con 0x5571b78d2520 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81aa8d0 con 0x5571b78c9940 2024-04-22T00:05:38.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02f2185740 10 client.0 _ll_register_callbacks cb 0x7f02d80096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34322/0 (socket says 172.21.15.38:34322) 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 -- 172.21.15.38:0/872545567 learned_addr learned my addr 172.21.15.38:0/872545567 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02eca5d640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 0x5571b78ddf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02eda5f640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 -- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 msgr2=0x5571b78ddf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 0x5571b78ddf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 -- 172.21.15.38:0/872545567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 msgr2=0x5571b78de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78de4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 -- 172.21.15.38:0/872545567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b78dece0 con 0x5571b78c9020 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02eca5d640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 0x5571b78ddf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02ed25e640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f02e00a4950 tx=0x7f02e0004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ea4f9dea7a47ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:38.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.908+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e009bd50 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f02e00988e0 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02e00bae50 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f02e009bfe0 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 10 client.16911 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5571b794ae30 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e00bae50 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f02e00d3c30 con 0x5571b78c9020 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 10 client.16911 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 10 client.16911 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 20 client.16911 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b794d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 4 client.16911 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571b7834dc0 con 0x5571b78e4910 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.909+0000 7f02eca5d640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b794d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:38.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02eca5d640 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b794d0c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=549c144ad3819c11 server_cookie=46f608c3e99c240b in_seq=0 out_seq=0 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02d0ff9640 10 client.16911 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02d0ff9640 1 -- 172.21.15.38:0/872545567 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571b7834dc0 con 0x5571b78e4910 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02d0ff9640 10 client.16911 handle_command_reply: tid=0 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 client.16911 shutdown 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 msgr2=0x5571b78e0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f02e00a4950 tx=0x7f02e0004910 comp rx=0 tx=0).stop 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 shutdown_connections 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5571b78e4910 0x5571b794d0c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571b78d2520 0x5571b78de4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571b78c9940 0x5571b78ddf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02d0ff9640 0 client.16911 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 --2- 172.21.15.38:0/872545567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5571b78c9020 0x5571b78e0be0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 >> 172.21.15.38:0/872545567 conn(0x5571b78c3750 msgr2=0x5571b78c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 shutdown_connections 2024-04-22T00:05:38.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 1 -- 172.21.15.38:0/872545567 wait complete. 2024-04-22T00:05:38.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.910+0000 7f02f2185740 20 client.16911 trim_cache size 0 max 16384 2024-04-22T00:05:38.911 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 msgr2=0x7f02d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 0x7f02d40a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f02e4005940 tx=0x7f02e402e040 comp rx=0 tx=0).stop 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 msgr2=0x7f02e8196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f02e000ebc0 tx=0x7f02e000c6a0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 shutdown_connections 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f02d40a2d00 0x7f02d40a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f02e81066b0 0x7f02e819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02e8100fd0 0x7f02e81973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 --2- 172.21.15.38:0/1147314437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02e8100610 0x7f02e8196e90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:38.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 >> 172.21.15.38:0/1147314437 conn(0x7f02e8077740 msgr2=0x7f02e80ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:38.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 shutdown_connections 2024-04-22T00:05:38.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:38.911+0000 7f02ef4c2640 1 -- 172.21.15.38:0/1147314437 wait complete. 2024-04-22T00:05:38.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd54259c640 1 -- 172.21.15.38:0/1150937 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c02cc80 con 0x7fd544069dc0 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 -- 172.21.15.38:0/1150937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd544069dc0 msgr2=0x7fd54406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 --2- 172.21.15.38:0/1150937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd544069dc0 0x7fd54406a240 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009930 tx=0x7fd52c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 -- 172.21.15.38:0/1150937 shutdown_connections 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 --2- 172.21.15.38:0/1150937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd54406a780 0x7fd54410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 --2- 172.21.15.38:0/1150937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd544069dc0 0x7fd54406a240 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 --2- 172.21.15.38:0/1150937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544107730 0x7fd544107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 -- 172.21.15.38:0/1150937 >> 172.21.15.38:0/1150937 conn(0x7fd5440776a0 msgr2=0x7fd544077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 -- 172.21.15.38:0/1150937 shutdown_connections 2024-04-22T00:05:39.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.024+0000 7fd549027640 1 -- 172.21.15.38:0/1150937 wait complete. 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 Processor -- start 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 -- start start 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 0x7fd544197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 0x7fd54419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd544111b90 con 0x7fd54406a780 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd544111a10 con 0x7fd544107730 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd549027640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd544111d10 con 0x7fd544069dc0 2024-04-22T00:05:39.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.025+0000 7fd53bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd53bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34338/0 (socket says 172.21.15.38:34338) 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 0x7fd544197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd53bfff640 1 -- 172.21.15.38:0/3412521719 learned_addr learned my addr 172.21.15.38:0/3412521719 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd54359e640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 0x7fd54419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 -- 172.21.15.38:0/3412521719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 msgr2=0x7fd54419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 0x7fd54419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 -- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 msgr2=0x7fd544198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 -- 172.21.15.38:0/3412521719 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54419cfb0 con 0x7fd544069dc0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd53bfff640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd54359e640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 0x7fd54419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 0x7fd544197fc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd53400d720 tx=0x7fd53400dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd540d99640 1 -- 172.21.15.38:0/3412521719 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534017070 con 0x7fd544069dc0 2024-04-22T00:05:39.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54419d2a0 con 0x7fd544069dc0 2024-04-22T00:05:39.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441a4b30 con 0x7fd544069dc0 2024-04-22T00:05:39.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd540d99640 1 -- 172.21.15.38:0/3412521719 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53400de30 con 0x7fd544069dc0 2024-04-22T00:05:39.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.026+0000 7fd540d99640 1 -- 172.21.15.38:0/3412521719 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534005020 con 0x7fd544069dc0 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd540d99640 1 -- 172.21.15.38:0/3412521719 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd5340046c0 con 0x7fd544069dc0 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 Processor -- start 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 -- start start 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cf9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cfc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5604d5c55530 con 0x5604d5cf1020 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5604d5bfd830 con 0x5604d5cf1940 2024-04-22T00:05:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604d5c514f0 con 0x5604d5cfa520 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34352/0 (socket says 172.21.15.38:34352) 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd54359e640 1 -- 172.21.15.38:0/2541748010 learned_addr learned my addr 172.21.15.38:0/2541748010 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd542d9d640 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cf9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.029+0000 7fd540d99640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 0x7fd5300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd53bfff640 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cfc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 0x7fd5300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 -- 172.21.15.38:0/2541748010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 msgr2=0x5604d5cfc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cfc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 -- 172.21.15.38:0/2541748010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 msgr2=0x5604d5cf9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cf9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 -- 172.21.15.38:0/2541748010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d5c212c0 con 0x5604d5cf1020 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd540d99640 1 -- 172.21.15.38:0/3412521719 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd5340cf1c0 con 0x7fd544069dc0 2024-04-22T00:05:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009fd0 tx=0x7fd53c013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0746326f0640e67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd54359e640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 0x7fd5300a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd53c01d000 tx=0x7fd53c01f000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd5397fa640 1 -- 172.21.15.38:0/2541748010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00f8b0 con 0x5604d5cf1020 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd5397fa640 1 -- 172.21.15.38:0/2541748010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53c022020 con 0x5604d5cf1020 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.030+0000 7fd5397fa640 1 -- 172.21.15.38:0/2541748010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c010a50 con 0x5604d5cf1020 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2541748010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 msgr2=0x5604d5cf1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009fd0 tx=0x7fd53c013830 comp rx=0 tx=0).stop 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2541748010 shutdown_connections 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cfc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cf9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2541748010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1020 0x5604d5cf1400 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2541748010 >> 172.21.15.38:0/2541748010 conn(0x5604d5ceb750 msgr2=0x5604d5cebb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2541748010 shutdown_connections 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2541748010 wait complete. 2024-04-22T00:05:39.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 -- start start 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.031+0000 7fd54bc0e740 1 Processor -- start 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5cf9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cf78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5604d5cfd2b0 con 0x5604d5cf1940 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5604d5c10f10 con 0x5604d5cf1020 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604d5cfd430 con 0x5604d5cfa520 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54bc0e740 10 client.0 _ll_register_callbacks cb 0x5604d5968be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34368/0 (socket says 172.21.15.38:34368) 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 -- 172.21.15.38:0/3223378550 learned_addr learned my addr 172.21.15.38:0/3223378550 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5cf9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54359e640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cf78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 -- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 msgr2=0x5604d5cf78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cf78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 -- 172.21.15.38:0/3223378550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 msgr2=0x5604d5cf9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5cf9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 -- 172.21.15.38:0/3223378550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d5be2c80 con 0x5604d5cf1940 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd54359e640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cf78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd53bfff640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009230 tx=0x7fd52c02c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd99b5e8b6884426 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd52a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.032+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c02cc70 con 0x5604d5cf1940 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd52c004040 con 0x5604d5cf1940 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53c022020 con 0x5604d5cf1940 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd52c004240 con 0x5604d5cf1940 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 10 client.16923 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:39.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5604d59693f0 con 0x5604d5cf1940 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53c022020 con 0x5604d5cf1940 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fd52c003950 con 0x5604d5cf1940 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 10 client.16923 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 10 client.16923 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 20 client.16923 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d0a690 0x5604d5d0caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 4 client.16923 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604d5c62a70 con 0x5604d5d0a690 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.033+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d0a690 0x5604d5d0caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.034+0000 7fd52a7fc640 10 client.16923 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.034+0000 7fd542d9d640 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d0a690 0x5604d5d0caf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f22de28c7c352caa server_cookie=8a0fdf4b87dc877c in_seq=0 out_seq=0 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.034+0000 7fd52a7fc640 1 -- 172.21.15.38:0/3223378550 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604d5c62a70 con 0x5604d5d0a690 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.034+0000 7fd52a7fc640 10 client.16923 handle_command_reply: tid=0 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 client.16923 shutdown 2024-04-22T00:05:39.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 msgr2=0x5604d5cf72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009230 tx=0x7fd52c02c880 comp rx=0 tx=0).stop 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 shutdown_connections 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d0a690 0x5604d5d0caf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cfa520 0x5604d5cf78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cf1940 0x5604d5cf72a0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd52a7fc640 0 client.16923 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/3223378550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5cf9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 >> 172.21.15.38:0/3223378550 conn(0x5604d5ceb750 msgr2=0x5604d5ced4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 shutdown_connections 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 1 -- 172.21.15.38:0/3223378550 wait complete. 2024-04-22T00:05:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.035+0000 7fd54bc0e740 20 client.16923 trim_cache size 0 max 16384 2024-04-22T00:05:39.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 Processor -- start 2024-04-22T00:05:39.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 -- start start 2024-04-22T00:05:39.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5d061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1940 0x5604d5cea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd52400fd70 con 0x5604d5cfa520 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5240119a0 con 0x5604d5cf1020 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.048+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd524010440 con 0x5604d5cf1940 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34376/0 (socket says 172.21.15.38:34376) 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 -- 172.21.15.38:0/1732778225 learned_addr learned my addr 172.21.15.38:0/1732778225 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 -- 172.21.15.38:0/1732778225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1940 msgr2=0x5604d5cea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1940 0x5604d5cea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 -- 172.21.15.38:0/1732778225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 msgr2=0x5604d5d061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5d061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 -- 172.21.15.38:0/1732778225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d5cf1770 con 0x5604d5cfa520 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd54359e640 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd53c012290 tx=0x7fd53c00fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1380b79a812977d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd538ff9640 1 -- 172.21.15.38:0/1732778225 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00f8b0 con 0x5604d5cfa520 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd538ff9640 1 -- 172.21.15.38:0/1732778225 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53c022040 con 0x5604d5cfa520 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.049+0000 7fd538ff9640 1 -- 172.21.15.38:0/1732778225 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c005460 con 0x5604d5cfa520 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- 172.21.15.38:0/1732778225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 msgr2=0x5604d5cf99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd53c012290 tx=0x7fd53c00fda0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- 172.21.15.38:0/1732778225 shutdown_connections 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cf99d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1940 0x5604d5cea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/1732778225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1020 0x5604d5d061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- 172.21.15.38:0/1732778225 >> 172.21.15.38:0/1732778225 conn(0x5604d5ceb750 msgr2=0x5604d5cebb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- 172.21.15.38:0/1732778225 shutdown_connections 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- 172.21.15.38:0/1732778225 wait complete. 2024-04-22T00:05:39.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- start start 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 Processor -- start 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1020 0x5604d5cfe270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cfe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd524011210 con 0x5604d5cfa520 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd524010b40 con 0x5604d5cf1940 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd52400dd50 con 0x5604d5cf1020 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.050+0000 7fd54bc0e740 10 client.0 _ll_register_callbacks cb 0x7fd52c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34382/0 (socket says 172.21.15.38:34382) 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 -- 172.21.15.38:0/2125372338 learned_addr learned my addr 172.21.15.38:0/2125372338 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd542d9d640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1020 0x5604d5cfe270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd53bfff640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cfe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 -- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1020 msgr2=0x5604d5cfe270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1020 0x5604d5cfe270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 -- 172.21.15.38:0/2125372338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 msgr2=0x5604d5cfe7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cfe7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 -- 172.21.15.38:0/2125372338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d5a337d0 con 0x5604d5cfa520 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd53bfff640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cfe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54359e640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00f720 tx=0x7fd53c00f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1c4ddf7a0a4fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd52a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c039020 con 0x5604d5cfa520 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd53c010700 con 0x5604d5cfa520 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53c022040 con 0x5604d5cfa520 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd53c010900 con 0x5604d5cfa520 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54bc0e740 10 client.16929 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:39.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5604d5d70f30 con 0x5604d5cfa520 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.051+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53c022040 con 0x5604d5cfa520 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fd53c027890 con 0x5604d5cfa520 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 10 client.16929 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 10 client.16929 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 20 client.16929 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d714e0 0x5604d5d73900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 4 client.16929 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604d5c5cdc0 con 0x5604d5d714e0 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd542d9d640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d714e0 0x5604d5d73900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd52a7fc640 10 client.16929 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.052+0000 7fd542d9d640 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d714e0 0x5604d5d73900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c33c57b1505444ea server_cookie=f85dedbb5bd17b99 in_seq=0 out_seq=0 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd52a7fc640 1 -- 172.21.15.38:0/2125372338 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604d5c5cdc0 con 0x5604d5d714e0 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd52a7fc640 10 client.16929 handle_command_reply: tid=0 2024-04-22T00:05:39.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 client.16929 shutdown 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 msgr2=0x5604d5cfecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00f720 tx=0x7fd53c00f4c0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 shutdown_connections 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x5604d5d714e0 0x5604d5d73900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604d5cfa520 0x5604d5cfecf0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5604d5cf1940 0x5604d5cfe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd52a7fc640 0 client.16929 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 --2- 172.21.15.38:0/2125372338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604d5cf1020 0x5604d5cfe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 >> 172.21.15.38:0/2125372338 conn(0x5604d5ceb750 msgr2=0x5604d5cea850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 shutdown_connections 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 1 -- 172.21.15.38:0/2125372338 wait complete. 2024-04-22T00:05:39.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd54bc0e740 20 client.16929 trim_cache size 0 max 16384 2024-04-22T00:05:39.054 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:39.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 msgr2=0x7fd5300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 0x7fd5300a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd53c01d000 tx=0x7fd53c01f000 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 msgr2=0x7fd544197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.053+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 0x7fd544197fc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd53400d720 tx=0x7fd53400dbe0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 shutdown_connections 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5300a2d00 0x7fd5300a51c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd544107730 0x7fd54419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd54406a780 0x7fd544198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 --2- 172.21.15.38:0/3412521719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544069dc0 0x7fd544197fc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 >> 172.21.15.38:0/3412521719 conn(0x7fd5440776a0 msgr2=0x7fd5440fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 shutdown_connections 2024-04-22T00:05:39.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.054+0000 7fd549027640 1 -- 172.21.15.38:0/3412521719 wait complete. 2024-04-22T00:05:39.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 -- 172.21.15.38:0/3031271898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc524103300 msgr2=0x7fc52410b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 --2- 172.21.15.38:0/3031271898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc524103300 0x7fc52410b7e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc518009900 tx=0x7fc51802bd70 comp rx=0 tx=0).stop 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 -- 172.21.15.38:0/3031271898 shutdown_connections 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 --2- 172.21.15.38:0/3031271898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc52410bd20 0x7fc52410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 --2- 172.21.15.38:0/3031271898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc524103300 0x7fc52410b7e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 --2- 172.21.15.38:0/3031271898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5241029e0 0x7fc524102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.225+0000 7fc52a82e640 1 -- 172.21.15.38:0/3031271898 >> 172.21.15.38:0/3031271898 conn(0x7fc524077740 msgr2=0x7fc524077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.226+0000 7fc52a82e640 1 -- 172.21.15.38:0/3031271898 shutdown_connections 2024-04-22T00:05:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.226+0000 7fc52a82e640 1 -- 172.21.15.38:0/3031271898 wait complete. 2024-04-22T00:05:39.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.227+0000 7fc52a82e640 1 Processor -- start 2024-04-22T00:05:39.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.227+0000 7fc52a82e640 1 -- start start 2024-04-22T00:05:39.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.228+0000 7fc52a82e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5241029e0 0x7fc52419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.228+0000 7fc52a82e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc52a82e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc52a82e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc524101d40 con 0x7fc5241029e0 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc528dcb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48974/0 (socket says 172.21.15.38:48974) 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc528dcb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43774/0 (socket says 172.21.15.38:43774) 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc5237fe640 1 -- 172.21.15.38:0/1369727904 learned_addr learned my addr 172.21.15.38:0/1369727904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc524101bc0 con 0x7fc52410bd20 2024-04-22T00:05:39.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc523fff640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5241029e0 0x7fc52419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.229+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc524101ec0 con 0x7fc524103300 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 -- 172.21.15.38:0/1369727904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 msgr2=0x7fc52419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 -- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5241029e0 msgr2=0x7fc52419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5241029e0 0x7fc52419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 -- 172.21.15.38:0/1369727904 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52419f250 con 0x7fc524103300 2024-04-22T00:05:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc528dcb640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc518000c00 tx=0x7fc51802dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5217fa640 1 -- 172.21.15.38:0/1369727904 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518043020 con 0x7fc524103300 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5217fa640 1 -- 172.21.15.38:0/1369727904 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180054e0 con 0x7fc524103300 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc5217fa640 1 -- 172.21.15.38:0/1369727904 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518003be0 con 0x7fc524103300 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.230+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52419f4e0 con 0x7fc524103300 2024-04-22T00:05:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.231+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52419f920 con 0x7fc524103300 2024-04-22T00:05:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.234+0000 7fc5217fa640 1 -- 172.21.15.38:0/1369727904 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc518046020 con 0x7fc524103300 2024-04-22T00:05:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.234+0000 7fc52d415740 1 Processor -- start 2024-04-22T00:05:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.234+0000 7fc52d415740 1 -- start start 2024-04-22T00:05:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c425cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc528dcb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc523fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c425cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5578c41b8530 con 0x5578c425d520 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc528dcb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:48982/0 (socket says 172.21.15.38:48982) 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5578c4160830 con 0x5578c4254940 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5578c41b44f0 con 0x5578c4254020 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc528dcb640 1 -- 172.21.15.38:0/3638747966 learned_addr learned my addr 172.21.15.38:0/3638747966 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.235+0000 7fc5217fa640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 0x7fc50c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 -- 172.21.15.38:0/3638747966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 msgr2=0x5578c425cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c425cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 -- 172.21.15.38:0/3638747966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 msgr2=0x5578c425f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 -- 172.21.15.38:0/3638747966 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c41842c0 con 0x5578c4254020 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc5217fa640 1 -- 172.21.15.38:0/1369727904 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc5180f1720 con 0x7fc524103300 2024-04-22T00:05:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 0x7fc50c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.236+0000 7fc528dcb640 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0118d0 tx=0x7fc51c011d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=346b3c1c06c26d29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.237+0000 7fc50a7fc640 1 -- 172.21.15.38:0/3638747966 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c017050 con 0x5578c4254020 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.237+0000 7fc50a7fc640 1 -- 172.21.15.38:0/3638747966 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc51c00e3b0 con 0x5578c4254020 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.237+0000 7fc50a7fc640 1 -- 172.21.15.38:0/3638747966 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c01a5c0 con 0x5578c4254020 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.237+0000 7fc528dcb640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 0x7fc50c0a5280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc51c023000 tx=0x7fc51c024000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 msgr2=0x5578c4254400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0118d0 tx=0x7fc51c011d90 comp rx=0 tx=0).stop 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 shutdown_connections 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c425cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 --2- 172.21.15.38:0/3638747966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c4254400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 >> 172.21.15.38:0/3638747966 conn(0x5578c424e750 msgr2=0x5578c424eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 shutdown_connections 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- 172.21.15.38:0/3638747966 wait complete. 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.238+0000 7fc52d415740 1 -- start start 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 Processor -- start 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c425a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c425a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc523fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc523fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43788/0 (socket says 172.21.15.38:43788) 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc523fff640 1 -- 172.21.15.38:0/337793778 learned_addr learned my addr 172.21.15.38:0/337793778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5578c4260310 con 0x5578c4254940 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5578c4173f10 con 0x5578c4254020 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5578c4260490 con 0x5578c425d520 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc5237fe640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c425a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.239+0000 7fc52d415740 10 client.0 _ll_register_callbacks cb 0x5578c3ecbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc528dcb640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c425a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 -- 172.21.15.38:0/337793778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 msgr2=0x5578c425a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c425a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 -- 172.21.15.38:0/337793778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 msgr2=0x5578c425a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c425a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c4145c80 con 0x5578c4254020 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc5237fe640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c425a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc523fff640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc51400b920 tx=0x7fc51400bd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf0be5639006a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc5037fe640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:39.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc514005020 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc5140051c0 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.240+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc51c00e3b0 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc5140053c0 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 10 client.26348 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578c3ecc3f0 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc514005560 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc514019030 con 0x5578c4254020 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 10 client.26348 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 10 client.26348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 20 client.26348 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.241+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c426d780 0x5578c426fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.242+0000 7fc52d415740 4 client.26348 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:39.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.242+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc51c00e3b0 con 0x5578c426d780 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.242+0000 7fc5237fe640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c426d780 0x5578c426fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.242+0000 7fc5237fe640 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c426d780 0x5578c426fbc0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2110bf4b20f5c9c server_cookie=aeb22e36ad29ffa3 in_seq=0 out_seq=0 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.242+0000 7fc5037fe640 10 client.26348 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc5037fe640 1 -- 172.21.15.38:0/337793778 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc51c00e3b0 con 0x5578c426d780 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc5037fe640 10 client.26348 handle_command_reply: tid=0 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 client.26348 shutdown 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 msgr2=0x5578c425cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc51400b920 tx=0x7fc51400bd20 comp rx=0 tx=0).stop 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 shutdown_connections 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c425a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c426d780 0x5578c426fbc0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c425a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 --2- 172.21.15.38:0/337793778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c425cf00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 >> 172.21.15.38:0/337793778 conn(0x5578c424e750 msgr2=0x5578c42504a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.243+0000 7fc5037fe640 0 client.26348 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:39.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.244+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 shutdown_connections 2024-04-22T00:05:39.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.244+0000 7fc52d415740 1 -- 172.21.15.38:0/337793778 wait complete. 2024-04-22T00:05:39.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.244+0000 7fc52d415740 20 client.26348 trim_cache size 0 max 16384 2024-04-22T00:05:39.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.269+0000 7fc52d415740 1 Processor -- start 2024-04-22T00:05:39.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.269+0000 7fc52d415740 1 -- start start 2024-04-22T00:05:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.269+0000 7fc52d415740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c424cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.269+0000 7fc52d415740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.269+0000 7fc52d415740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc52d415740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b0cb0 con 0x5578c425d520 2024-04-22T00:05:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc52d415740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b28e0 con 0x5578c4254940 2024-04-22T00:05:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc52d415740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b1380 con 0x5578c4254020 2024-04-22T00:05:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43800/0 (socket says 172.21.15.38:43800) 2024-04-22T00:05:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 -- 172.21.15.38:0/1943718178 learned_addr learned my addr 172.21.15.38:0/1943718178 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 -- 172.21.15.38:0/1943718178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 msgr2=0x5578c424cfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c424cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 -- 172.21.15.38:0/1943718178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 msgr2=0x5578c425ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 -- 172.21.15.38:0/1943718178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c3f967d0 con 0x5578c4254940 2024-04-22T00:05:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.270+0000 7fc5237fe640 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5180bd940 tx=0x7fc518009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7660f07753146ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.271+0000 7fc509ffb640 1 -- 172.21.15.38:0/1943718178 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518005690 con 0x5578c4254940 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.271+0000 7fc509ffb640 1 -- 172.21.15.38:0/1943718178 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180be0b0 con 0x5578c4254940 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.271+0000 7fc509ffb640 1 -- 172.21.15.38:0/1943718178 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180bbae0 con 0x5578c4254940 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.272+0000 7fc52d415740 1 -- 172.21.15.38:0/1943718178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 msgr2=0x5578c4259ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.272+0000 7fc52d415740 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc5180bd940 tx=0x7fc518009260 comp rx=0 tx=0).stop 2024-04-22T00:05:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.273+0000 7fc52d415740 1 -- 172.21.15.38:0/1943718178 shutdown_connections 2024-04-22T00:05:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.273+0000 7fc52d415740 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c425d520 0x5578c425ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.273+0000 7fc52d415740 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254940 0x5578c4259ab0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.273+0000 7fc52d415740 1 --2- 172.21.15.38:0/1943718178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c4254020 0x5578c424cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.273+0000 7fc52d415740 1 -- 172.21.15.38:0/1943718178 >> 172.21.15.38:0/1943718178 conn(0x5578c424e750 msgr2=0x5578c42502e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.276+0000 7fc52d415740 1 -- 172.21.15.38:0/1943718178 shutdown_connections 2024-04-22T00:05:39.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.276+0000 7fc52d415740 1 -- 172.21.15.38:0/1943718178 wait complete. 2024-04-22T00:05:39.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.276+0000 7fc52d415740 1 -- start start 2024-04-22T00:05:39.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 Processor -- start 2024-04-22T00:05:39.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c4265b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c4261460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b2150 con 0x5578c4254940 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241b1a80 con 0x5578c4254020 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc5241aec90 con 0x5578c425d520 2024-04-22T00:05:39.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.277+0000 7fc52d415740 10 client.0 _ll_register_callbacks cb 0x7fc514004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:39.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34428/0 (socket says 172.21.15.38:34428) 2024-04-22T00:05:39.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 -- 172.21.15.38:0/3410160494 learned_addr learned my addr 172.21.15.38:0/3410160494 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:39.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 -- 172.21.15.38:0/3410160494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 msgr2=0x5578c4261460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:39.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c4261460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 -- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 msgr2=0x5578c4265b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c4265b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.278+0000 7fc5237fe640 1 -- 172.21.15.38:0/3410160494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c4261c90 con 0x5578c4254940 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.280+0000 7fc5237fe640 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc5180ba380 tx=0x7fc518005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4b4ec98117490a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.280+0000 7fc5037fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:39.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.280+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180bd9b0 con 0x5578c4254940 2024-04-22T00:05:39.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.280+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc5180bbcb0 con 0x5578c4254940 2024-04-22T00:05:39.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.281+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5180be0b0 con 0x5578c4254940 2024-04-22T00:05:39.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.281+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc5180ba020 con 0x5578c4254940 2024-04-22T00:05:39.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.284+0000 7fc52d415740 10 client.16941 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:39.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.284+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578c42dc5b0 con 0x5578c4254940 2024-04-22T00:05:39.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180be0b0 con 0x5578c4254940 2024-04-22T00:05:39.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7fc5180bece0 con 0x5578c4254940 2024-04-22T00:05:39.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 10 client.16941 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:39.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:39.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 20 client.16941 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:39.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c42dcc40 0x5578c42df060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:39.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 4 client.16941 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:05:39.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.285+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578c41c5a70 con 0x5578c42dcc40 2024-04-22T00:05:39.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.287+0000 7fc523fff640 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c42dcc40 0x5578c42df060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:39.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.288+0000 7fc523fff640 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c42dcc40 0x5578c42df060 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f9f922e4bef75a0 server_cookie=15cc4e6dd9314dcf in_seq=0 out_seq=0 2024-04-22T00:05:39.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.289+0000 7fc5037fe640 10 client.16941 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:39.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.289+0000 7fc5037fe640 1 -- 172.21.15.38:0/3410160494 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578c41c5a70 con 0x5578c42dcc40 2024-04-22T00:05:39.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.289+0000 7fc5037fe640 10 client.16941 handle_command_reply: tid=0 2024-04-22T00:05:39.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.289+0000 7fc52d415740 1 client.16941 shutdown 2024-04-22T00:05:39.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.289+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 msgr2=0x5578c4260ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc5180ba380 tx=0x7fc518005780 comp rx=0 tx=0).stop 2024-04-22T00:05:39.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 shutdown_connections 2024-04-22T00:05:39.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5578c425d520 0x5578c4261460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5578c42dcc40 0x5578c42df060 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5578c4254940 0x5578c4260ee0 secure :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc5180ba380 tx=0x7fc518005780 comp rx=0 tx=0).stop 2024-04-22T00:05:39.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 --2- 172.21.15.38:0/3410160494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5578c4254020 0x5578c4265b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 >> 172.21.15.38:0/3410160494 conn(0x5578c424e750 msgr2=0x5578c4254e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc5037fe640 0 client.16941 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:39.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 shutdown_connections 2024-04-22T00:05:39.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.290+0000 7fc52d415740 1 -- 172.21.15.38:0/3410160494 wait complete. 2024-04-22T00:05:39.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.291+0000 7fc52d415740 20 client.16941 trim_cache size 0 max 16384 2024-04-22T00:05:39.288 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:05:39.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 msgr2=0x7fc50c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 0x7fc50c0a5280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc51c023000 tx=0x7fc51c024000 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 msgr2=0x7fc52419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc518000c00 tx=0x7fc51802dd40 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 shutdown_connections 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc50c0a2dc0 0x7fc50c0a5280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc52410bd20 0x7fc52419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc524103300 0x7fc52419a7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 --2- 172.21.15.38:0/1369727904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5241029e0 0x7fc52419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 >> 172.21.15.38:0/1369727904 conn(0x7fc524077740 msgr2=0x7fc5240fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:39.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 shutdown_connections 2024-04-22T00:05:39.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:39.295+0000 7fc52a82e640 1 -- 172.21.15.38:0/1369727904 wait complete. 2024-04-22T00:05:40.307 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:05:40.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:05:40.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:05:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 -- 172.21.15.38:0/4040745161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 msgr2=0x7ffb981008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/4040745161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 0x7ffb981008e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009930 tx=0x7ffb8802bc20 comp rx=0 tx=0).stop 2024-04-22T00:05:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 -- 172.21.15.38:0/4040745161 shutdown_connections 2024-04-22T00:05:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/4040745161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98100e20 0x7ffb9810d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/4040745161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 0x7ffb981008e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/4040745161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98106500 0x7ffb981068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 -- 172.21.15.38:0/4040745161 >> 172.21.15.38:0/4040745161 conn(0x7ffb980fbf90 msgr2=0x7ffb980fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 -- 172.21.15.38:0/4040745161 shutdown_connections 2024-04-22T00:05:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.408+0000 7ffb9e11e640 1 -- 172.21.15.38:0/4040745161 wait complete. 2024-04-22T00:05:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.409+0000 7ffb9e11e640 1 Processor -- start 2024-04-22T00:05:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.409+0000 7ffb9e11e640 1 -- start start 2024-04-22T00:05:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 0x7ffb980fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98100e20 0x7ffb980fca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffb98110860 con 0x7ffb98100e20 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981106e0 con 0x7ffb98106500 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb9e11e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb981109e0 con 0x7ffb98100460 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43836/0 (socket says 172.21.15.38:43836) 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 -- 172.21.15.38:0/256310973 learned_addr learned my addr 172.21.15.38:0/256310973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98100e20 0x7ffb980fca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 -- 172.21.15.38:0/256310973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 msgr2=0x7ffb980fc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 0x7ffb980fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 -- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98100e20 msgr2=0x7ffb980fca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98100e20 0x7ffb980fca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.410+0000 7ffb97fff640 1 -- 172.21.15.38:0/256310973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb981a6400 con 0x7ffb98106500 2024-04-22T00:05:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.411+0000 7ffb97fff640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000efd0 tx=0x7ffb9000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.411+0000 7ffb94ff9640 1 -- 172.21.15.38:0/256310973 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90003b70 con 0x7ffb98106500 2024-04-22T00:05:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.411+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb981a66f0 con 0x7ffb98106500 2024-04-22T00:05:40.406 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.411+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb981a6c10 con 0x7ffb98106500 2024-04-22T00:05:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.412+0000 7ffb94ff9640 1 -- 172.21.15.38:0/256310973 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb90003d10 con 0x7ffb98106500 2024-04-22T00:05:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.412+0000 7ffb94ff9640 1 -- 172.21.15.38:0/256310973 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90010a90 con 0x7ffb98106500 2024-04-22T00:05:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.416+0000 7ffba0de2740 1 Processor -- start 2024-04-22T00:05:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.416+0000 7ffba0de2740 1 -- start start 2024-04-22T00:05:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffb94ff9640 1 -- 172.21.15.38:0/256310973 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffb90010c30 con 0x7ffb98106500 2024-04-22T00:05:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49ed40 0x562c0f49f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c0f402b70 con 0x562c0f49ed40 2024-04-22T00:05:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c0f3ab470 con 0x562c0f49f660 2024-04-22T00:05:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffba0de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c0f3feb30 con 0x562c0f4b0c40 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.417+0000 7ffb94ff9640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 0x7ffb800a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.418+0000 7ffb94ff9640 1 -- 172.21.15.38:0/256310973 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ffb900cfbb0 con 0x7ffb98106500 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.418+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.418+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43848/0 (socket says 172.21.15.38:43848) 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.418+0000 7ffb977fe640 1 -- 172.21.15.38:0/3863925821 learned_addr learned my addr 172.21.15.38:0/3863925821 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 -- 172.21.15.38:0/3863925821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 msgr2=0x562c0f4b3030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb97fff640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 0x7ffb800a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 -- 172.21.15.38:0/3863925821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49ed40 msgr2=0x562c0f49f120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:05:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49ed40 0x562c0f49f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 -- 172.21.15.38:0/3863925821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c0f117210 con 0x562c0f49f660 2024-04-22T00:05:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.419+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009bc0 tx=0x7ffb8c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d74d4b68cb56ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.420+0000 7ffb97fff640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 0x7ffb800a53c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb90098190 tx=0x7ffb90011f30 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.422+0000 7ffb7dffb640 1 -- 172.21.15.38:0/3863925821 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c005020 con 0x562c0f49f660 2024-04-22T00:05:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.422+0000 7ffb7dffb640 1 -- 172.21.15.38:0/3863925821 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8c0047a0 con 0x562c0f49f660 2024-04-22T00:05:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.423+0000 7ffba0de2740 1 -- 172.21.15.38:0/3863925821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 msgr2=0x562c0f4b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.423+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009bc0 tx=0x7ffb8c00e720 comp rx=0 tx=0).stop 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 -- 172.21.15.38:0/3863925821 shutdown_connections 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49f660 0x562c0f4b0700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3863925821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49ed40 0x562c0f49f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 -- 172.21.15.38:0/3863925821 >> 172.21.15.38:0/3863925821 conn(0x562c0f499660 msgr2=0x562c0f499a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 -- 172.21.15.38:0/3863925821 shutdown_connections 2024-04-22T00:05:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.424+0000 7ffba0de2740 1 -- 172.21.15.38:0/3863925821 wait complete. 2024-04-22T00:05:40.419 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.425+0000 7ffba0de2740 1 -- start start 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.425+0000 7ffba0de2740 1 Processor -- start 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.425+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.425+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4aac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.425+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffba0de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c0f4a0190 con 0x562c0f49f660 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffba0de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c0f3bea10 con 0x562c0f49ed40 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffba0de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c0f4a0310 con 0x562c0f4b0c40 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffba0de2740 10 client.0 _ll_register_callbacks cb 0x562c0f3f9b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43862/0 (socket says 172.21.15.38:43862) 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 -- 172.21.15.38:0/3393740246 learned_addr learned my addr 172.21.15.38:0/3393740246 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4aac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 -- 172.21.15.38:0/3393740246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 msgr2=0x562c0f4b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 -- 172.21.15.38:0/3393740246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 msgr2=0x562c0f4aac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4aac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb96ffd640 1 -- 172.21.15.38:0/3393740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c0f1e3620 con 0x562c0f49ed40 2024-04-22T00:05:40.421 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.426+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4aac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.427+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffb8802c230 tx=0x7ffb88005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28162ff970f7370a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.427+0000 7ffb76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.427+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb88049060 con 0x562c0f49ed40 2024-04-22T00:05:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.427+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ffb88005050 con 0x562c0f49ed40 2024-04-22T00:05:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.429+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb8c0047a0 con 0x562c0f49ed40 2024-04-22T00:05:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.429+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffb88005250 con 0x562c0f49ed40 2024-04-22T00:05:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.431+0000 7ffba0de2740 10 client.26363 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:40.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.431+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562c0f4d9340 con 0x562c0f49ed40 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8c0047a0 con 0x562c0f49ed40 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7ffb88040040 con 0x562c0f49ed40 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffba0de2740 10 client.26363 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffba0de2740 10 client.26363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffba0de2740 20 client.26363 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.432+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f4dad10 0x562c0f4dd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.433+0000 7ffba0de2740 4 client.26363 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:40.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.433+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c0f4100b0 con 0x562c0f4dad10 2024-04-22T00:05:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.433+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f4dad10 0x562c0f4dd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.434+0000 7ffb977fe640 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f4dad10 0x562c0f4dd1b0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db0ee763f2ba8018 server_cookie=1056bc049b48f936 in_seq=0 out_seq=0 2024-04-22T00:05:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.434+0000 7ffb76ffd640 10 client.26363 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.434+0000 7ffb76ffd640 1 -- 172.21.15.38:0/3393740246 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c0f4100b0 con 0x562c0f4dad10 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.434+0000 7ffb76ffd640 10 client.26363 handle_command_reply: tid=0 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 client.26363 shutdown 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 msgr2=0x562c0f4aa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffb8802c230 tx=0x7ffb88005fd0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 shutdown_connections 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f4dad10 0x562c0f4dd1b0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4aac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 --2- 172.21.15.38:0/3393740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4aa6e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffb76ffd640 0 client.26363 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 >> 172.21.15.38:0/3393740246 conn(0x562c0f499660 msgr2=0x562c0f49b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 shutdown_connections 2024-04-22T00:05:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 1 -- 172.21.15.38:0/3393740246 wait complete. 2024-04-22T00:05:40.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.435+0000 7ffba0de2740 20 client.26363 trim_cache size 0 max 16384 2024-04-22T00:05:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.451+0000 7ffba0de2740 1 Processor -- start 2024-04-22T00:05:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.451+0000 7ffba0de2740 1 -- start start 2024-04-22T00:05:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4b0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.445 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.446 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4e2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb96ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43874/0 (socket says 172.21.15.38:43874) 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34448/0 (socket says 172.21.15.38:34448) 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb96ffd640 1 -- 172.21.15.38:0/1103528896 learned_addr learned my addr 172.21.15.38:0/1103528896 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8400fe80 con 0x562c0f49f660 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84011ab0 con 0x562c0f49ed40 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84010550 con 0x562c0f4b0c40 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4e2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 -- 172.21.15.38:0/1103528896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 msgr2=0x562c0f4e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4e2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 -- 172.21.15.38:0/1103528896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 msgr2=0x562c0f4b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4b0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.452+0000 7ffb977fe640 1 -- 172.21.15.38:0/1103528896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c0f4937a0 con 0x562c0f49f660 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.453+0000 7ffb977fe640 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x562c0f4a5340 tx=0x7ffb8c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8f4867bf7af29e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.453+0000 7ffb7d7fa640 1 -- 172.21.15.38:0/1103528896 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c027070 con 0x562c0f49f660 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.453+0000 7ffb7d7fa640 1 -- 172.21.15.38:0/1103528896 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8c004520 con 0x562c0f49f660 2024-04-22T00:05:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.453+0000 7ffb7d7fa640 1 -- 172.21.15.38:0/1103528896 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c01d040 con 0x562c0f49f660 2024-04-22T00:05:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 msgr2=0x562c0f4e2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x562c0f4a5340 tx=0x7ffb8c00ef70 comp rx=0 tx=0).stop 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 shutdown_connections 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f4b0c40 0x562c0f4e2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4e2a80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 --2- 172.21.15.38:0/1103528896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f49ed40 0x562c0f4b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 >> 172.21.15.38:0/1103528896 conn(0x562c0f499660 msgr2=0x562c0f499a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 shutdown_connections 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- 172.21.15.38:0/1103528896 wait complete. 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.454+0000 7ffba0de2740 1 -- start start 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 Processor -- start 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 0x562c0f4ee570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f4b0c40 0x562c0f4ec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84011320 con 0x562c0f49f660 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffb84010c50 con 0x562c0f4b0c40 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8400de60 con 0x562c0f49ed40 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffba0de2740 10 client.0 _ll_register_callbacks cb 0x7ffb8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34454/0 (socket says 172.21.15.38:34454) 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 -- 172.21.15.38:0/934444883 learned_addr learned my addr 172.21.15.38:0/934444883 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f4b0c40 0x562c0f4ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 0x562c0f4ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 -- 172.21.15.38:0/934444883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 msgr2=0x562c0f4ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 0x562c0f4ee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 -- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f4b0c40 msgr2=0x562c0f4ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f4b0c40 0x562c0f4ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb977fe640 1 -- 172.21.15.38:0/934444883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c0f53fde0 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.455+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 0x562c0f4ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb977fe640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00ec20 tx=0x7ffb8c010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f7beb4023600cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c027070 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ffb8c009530 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb8c004520 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffb8c009e70 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffba0de2740 10 client.16953 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562c0f540f20 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.456+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb8c004520 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7ffb8c0047a0 con 0x562c0f49f660 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 10 client.16953 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 10 client.16953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 20 client.16953 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f541520 0x562c0f543940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 4 client.16953 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c0f40a400 con 0x562c0f541520 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f541520 0x562c0f543940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffb76ffd640 10 client.16953 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.457+0000 7ffb96ffd640 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f541520 0x562c0f543940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ca81ebcf071036 server_cookie=33c43b5061140f2b in_seq=0 out_seq=0 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 -- 172.21.15.38:0/3954347502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac1066b0 msgr2=0x7f6fac106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3954347502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac1066b0 0x7f6fac106a90 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009930 tx=0x7f6f9c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 -- 172.21.15.38:0/3954347502 shutdown_connections 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3954347502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac100fd0 0x7f6fac10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3954347502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100610 0x7f6fac100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3954347502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac1066b0 0x7f6fac106a90 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 -- 172.21.15.38:0/3954347502 >> 172.21.15.38:0/3954347502 conn(0x7f6fac077740 msgr2=0x7f6fac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 -- 172.21.15.38:0/3954347502 shutdown_connections 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.490+0000 7f6fb267d640 1 -- 172.21.15.38:0/3954347502 wait complete. 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 Processor -- start 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 -- start start 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 0x7f6fac198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac1066b0 0x7f6fac19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1109f0 con 0x7f6fac100610 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac110870 con 0x7f6fac1066b0 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb267d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac110b70 con 0x7f6fac100fd0 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34474/0 (socket says 172.21.15.38:34474) 2024-04-22T00:05:40.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fabfff640 1 -- 172.21.15.38:0/3323902449 learned_addr learned my addr 172.21.15.38:0/3323902449 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 0x7f6fac198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.491+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac1066b0 0x7f6fac19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 -- 172.21.15.38:0/3323902449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 msgr2=0x7f6fac198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 0x7f6fac198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 -- 172.21.15.38:0/3323902449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac1066b0 msgr2=0x7f6fac19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac1066b0 0x7f6fac19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 -- 172.21.15.38:0/3323902449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac19cfb0 con 0x7f6fac100610 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 0x7f6fac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fabfff640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009230 tx=0x7f6f9c02cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fa97fa640 1 -- 172.21.15.38:0/3323902449 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c041070 con 0x7f6fac100610 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fa97fa640 1 -- 172.21.15.38:0/3323902449 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f9c004450 con 0x7f6fac100610 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac19d240 con 0x7f6fac100610 2024-04-22T00:05:40.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fa97fa640 1 -- 172.21.15.38:0/3323902449 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c003d00 con 0x7f6fac100610 2024-04-22T00:05:40.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.492+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac1a4b30 con 0x7f6fac100610 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.494+0000 7f6fa97fa640 1 -- 172.21.15.38:0/3323902449 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6f9c044070 con 0x7f6fac100610 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 Processor -- start 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 -- start start 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5abfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fa97fa640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 0x7f6f940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5620d5a23170 con 0x5620d5ac0720 2024-04-22T00:05:40.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5620d59cb7c0 con 0x5620d5ac8db0 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb5269740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5620d5a1f130 con 0x5620d5abfa40 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5abfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34480/0 (socket says 172.21.15.38:34480) 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 -- 172.21.15.38:0/1961159789 learned_addr learned my addr 172.21.15.38:0/1961159789 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 0x7f6f940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fa97fa640 1 -- 172.21.15.38:0/3323902449 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6f9c0f10b0 con 0x7f6fac100610 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 -- 172.21.15.38:0/1961159789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 msgr2=0x5620d5abfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5abfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 -- 172.21.15.38:0/1961159789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 msgr2=0x5620d5acb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.495+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6fb0c1a640 1 -- 172.21.15.38:0/1961159789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620d59dce60 con 0x5620d5ac0720 2024-04-22T00:05:40.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6fabfff640 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5abfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 0x7f6f940a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6fac069990 tx=0x7f6fa0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4013b70 tx=0x7f6fa4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4f33976a4d29b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6f927fc640 1 -- 172.21.15.38:0/1961159789 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4013130 con 0x5620d5ac0720 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6f927fc640 1 -- 172.21.15.38:0/1961159789 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa4013c60 con 0x5620d5ac0720 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.496+0000 7f6f927fc640 1 -- 172.21.15.38:0/1961159789 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4003d90 con 0x5620d5ac0720 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- 172.21.15.38:0/1961159789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 msgr2=0x5620d5abf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4013b70 tx=0x7f6fa4009900 comp rx=0 tx=0).stop 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- 172.21.15.38:0/1961159789 shutdown_connections 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5abfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 --2- 172.21.15.38:0/1961159789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5abf470 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- 172.21.15.38:0/1961159789 >> 172.21.15.38:0/1961159789 conn(0x5620d5a864f0 msgr2=0x5620d5aba490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- 172.21.15.38:0/1961159789 shutdown_connections 2024-04-22T00:05:40.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- 172.21.15.38:0/1961159789 wait complete. 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 -- start start 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.497+0000 7f6fb5269740 1 Processor -- start 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5abfa40 0x5620d5ac5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5ac8db0 0x5620d5ac6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5620d5acbbc0 con 0x5620d5ac0720 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5620d59ded40 con 0x5620d5abfa40 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5620d5acbd40 con 0x5620d5ac8db0 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 10 client.0 _ll_register_callbacks cb 0x5620d5a1a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34486/0 (socket says 172.21.15.38:34486) 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2096958543 learned_addr learned my addr 172.21.15.38:0/2096958543 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5abfa40 0x5620d5ac5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5ac8db0 0x5620d5ac6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2096958543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5ac8db0 msgr2=0x5620d5ac6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5ac8db0 0x5620d5ac6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5abfa40 msgr2=0x5620d5ac5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5abfa40 0x5620d5ac5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2096958543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620d57ff7e0 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0003760 tx=0x7f6fa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1fc9637f3e57c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6f8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000f840 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6fa000f9e0 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.498+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fa4013c60 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6fa000fbe0 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 10 client.16965 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5620d585ff70 con 0x5620d5ac0720 2024-04-22T00:05:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa4013c60 con 0x5620d5ac0720 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f6fa001a070 con 0x5620d5ac0720 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 10 client.16965 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 20 client.16965 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5ad8f60 0x5620d5adb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 4 client.16965 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620d59f7450 con 0x5620d5ad8f60 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.499+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5ad8f60 0x5620d5adb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.500+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5ad8f60 0x5620d5adb3d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f71d76a902409777 server_cookie=c11a04b821f2dbc0 in_seq=0 out_seq=0 2024-04-22T00:05:40.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.500+0000 7f6f8b7fe640 10 client.16965 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:40 smithi038 ceph-mon[27972]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 3.79k op/s 2024-04-22T00:05:40.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:40 smithi203 ceph-mon[31431]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 3.79k op/s 2024-04-22T00:05:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.821+0000 7ffb76ffd640 1 -- 172.21.15.38:0/934444883 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c0f40a400 con 0x562c0f541520 2024-04-22T00:05:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffb76ffd640 10 client.16953 handle_command_reply: tid=0 2024-04-22T00:05:40.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.822+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2096958543 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620d59f7450 con 0x5620d5ad8f60 2024-04-22T00:05:40.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.822+0000 7f6f8b7fe640 10 client.16965 handle_command_reply: tid=0 2024-04-22T00:05:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 client.16953 shutdown 2024-04-22T00:05:40.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.822+0000 7f6fb5269740 1 client.16965 shutdown 2024-04-22T00:05:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 msgr2=0x562c0f4eb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00ec20 tx=0x7ffb8c010830 comp rx=0 tx=0).stop 2024-04-22T00:05:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 shutdown_connections 2024-04-22T00:05:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c0f4b0c40 0x562c0f4ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c0f541520 0x562c0f543940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c0f49f660 0x562c0f4eb950 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 --2- 172.21.15.38:0/934444883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0f49ed40 0x562c0f4ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 >> 172.21.15.38:0/934444883 conn(0x562c0f499660 msgr2=0x562c0f49a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffb76ffd640 0 client.16953 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 shutdown_connections 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 1 -- 172.21.15.38:0/934444883 wait complete. 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.822+0000 7ffba0de2740 20 client.16953 trim_cache size 0 max 16384 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:05:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:05:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:05:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:05:40.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 msgr2=0x5620d5ac6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0003760 tx=0x7f6fa0004730 comp rx=0 tx=0).stop 2024-04-22T00:05:40.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 shutdown_connections 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5ac8db0 0x5620d5ac6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5ad8f60 0x5620d5adb3d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6f8b7fe640 0 client.16965 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5ac6300 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2096958543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5abfa40 0x5620d5ac5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 >> 172.21.15.38:0/2096958543 conn(0x5620d5a864f0 msgr2=0x5620d5abb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 msgr2=0x7ffb800a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 0x7ffb800a53c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb90098190 tx=0x7ffb90011f30 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 msgr2=0x7ffb980fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000efd0 tx=0x7ffb9000c6a0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 shutdown_connections 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffb800a2f00 0x7ffb800a53c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb98106500 0x7ffb980fcfb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb98100e20 0x7ffb980fca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 --2- 172.21.15.38:0/256310973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98100460 0x7ffb980fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.823+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 >> 172.21.15.38:0/256310973 conn(0x7ffb980fbf90 msgr2=0x7ffb98069f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 shutdown_connections 2024-04-22T00:05:40.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.823+0000 7f6fb5269740 1 -- 172.21.15.38:0/2096958543 wait complete. 2024-04-22T00:05:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.824+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 shutdown_connections 2024-04-22T00:05:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:40.824+0000 7ffb9e11e640 1 -- 172.21.15.38:0/256310973 wait complete. 2024-04-22T00:05:40.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.824+0000 7f6fb5269740 20 client.16965 trim_cache size 0 max 16384 2024-04-22T00:05:40.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:40 smithi111 ceph-mon[31543]: pgmap v110: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 3.79k op/s 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 Processor -- start 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 -- start start 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5ac84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac8db0 0x5620d5ad4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1aea80 con 0x5620d5ac8db0 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1b06b0 con 0x5620d5ac0720 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fb5269740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1af150 con 0x5620d5abfa40 2024-04-22T00:05:40.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5ac84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fab7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:43908/0 (socket says 172.21.15.38:43908) 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.850+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2151632152 learned_addr learned my addr 172.21.15.38:0/2151632152 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac8db0 0x5620d5ad4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2151632152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 msgr2=0x5620d5ac84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5ac84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2151632152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac8db0 msgr2=0x5620d5ad4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac8db0 0x5620d5ad4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 -- 172.21.15.38:0/2151632152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620d56f7170 con 0x5620d5ac0720 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5ac84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c730 tx=0x7f6fa000f500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44add144cc55d858 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6f91ffb640 1 -- 172.21.15.38:0/2151632152 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0038020 con 0x5620d5ac0720 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6f91ffb640 1 -- 172.21.15.38:0/2151632152 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa0011d20 con 0x5620d5ac0720 2024-04-22T00:05:40.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.851+0000 7f6f91ffb640 1 -- 172.21.15.38:0/2151632152 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00346c0 con 0x5620d5ac0720 2024-04-22T00:05:40.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 -- 172.21.15.38:0/2151632152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 msgr2=0x5620d5ac4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c730 tx=0x7f6fa000f500 comp rx=0 tx=0).stop 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 -- 172.21.15.38:0/2151632152 shutdown_connections 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac8db0 0x5620d5ad4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac0720 0x5620d5ac4d00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2151632152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5ac84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.852+0000 7f6fb5269740 1 -- 172.21.15.38:0/2151632152 >> 172.21.15.38:0/2151632152 conn(0x5620d5a864f0 msgr2=0x5620d5ab94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.853+0000 7f6fb5269740 1 -- 172.21.15.38:0/2151632152 shutdown_connections 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.853+0000 7f6fb5269740 1 -- 172.21.15.38:0/2151632152 wait complete. 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.853+0000 7f6fb5269740 1 -- start start 2024-04-22T00:05:40.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.853+0000 7f6fb5269740 1 Processor -- start 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5b38da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1aff20 con 0x5620d5ac0720 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1af850 con 0x5620d5ac8db0 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac1aca60 con 0x5620d5abfa40 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fabfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:49080/0 (socket says 172.21.15.38:49080) 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fabfff640 1 -- 172.21.15.38:0/2621150246 learned_addr learned my addr 172.21.15.38:0/2621150246 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb5269740 10 client.0 _ll_register_callbacks cb 0x7f6fa00035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5b38da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.854+0000 7f6fb0c1a640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 -- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 msgr2=0x5620d5acd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 -- 172.21.15.38:0/2621150246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 msgr2=0x5620d5b38da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5b38da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 -- 172.21.15.38:0/2621150246 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620d5acd900 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fabfff640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0ee680 tx=0x7f6f9c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c400e2111bf36574 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6f8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c04b040 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f6f9c0b9a10 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.855+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fa0011d20 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6f9c0035f0 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 10 client.26239 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5620d5b400a0 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa0011d20 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7f6f9c0d4080 con 0x5620d5abfa40 2024-04-22T00:05:40.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 10 client.26239 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 20 client.26239 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.856+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5b40830 0x5620d5b42c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.857+0000 7f6fb5269740 4 client.26239 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.857+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5620d5a306b0 con 0x5620d5b40830 2024-04-22T00:05:40.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.857+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5b40830 0x5620d5b42c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.858+0000 7f6fab7fe640 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5b40830 0x5620d5b42c50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b23b61d7e9b1e8 server_cookie=9790ff02826a653c in_seq=0 out_seq=0 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.858+0000 7f6f8b7fe640 10 client.26239 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6f8b7fe640 1 -- 172.21.15.38:0/2621150246 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5620d5a306b0 con 0x5620d5b40830 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6f8b7fe640 10 client.26239 handle_command_reply: tid=0 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6fb5269740 1 client.26239 shutdown 2024-04-22T00:05:40.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 msgr2=0x5620d5b38860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0ee680 tx=0x7f6f9c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:05:40.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 shutdown_connections 2024-04-22T00:05:40.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.859+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5620d5ac8db0 0x5620d5acd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5620d5b40830 0x5620d5b42c50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6f8b7fe640 0 client.26239 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5620d5ac0720 0x5620d5b38da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 --2- 172.21.15.38:0/2621150246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620d5abfa40 0x5620d5b38860 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 >> 172.21.15.38:0/2621150246 conn(0x5620d5a864f0 msgr2=0x5620d5ac5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 shutdown_connections 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 1 -- 172.21.15.38:0/2621150246 wait complete. 2024-04-22T00:05:40.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.860+0000 7f6fb5269740 20 client.26239 trim_cache size 0 max 16384 2024-04-22T00:05:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:40.861 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:05:40.861 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "f738553f-79b4-4c14-9209-c927a2e50605", 2024-04-22T00:05:40.862 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:05:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:40.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 msgr2=0x7f6f940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 0x7f6f940a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6fac069990 tx=0x7f6fa0005f90 comp rx=0 tx=0).stop 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 msgr2=0x7f6fac197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009230 tx=0x7f6f9c02cb20 comp rx=0 tx=0).stop 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 shutdown_connections 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f6f940a2d00 0x7f6f940a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6fac1066b0 0x7f6fac19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fac100fd0 0x7f6fac198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 --2- 172.21.15.38:0/3323902449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fac100610 0x7f6fac197fc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:40.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.861+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 >> 172.21.15.38:0/3323902449 conn(0x7f6fac077740 msgr2=0x7f6fac104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:40.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.862+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 shutdown_connections 2024-04-22T00:05:40.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:40.862+0000 7f6fb267d640 1 -- 172.21.15.38:0/3323902449 wait complete. 2024-04-22T00:05:40.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:05:40.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:05:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.054+0000 7ff7282e2640 1 -- 172.21.15.38:0/1050447404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720114890 msgr2=0x7ff720116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.054+0000 7ff7282e2640 1 --2- 172.21.15.38:0/1050447404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720114890 0x7ff720116c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff71000fb10 tx=0x7ff71002fdd0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.055+0000 7ff7282e2640 1 -- 172.21.15.38:0/1050447404 shutdown_connections 2024-04-22T00:05:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.055+0000 7ff7282e2640 1 --2- 172.21.15.38:0/1050447404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720114890 0x7ff720116c80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.055+0000 7ff7282e2640 1 --2- 172.21.15.38:0/1050447404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720109270 0x7ff720114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.055+0000 7ff7282e2640 1 --2- 172.21.15.38:0/1050447404 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff720108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.055+0000 7ff7282e2640 1 -- 172.21.15.38:0/1050447404 >> 172.21.15.38:0/1050447404 conn(0x7ff72006c850 msgr2=0x7ff72006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.056+0000 7ff7282e2640 1 -- 172.21.15.38:0/1050447404 shutdown_connections 2024-04-22T00:05:41.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.056+0000 7ff7282e2640 1 -- 172.21.15.38:0/1050447404 wait complete. 2024-04-22T00:05:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.057+0000 7ff7282e2640 1 Processor -- start 2024-04-22T00:05:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.057+0000 7ff7282e2640 1 -- start start 2024-04-22T00:05:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff7201a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720109270 0x7ff7201a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 0x7ff7201ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201181c0 con 0x7ff720108950 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff720118040 con 0x7ff720109270 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff7282e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff720118340 con 0x7ff720114890 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72687f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 0x7ff7201ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72687f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 0x7ff7201ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:49100/0 (socket says 172.21.15.38:49100) 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff7201a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72687f640 1 -- 172.21.15.38:0/2894611110 learned_addr learned my addr 172.21.15.38:0/2894611110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 -- 172.21.15.38:0/2894611110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 msgr2=0x7ff7201ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 0x7ff7201ab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 -- 172.21.15.38:0/2894611110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720109270 msgr2=0x7ff7201a73f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:05:41.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720109270 0x7ff7201a73f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.058+0000 7ff72607e640 1 -- 172.21.15.38:0/2894611110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7201abea0 con 0x7ff720108950 2024-04-22T00:05:41.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.059+0000 7ff72607e640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff7201a6eb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff71800ec80 tx=0x7ff71800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.060+0000 7ff7177fe640 1 -- 172.21.15.38:0/2894611110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718003b70 con 0x7ff720108950 2024-04-22T00:05:41.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.060+0000 7ff7177fe640 1 -- 172.21.15.38:0/2894611110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff718003d10 con 0x7ff720108950 2024-04-22T00:05:41.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.060+0000 7ff7177fe640 1 -- 172.21.15.38:0/2894611110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718010a30 con 0x7ff720108950 2024-04-22T00:05:41.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.060+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7201ac190 con 0x7ff720108950 2024-04-22T00:05:41.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.060+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7201a0400 con 0x7ff720108950 2024-04-22T00:05:41.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.064+0000 7ff72afa9740 1 Processor -- start 2024-04-22T00:05:41.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.066+0000 7ff7177fe640 1 -- 172.21.15.38:0/2894611110 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff718003ec0 con 0x7ff720108950 2024-04-22T00:05:41.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.067+0000 7ff7177fe640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 0x7ff70c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.068+0000 7ff7177fe640 1 -- 172.21.15.38:0/2894611110 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff7180d00a0 con 0x7ff720108950 2024-04-22T00:05:41.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.069+0000 7ff72afa9740 1 -- start start 2024-04-22T00:05:41.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 0x55994abd9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55994ab32560 con 0x55994abce850 2024-04-22T00:05:41.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55994aada730 con 0x55994abd7430 2024-04-22T00:05:41.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.070+0000 7ff72afa9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55994ab2e520 con 0x55994abcdf30 2024-04-22T00:05:41.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72687f640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 0x7ff70c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72587d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72587d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34542/0 (socket says 172.21.15.38:34542) 2024-04-22T00:05:41.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72587d640 1 -- 172.21.15.38:0/984979594 learned_addr learned my addr 172.21.15.38:0/984979594 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:41.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72687f640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 0x55994abd9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.071+0000 7ff72607e640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72587d640 1 -- 172.21.15.38:0/984979594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 msgr2=0x55994abce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72687f640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 0x7ff70c0a5390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff710005ab0 tx=0x7ff71000f5c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72587d640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72587d640 1 -- 172.21.15.38:0/984979594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 msgr2=0x55994abd9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72587d640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 0x55994abd9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72587d640 1 -- 172.21.15.38:0/984979594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55994a9107d0 con 0x55994abce850 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72687f640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 0x55994abd9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.072+0000 7ff72607e640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff72587d640 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b9e0 tx=0x7ff71c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e33f517d1728ec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff714bfe640 1 -- 172.21.15.38:0/984979594 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c016070 con 0x55994abce850 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff714bfe640 1 -- 172.21.15.38:0/984979594 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff71c005500 con 0x55994abce850 2024-04-22T00:05:41.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff714bfe640 1 -- 172.21.15.38:0/984979594 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c004d40 con 0x55994abce850 2024-04-22T00:05:41.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff72afa9740 1 -- 172.21.15.38:0/984979594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 msgr2=0x55994abd6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.073+0000 7ff72afa9740 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00b9e0 tx=0x7ff71c00bea0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- 172.21.15.38:0/984979594 shutdown_connections 2024-04-22T00:05:41.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abd7430 0x55994abd9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abce850 0x55994abd6ef0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- 172.21.15.38:0/984979594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- 172.21.15.38:0/984979594 >> 172.21.15.38:0/984979594 conn(0x55994abc8660 msgr2=0x55994abc8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- 172.21.15.38:0/984979594 shutdown_connections 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- 172.21.15.38:0/984979594 wait complete. 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- start start 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 Processor -- start 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abd4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55994abda220 con 0x55994abd7430 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55994aaede00 con 0x55994abce850 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.074+0000 7ff72afa9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55994abda3a0 con 0x55994abcdf30 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72afa9740 10 client.0 _ll_register_callbacks cb 0x55994ab28eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72607e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34554/0 (socket says 172.21.15.38:34554) 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 -- 172.21.15.38:0/3702901117 learned_addr learned my addr 172.21.15.38:0/3702901117 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72607e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:49108/0 (socket says 172.21.15.38:49108) 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72587d640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abd4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 -- 172.21.15.38:0/3702901117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 msgr2=0x55994abd44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 -- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 msgr2=0x55994abd4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abd4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 -- 172.21.15.38:0/3702901117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55994a8463f0 con 0x55994abd7430 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72587d640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abd4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72607e640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff72687f640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff71000f860 tx=0x7ff71002fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f42f0478abc50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff709ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71003d040 con 0x55994abd7430 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.075+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff710004020 con 0x55994abd7430 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff71c005500 con 0x55994abd7430 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff71005a050 con 0x55994abd7430 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 10 client.16983 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:41.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55994a8060f0 con 0x55994abd7430 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff71c005500 con 0x55994abd7430 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7ff7100035f0 con 0x55994abd7430 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 10 client.16983 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 20 client.16983 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994abe78b0 0x55994abe9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 4 client.16983 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55994ab3faa0 con 0x55994abe78b0 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.076+0000 7ff72607e640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994abe78b0 0x55994abe9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.077+0000 7ff709ffb640 10 client.16983 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.077+0000 7ff72607e640 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994abe78b0 0x55994abe9d30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f150873df688120 server_cookie=e510714ca0bea2b6 in_seq=0 out_seq=0 2024-04-22T00:05:41.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.077+0000 7ff709ffb640 1 -- 172.21.15.38:0/3702901117 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55994ab3faa0 con 0x55994abe78b0 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.077+0000 7ff709ffb640 10 client.16983 handle_command_reply: tid=0 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 client.16983 shutdown 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 msgr2=0x55994abd4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff71000f860 tx=0x7ff71002fdd0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 shutdown_connections 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994abe78b0 0x55994abe9d30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abd4f60 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abd4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff709ffb640 0 client.16983 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:41.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 --2- 172.21.15.38:0/3702901117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abcdf30 0x55994abd44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 >> 172.21.15.38:0/3702901117 conn(0x55994abc8660 msgr2=0x55994abca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 shutdown_connections 2024-04-22T00:05:41.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 1 -- 172.21.15.38:0/3702901117 wait complete. 2024-04-22T00:05:41.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.078+0000 7ff72afa9740 20 client.16983 trim_cache size 0 max 16384 2024-04-22T00:05:41.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 Processor -- start 2024-04-22T00:05:41.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 -- start start 2024-04-22T00:05:41.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 0x55994abd6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abce850 0x55994abe30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201c6160 con 0x55994abd7430 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201c7d90 con 0x55994abcdf30 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72afa9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201c6830 con 0x55994abce850 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34570/0 (socket says 172.21.15.38:34570) 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 -- 172.21.15.38:0/500478791 learned_addr learned my addr 172.21.15.38:0/500478791 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72607e640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 0x55994abd6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72587d640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abce850 0x55994abe30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 -- 172.21.15.38:0/500478791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abce850 msgr2=0x55994abe30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abce850 0x55994abe30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 -- 172.21.15.38:0/500478791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 msgr2=0x55994abd6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 0x55994abd6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 -- 172.21.15.38:0/500478791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55994abd9820 con 0x55994abd7430 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72607e640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 0x55994abd6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.096+0000 7ff72687f640 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff710003a90 tx=0x7ff710030c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0dad6edbb7c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff70bfff640 1 -- 172.21.15.38:0/500478791 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710062070 con 0x55994abd7430 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff70bfff640 1 -- 172.21.15.38:0/500478791 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710049040 con 0x55994abd7430 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff70bfff640 1 -- 172.21.15.38:0/500478791 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710004160 con 0x55994abd7430 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 -- 172.21.15.38:0/500478791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 msgr2=0x55994abc8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff710003a90 tx=0x7ff710030c00 comp rx=0 tx=0).stop 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 -- 172.21.15.38:0/500478791 shutdown_connections 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abd7430 0x55994abc8a40 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abce850 0x55994abe30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 --2- 172.21.15.38:0/500478791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abcdf30 0x55994abd6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 -- 172.21.15.38:0/500478791 >> 172.21.15.38:0/500478791 conn(0x55994abc8660 msgr2=0x55994abc6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 -- 172.21.15.38:0/500478791 shutdown_connections 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.097+0000 7ff72afa9740 1 -- 172.21.15.38:0/500478791 wait complete. 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 -- start start 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 Processor -- start 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abdfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abd7430 0x55994abd0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201c7600 con 0x55994abcdf30 2024-04-22T00:05:41.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201c6f30 con 0x55994abce850 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7201a6800 con 0x55994abd7430 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72afa9740 10 client.0 _ll_register_callbacks cb 0x7ff71000f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34580/0 (socket says 172.21.15.38:34580) 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 -- 172.21.15.38:0/1132305808 learned_addr learned my addr 172.21.15.38:0/1132305808 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72587d640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abdfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72687f640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abd7430 0x55994abd0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 -- 172.21.15.38:0/1132305808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abd7430 msgr2=0x55994abd0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abd7430 0x55994abd0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 -- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 msgr2=0x55994abdfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abdfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.098+0000 7ff72607e640 1 -- 172.21.15.38:0/1132305808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55994abd1530 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72607e640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff71800cba0 tx=0x7ff71800ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c9e20f7d0b04f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7180b4070 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff718004620 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff710049040 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff7180bae60 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 10 client.16989 fetch_fsmap learned FSMap version 33 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55994ac4dbb0 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff710049040 con 0x55994abcdf30 2024-04-22T00:05:41.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) ==== 8382+0+0 (secure 0 0 0) 0x7ff7180aa020 con 0x55994abcdf30 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 10 client.16989 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 20 client.16989 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994ac4e3a0 0x55994ac507c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 4 client.16989 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:05:41.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.099+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55994ab39df0 con 0x55994ac4e3a0 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72587d640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994ac4e3a0 0x55994ac507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff709ffb640 10 client.16989 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72587d640 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994ac4e3a0 0x55994ac507c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18eaadafcab7a016 server_cookie=65c62e5254ff0a57 in_seq=0 out_seq=0 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff709ffb640 1 -- 172.21.15.38:0/1132305808 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55994ab39df0 con 0x55994ac4e3a0 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff709ffb640 10 client.16989 handle_command_reply: tid=0 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 client.16989 shutdown 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 msgr2=0x55994abdf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff71800cba0 tx=0x7ff71800ef80 comp rx=0 tx=0).stop 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 shutdown_connections 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55994abd7430 0x55994abd0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55994abce850 0x55994abdfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55994ac4e3a0 0x55994ac507c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 --2- 172.21.15.38:0/1132305808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55994abcdf30 0x55994abdf5f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 >> 172.21.15.38:0/1132305808 conn(0x55994abc8660 msgr2=0x55994abd4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff709ffb640 0 client.16989 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 shutdown_connections 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 1 -- 172.21.15.38:0/1132305808 wait complete. 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.100+0000 7ff72afa9740 20 client.16989 trim_cache size 0 max 16384 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (134 inodes in the stack)", 2024-04-22T00:05:41.104 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: "f738553f-79b4-4c14-9209-c927a2e50605": { 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "f738553f-79b4-4c14-9209-c927a2e50605", 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:05:41.105 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 msgr2=0x7ff70c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 0x7ff70c0a5390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff710005ab0 tx=0x7ff71000f5c0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 msgr2=0x7ff7201a6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff7201a6eb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ff71800ec80 tx=0x7ff71800c6a0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 shutdown_connections 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff70c0a2ed0 0x7ff70c0a5390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff720114890 0x7ff7201ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff720109270 0x7ff7201a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 --2- 172.21.15.38:0/2894611110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff720108950 0x7ff7201a6eb0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 >> 172.21.15.38:0/2894611110 conn(0x7ff72006c850 msgr2=0x7ff720070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 shutdown_connections 2024-04-22T00:05:41.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:05:41.101+0000 7ff7282e2640 1 -- 172.21.15.38:0/2894611110 wait complete. 2024-04-22T00:05:41.108 INFO:tasks.cephfs.filesystem:scrub status for tag:f738553f-79b4-4c14-9209-c927a2e50605 - {'path': '/', 'tag': 'f738553f-79b4-4c14-9209-c927a2e50605', 'options': 'recursive,force'} 2024-04-22T00:05:42.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:42 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:05:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:42 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:42 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:43.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:42 smithi038 ceph-mon[27972]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 3.58k op/s 2024-04-22T00:05:43.562 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:42 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:05:43.562 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:42 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:42 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:42 smithi203 ceph-mon[31431]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 3.58k op/s 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:42 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:42 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:42 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:42 smithi111 ceph-mon[31543]: pgmap v111: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 3.58k op/s 2024-04-22T00:05:43.563 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:05:42 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:05:42.770+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v252>,<0x100000001fb/client.0 v237>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v244>,<0x10000000001/sv_1 v266>,<0x10000000000/qa v246>,<0x1/volumes v236>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:05:43.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:43 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:43.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:43 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:43.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:43 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:43.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:43 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:43 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:05:43.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:43 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 22 MiB/s wr, 4.59k op/s 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:44 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 22 MiB/s wr, 4.59k op/s 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.729 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:44 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: pgmap v112: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 22 MiB/s wr, 4.59k op/s 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:05:44.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:44.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:44 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:05:46.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:46 smithi038 ceph-mon[27972]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 18 MiB/s wr, 3.71k op/s 2024-04-22T00:05:46.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:46 smithi203 ceph-mon[31431]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 18 MiB/s wr, 3.71k op/s 2024-04-22T00:05:46.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:46 smithi111 ceph-mon[31543]: pgmap v113: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 18 MiB/s wr, 3.71k op/s 2024-04-22T00:05:48.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:48 smithi038 ceph-mon[27972]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 4.33k op/s 2024-04-22T00:05:48.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:48 smithi203 ceph-mon[31431]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 4.33k op/s 2024-04-22T00:05:48.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:48 smithi111 ceph-mon[31543]: pgmap v114: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 4.33k op/s 2024-04-22T00:05:50.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:50 smithi038 ceph-mon[27972]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.77k op/s 2024-04-22T00:05:50.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:50 smithi203 ceph-mon[31431]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.77k op/s 2024-04-22T00:05:50.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:50 smithi111 ceph-mon[31543]: pgmap v115: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.77k op/s 2024-04-22T00:05:52.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:52 smithi038 ceph-mon[27972]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 2.53k op/s 2024-04-22T00:05:52.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:52 smithi203 ceph-mon[31431]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 2.53k op/s 2024-04-22T00:05:52.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:52 smithi111 ceph-mon[31543]: pgmap v116: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 2.53k op/s 2024-04-22T00:05:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1710170874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844100180 msgr2=0x7fe8441005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1710170874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844100180 0x7fe8441005e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe834009930 tx=0x7fe83402bc40 comp rx=0 tx=0).stop 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1710170874 shutdown_connections 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1710170874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 0x7fe8441107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1710170874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844100180 0x7fe8441005e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1710170874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844102840 0x7fe8440ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1710170874 >> 172.21.15.38:0/1710170874 conn(0x7fe844076bd0 msgr2=0x7fe8440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1710170874 shutdown_connections 2024-04-22T00:05:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.955+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1710170874 wait complete. 2024-04-22T00:05:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 Processor -- start 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 -- start start 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 0x7fe84419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 0x7fe84419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe844114000 con 0x7fe84410e3c0 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe844113e80 con 0x7fe844100180 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.956+0000 7fe84ba9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe844114180 con 0x7fe844102840 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:59440/0 (socket says 172.21.15.38:59440) 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 -- 172.21.15.38:0/1457834441 learned_addr learned my addr 172.21.15.38:0/1457834441 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 0x7fe84419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849038640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 0x7fe84419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 -- 172.21.15.38:0/1457834441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 msgr2=0x7fe84419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 0x7fe84419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 -- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 msgr2=0x7fe84419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 0x7fe84419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 -- 172.21.15.38:0/1457834441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84419f470 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849038640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 0x7fe84419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.957+0000 7fe849839640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe84000b450 tx=0x7fe84000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.958+0000 7fe83affd640 1 -- 172.21.15.38:0/1457834441 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840015020 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.958+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84419f760 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.958+0000 7fe83affd640 1 -- 172.21.15.38:0/1457834441 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe840011d70 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.958+0000 7fe83affd640 1 -- 172.21.15.38:0/1457834441 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84000f340 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.958+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8441a6ff0 con 0x7fe844100180 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.960+0000 7fe84e763740 1 Processor -- start 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 -- start start 2024-04-22T00:05:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564c76636b70 con 0x564c766d2b20 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c765df470 con 0x564c766dc210 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84e763740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c76632b30 con 0x564c766d3440 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe83affd640 1 -- 172.21.15.38:0/1457834441 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe840018020 con 0x7fe844100180 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:59448/0 (socket says 172.21.15.38:59448) 2024-04-22T00:05:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 -- 172.21.15.38:0/2602200691 learned_addr learned my addr 172.21.15.38:0/2602200691 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42712/0 (socket says 172.21.15.38:42712) 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849839640 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 -- 172.21.15.38:0/2602200691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 msgr2=0x564c766dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766dbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 -- 172.21.15.38:0/2602200691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 msgr2=0x564c766d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.961+0000 7fe849038640 1 -- 172.21.15.38:0/2602200691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c7634b210 con 0x564c766dc210 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.962+0000 7fe849038640 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe834005e50 tx=0x7fe834005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb81c50ebb5cc2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.962+0000 7fe82ffff640 1 -- 172.21.15.38:0/2602200691 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83402ca90 con 0x564c766dc210 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.962+0000 7fe82ffff640 1 -- 172.21.15.38:0/2602200691 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83402cc30 con 0x564c766dc210 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.962+0000 7fe82ffff640 1 -- 172.21.15.38:0/2602200691 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834004920 con 0x564c766dc210 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.962+0000 7fe83affd640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 0x7fe8300a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 0x7fe8300a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe83affd640 1 -- 172.21.15.38:0/1457834441 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe8400cf120 con 0x7fe844100180 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- 172.21.15.38:0/2602200691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 msgr2=0x564c766de600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe834005e50 tx=0x7fe834005e80 comp rx=0 tx=0).stop 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- 172.21.15.38:0/2602200691 shutdown_connections 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766de600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766dbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 --2- 172.21.15.38:0/2602200691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- 172.21.15.38:0/2602200691 >> 172.21.15.38:0/2602200691 conn(0x564c766cd440 msgr2=0x564c766cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 0x7fe8300a52e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00a400 tx=0x7fe83c00d040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- 172.21.15.38:0/2602200691 shutdown_connections 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- 172.21.15.38:0/2602200691 wait complete. 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 -- start start 2024-04-22T00:05:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.963+0000 7fe84e763740 1 Processor -- start 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766d96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564c766dede0 con 0x564c766d2b20 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564c765f2a10 con 0x564c766dc210 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c766def60 con 0x564c766d3440 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84e763740 10 client.0 _ll_register_callbacks cb 0x564c7662db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe849839640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe849038640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766d96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:59452/0 (socket says 172.21.15.38:59452) 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe849038640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766d96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:33326/0 (socket says 172.21.15.38:33326) 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe849839640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42718/0 (socket says 172.21.15.38:42718) 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.964+0000 7fe84a03a640 1 -- 172.21.15.38:0/2207575516 learned_addr learned my addr 172.21.15.38:0/2207575516 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 -- 172.21.15.38:0/2207575516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 msgr2=0x564c766d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766d96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 -- 172.21.15.38:0/2207575516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 msgr2=0x564c766d91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 -- 172.21.15.38:0/2207575516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c76417620 con 0x564c766dc210 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84a03a640 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe83c018ed0 tx=0x7fe83c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b96f9b15b6ab1d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe82d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c028060 con 0x564c766dc210 2024-04-22T00:05:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe83c0177d0 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83402cc30 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe83c02f050 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84e763740 10 client.26411 fetch_fsmap learned FSMap version 34 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564c766eb040 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.965+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83402cc30 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fe83c023a10 con 0x564c766dc210 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 10 client.26411 fetch_fsmap finished waiting for FSMap version 34 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 10 client.26411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 20 client.26411 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c766ec550 0x564c766ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 4 client.26411 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c766440b0 con 0x564c766ec550 2024-04-22T00:05:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe849839640 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c766ec550 0x564c766ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe82d7fa640 10 client.26411 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.966+0000 7fe849839640 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c766ec550 0x564c766ee9d0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36ba1fc81011433f server_cookie=76c39250686bcf6 in_seq=0 out_seq=0 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe82d7fa640 1 -- 172.21.15.38:0/2207575516 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c766440b0 con 0x564c766ec550 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe82d7fa640 10 client.26411 handle_command_reply: tid=0 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 client.26411 shutdown 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 msgr2=0x564c766d9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe83c018ed0 tx=0x7fe83c004a40 comp rx=0 tx=0).stop 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 shutdown_connections 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766dc210 0x564c766d9c30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c766ec550 0x564c766ee9d0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe82d7fa640 0 client.26411 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 --2- 172.21.15.38:0/2207575516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d2b20 0x564c766d91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 >> 172.21.15.38:0/2207575516 conn(0x564c766cd440 msgr2=0x564c766cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 shutdown_connections 2024-04-22T00:05:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.967+0000 7fe84e763740 1 -- 172.21.15.38:0/2207575516 wait complete. 2024-04-22T00:05:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.968+0000 7fe84e763740 20 client.26411 trim_cache size 0 max 16384 2024-04-22T00:05:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.996+0000 7fe84e763740 1 Processor -- start 2024-04-22T00:05:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.996+0000 7fe84e763740 1 -- start start 2024-04-22T00:05:53.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766db8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d3440 0x564c766e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:53.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441ac8e0 con 0x564c766d3440 2024-04-22T00:05:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441ae4e0 con 0x564c766d2b20 2024-04-22T00:05:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84e763740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441acfb0 con 0x564c766dc210 2024-04-22T00:05:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:33332/0 (socket says 172.21.15.38:33332) 2024-04-22T00:05:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:53.997+0000 7fe84a03a640 1 -- 172.21.15.38:0/1129224398 learned_addr learned my addr 172.21.15.38:0/1129224398 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.000+0000 7fe84a03a640 1 -- 172.21.15.38:0/1129224398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 msgr2=0x564c766db8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:05:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.000+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766db8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.000+0000 7fe84a03a640 1 -- 172.21.15.38:0/1129224398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d3440 msgr2=0x564c766e7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.000+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d3440 0x564c766e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.000+0000 7fe84a03a640 1 -- 172.21.15.38:0/1129224398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c766de930 con 0x564c766dc210 2024-04-22T00:05:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.001+0000 7fe84a03a640 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe83c023de0 tx=0x7fe83c018be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884f70a38d9c9120 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.004+0000 7fe82f7fe640 1 -- 172.21.15.38:0/1129224398 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c037070 con 0x564c766dc210 2024-04-22T00:05:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.004+0000 7fe82f7fe640 1 -- 172.21.15.38:0/1129224398 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c01f040 con 0x564c766dc210 2024-04-22T00:05:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.004+0000 7fe82f7fe640 1 -- 172.21.15.38:0/1129224398 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c017a30 con 0x564c766dc210 2024-04-22T00:05:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.005+0000 7fe84e763740 1 -- 172.21.15.38:0/1129224398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 msgr2=0x564c766f42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.005+0000 7fe84e763740 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe83c023de0 tx=0x7fe83c018be0 comp rx=0 tx=0).stop 2024-04-22T00:05:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.006+0000 7fe84e763740 1 -- 172.21.15.38:0/1129224398 shutdown_connections 2024-04-22T00:05:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.006+0000 7fe84e763740 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766dc210 0x564c766f42e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.006+0000 7fe84e763740 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766d3440 0x564c766e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.006+0000 7fe84e763740 1 --2- 172.21.15.38:0/1129224398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766db8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.006+0000 7fe84e763740 1 -- 172.21.15.38:0/1129224398 >> 172.21.15.38:0/1129224398 conn(0x564c766cd440 msgr2=0x564c766d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.007+0000 7fe84e763740 1 -- 172.21.15.38:0/1129224398 shutdown_connections 2024-04-22T00:05:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.007+0000 7fe84e763740 1 -- 172.21.15.38:0/1129224398 wait complete. 2024-04-22T00:05:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 -- start start 2024-04-22T00:05:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 Processor -- start 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766e02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766e07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441add50 con 0x564c766dc210 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441ad680 con 0x564c766d2b20 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441aa8e0 con 0x564c766d3440 2024-04-22T00:05:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.008+0000 7fe84e763740 10 client.0 _ll_register_callbacks cb 0x7fe83c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:05:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.009+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.009+0000 7fe84a03a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42740/0 (socket says 172.21.15.38:42740) 2024-04-22T00:05:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.009+0000 7fe84a03a640 1 -- 172.21.15.38:0/594946771 learned_addr learned my addr 172.21.15.38:0/594946771 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe849839640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766e02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 -- 172.21.15.38:0/594946771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 msgr2=0x564c766e07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766e07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 -- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 msgr2=0x564c766e02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766e02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 -- 172.21.15.38:0/594946771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c767630a0 con 0x564c766dc210 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.010+0000 7fe84a03a640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe83c018ed0 tx=0x7fe83c009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90c034dc32cfb9d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.011+0000 7fe82d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.011+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c037050 con 0x564c766dc210 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.011+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83c01f040 con 0x564c766dc210 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.011+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe83c023880 con 0x564c766dc210 2024-04-22T00:05:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.011+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe83c023a80 con 0x564c766dc210 2024-04-22T00:05:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.012+0000 7fe84e763740 10 client.17001 fetch_fsmap learned FSMap version 34 2024-04-22T00:05:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.012+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564c76764320 con 0x564c766dc210 2024-04-22T00:05:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.016+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c01f040 con 0x564c766dc210 2024-04-22T00:05:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.016+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fe83c03c790 con 0x564c766dc210 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 10 client.17001 fetch_fsmap finished waiting for FSMap version 34 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 10 client.17001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 20 client.17001 populate_metadata read hostname 'smithi038' 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c767649d0 0x564c76766df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 4 client.17001 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:05:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.020+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c7663e400 con 0x564c767649d0 2024-04-22T00:05:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.023+0000 7fe849839640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c767649d0 0x564c76766df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:05:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.027+0000 7fe849839640 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c767649d0 0x564c76766df0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3291d32d3d64eb5 server_cookie=b08bf46026848608 in_seq=0 out_seq=0 2024-04-22T00:05:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.027+0000 7fe82d7fa640 10 client.17001 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe82d7fa640 1 -- 172.21.15.38:0/594946771 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564c7663e400 con 0x564c767649d0 2024-04-22T00:05:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe82d7fa640 10 client.17001 handle_command_reply: tid=0 2024-04-22T00:05:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 client.17001 shutdown 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 msgr2=0x564c76751b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe83c018ed0 tx=0x7fe83c009a40 comp rx=0 tx=0).stop 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 shutdown_connections 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564c767649d0 0x564c76766df0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.419+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564c766dc210 0x564c76751b50 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c766d3440 0x564c766e07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 1 --2- 172.21.15.38:0/594946771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c766d2b20 0x564c766e02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 >> 172.21.15.38:0/594946771 conn(0x564c766cd440 msgr2=0x564c766d9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe82d7fa640 0 client.17001 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:05:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 shutdown_connections 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 1 -- 172.21.15.38:0/594946771 wait complete. 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.420+0000 7fe84e763740 20 client.17001 trim_cache size 0 max 16384 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:05:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 msgr2=0x7fe8300a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 0x7fe8300a52e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00a400 tx=0x7fe83c00d040 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 msgr2=0x7fe84419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe84000b450 tx=0x7fe84000b910 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 shutdown_connections 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8300a2e20 0x7fe8300a52e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe84410e3c0 0x7fe84419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe844102840 0x7fe84419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 --2- 172.21.15.38:0/1457834441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe844100180 0x7fe84419a480 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:05:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.421+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 >> 172.21.15.38:0/1457834441 conn(0x7fe844076bd0 msgr2=0x7fe84410f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:05:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.423+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 shutdown_connections 2024-04-22T00:05:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:05:54.423+0000 7fe84ba9d640 1 -- 172.21.15.38:0/1457834441 wait complete. 2024-04-22T00:05:54.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:54 smithi038 ceph-mon[27972]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 16 MiB/s wr, 3.63k op/s 2024-04-22T00:05:54.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:54 smithi203 ceph-mon[31431]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 16 MiB/s wr, 3.63k op/s 2024-04-22T00:05:54.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:54 smithi111 ceph-mon[31543]: pgmap v117: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 16 MiB/s wr, 3.63k op/s 2024-04-22T00:05:56.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:56 smithi038 ceph-mon[27972]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:05:56.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:56 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:56 smithi203 ceph-mon[31431]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:05:56.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:56 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:56.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:56 smithi111 ceph-mon[31543]: pgmap v118: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:05:56.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:56 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:57.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:05:57.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:57.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:57.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:57.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:05:57.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:57.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:57.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:57.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:05:58.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:05:58 smithi038 ceph-mon[27972]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:05:58.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:05:58 smithi203 ceph-mon[31431]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:05:58.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:05:58 smithi111 ceph-mon[31543]: pgmap v119: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:06:00.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:00 smithi038 ceph-mon[27972]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-04-22T00:06:00.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:00 smithi203 ceph-mon[31431]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-04-22T00:06:00.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:00 smithi111 ceph-mon[31543]: pgmap v120: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 2.95k op/s 2024-04-22T00:06:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:02 smithi038 ceph-mon[27972]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-22T00:06:02.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:02 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:06:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:02 smithi203 ceph-mon[31431]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-22T00:06:02.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:02 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:06:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:02 smithi111 ceph-mon[31543]: pgmap v121: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-22T00:06:02.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:02 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:06:04.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:04 smithi038 ceph-mon[27972]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 14 MiB/s wr, 3.46k op/s 2024-04-22T00:06:04.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:04 smithi203 ceph-mon[31431]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 14 MiB/s wr, 3.46k op/s 2024-04-22T00:06:04.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:04 smithi111 ceph-mon[31543]: pgmap v122: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 14 MiB/s wr, 3.46k op/s 2024-04-22T00:06:06.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:06 smithi038 ceph-mon[27972]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:06:06.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:06 smithi203 ceph-mon[31431]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:06:06.827 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:06 smithi111 ceph-mon[31543]: pgmap v123: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/1409057336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe181006a0 msgr2=0x7fbe18100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 --2- 172.21.15.38:0/1409057336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe181006a0 0x7fbe18100b20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800fb40 tx=0x7fbe080341d0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/1409057336 shutdown_connections 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 --2- 172.21.15.38:0/1409057336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18101060 0x7fbe1810d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 --2- 172.21.15.38:0/1409057336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe181006a0 0x7fbe18100b20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 --2- 172.21.15.38:0/1409057336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe18106660 0x7fbe18106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.585+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/1409057336 >> 172.21.15.38:0/1409057336 conn(0x7fbe18077790 msgr2=0x7fbe18077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.589+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/1409057336 shutdown_connections 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.589+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/1409057336 wait complete. 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 Processor -- start 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 -- start start 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 0x7fbe181a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 0x7fbe181a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18106660 0x7fbe181ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810f9d0 con 0x7fbe18101060 2024-04-22T00:06:07.586 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810f850 con 0x7fbe181006a0 2024-04-22T00:06:07.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.590+0000 7fbe1fb4f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810fb50 con 0x7fbe18106660 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1e0ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18106660 0x7fbe181ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 0x7fbe181a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 0x7fbe181a6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60364/0 (socket says 172.21.15.38:60364) 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d8eb640 1 -- 172.21.15.38:0/2298490275 learned_addr learned my addr 172.21.15.38:0/2298490275 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 0x7fbe181a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2298490275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18106660 msgr2=0x7fbe181ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18106660 0x7fbe181ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2298490275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 msgr2=0x7fbe181a6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 0x7fbe181a6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.592+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2298490275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe181abdd0 con 0x7fbe18101060 2024-04-22T00:06:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.593+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 0x7fbe181a7320 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbe080347e0 tx=0x7fbe0800e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.593+0000 7fbe0effd640 1 -- 172.21.15.38:0/2298490275 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0800f570 con 0x7fbe18101060 2024-04-22T00:06:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.594+0000 7fbe0effd640 1 -- 172.21.15.38:0/2298490275 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe08018410 con 0x7fbe18101060 2024-04-22T00:06:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.594+0000 7fbe0effd640 1 -- 172.21.15.38:0/2298490275 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080369e0 con 0x7fbe18101060 2024-04-22T00:06:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.594+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/2298490275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe181ac060 con 0x7fbe18101060 2024-04-22T00:06:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.594+0000 7fbe1fb4f640 1 -- 172.21.15.38:0/2298490275 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe181ac4d0 con 0x7fbe18101060 2024-04-22T00:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.599+0000 7fbe22815740 1 Processor -- start 2024-04-22T00:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.599+0000 7fbe0effd640 1 -- 172.21.15.38:0/2298490275 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbe0803a030 con 0x7fbe18101060 2024-04-22T00:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.599+0000 7fbe22815740 1 -- start start 2024-04-22T00:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe22815740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe22815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb61d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe22815740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb64640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb61d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb61d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58936/0 (socket says 172.21.15.38:58936) 2024-04-22T00:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe0effd640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 0x7fbe040a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.600+0000 7fbe1e0ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe1e0ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59080/0 (socket says 172.21.15.38:59080) 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe0effd640 1 -- 172.21.15.38:0/2298490275 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbe080f0d30 con 0x7fbe18101060 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe22815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef8eabcb70 con 0x55ef8eb59480 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe22815740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ef8ea65470 con 0x55ef8eb62250 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe22815740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef8eab8b30 con 0x55ef8eb58b60 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb64640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.601+0000 7fbe1d8eb640 1 -- 172.21.15.38:0/2173984681 learned_addr learned my addr 172.21.15.38:0/2173984681 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2173984681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 msgr2=0x55ef8eb58f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2173984681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 msgr2=0x55ef8eb61d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb61d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2173984681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef8e7d1210 con 0x55ef8eb62250 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb64640 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe080ee250 tx=0x7fbe080be200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36af2462ee3fe902 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.602+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 0x7fbe040a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.603+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 0x7fbe040a5410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400be30 tx=0x7fbe14004730 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.603+0000 7fbe03fff640 1 -- 172.21.15.38:0/2173984681 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080be430 con 0x55ef8eb62250 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.603+0000 7fbe03fff640 1 -- 172.21.15.38:0/2173984681 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe080c27e0 con 0x55ef8eb62250 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.604+0000 7fbe03fff640 1 -- 172.21.15.38:0/2173984681 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080ee510 con 0x55ef8eb62250 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.605+0000 7fbe22815740 1 -- 172.21.15.38:0/2173984681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 msgr2=0x55ef8eb64640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.605+0000 7fbe22815740 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb64640 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe080ee250 tx=0x7fbe080be200 comp rx=0 tx=0).stop 2024-04-22T00:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.607+0000 7fbe22815740 1 -- 172.21.15.38:0/2173984681 shutdown_connections 2024-04-22T00:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.607+0000 7fbe22815740 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb64640 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.607+0000 7fbe22815740 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb61d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.607+0000 7fbe22815740 1 --2- 172.21.15.38:0/2173984681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb58f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.607+0000 7fbe22815740 1 -- 172.21.15.38:0/2173984681 >> 172.21.15.38:0/2173984681 conn(0x55ef8eb53480 msgr2=0x55ef8eb53860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:07.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.609+0000 7fbe22815740 1 -- 172.21.15.38:0/2173984681 shutdown_connections 2024-04-22T00:06:07.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.609+0000 7fbe22815740 1 -- 172.21.15.38:0/2173984681 wait complete. 2024-04-22T00:06:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.610+0000 7fbe22815740 1 -- start start 2024-04-22T00:06:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.610+0000 7fbe22815740 1 Processor -- start 2024-04-22T00:06:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb5f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb5fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ef8eb64c40 con 0x55ef8eb59480 2024-04-22T00:06:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ef8ea78a10 con 0x55ef8eb62250 2024-04-22T00:06:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef8eb64dc0 con 0x55ef8eb58b60 2024-04-22T00:06:07.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.611+0000 7fbe22815740 10 client.0 _ll_register_callbacks cb 0x55ef8eab3b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.612+0000 7fbe1d0ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.612+0000 7fbe1d0ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58950/0 (socket says 172.21.15.38:58950) 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.612+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2147465831 learned_addr learned my addr 172.21.15.38:0/2147465831 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb5fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2147465831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 msgr2=0x55ef8eb5f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb5f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 msgr2=0x55ef8eb5fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb5fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/2147465831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef8e89d620 con 0x55ef8eb59480 2024-04-22T00:06:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbe080c6b70 tx=0x7fbe080bec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b422d3bde46e34e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe017fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080d7020 con 0x55ef8eb59480 2024-04-22T00:06:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.613+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbe0800eb90 con 0x55ef8eb59480 2024-04-22T00:06:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.616+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe080c27e0 con 0x55ef8eb59480 2024-04-22T00:06:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.618+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbe080be690 con 0x55ef8eb59480 2024-04-22T00:06:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.618+0000 7fbe22815740 10 client.17013 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.619+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ef8eb64c40 con 0x55ef8eb59480 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.619+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe080be830 con 0x55ef8eb59480 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.619+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbe080c8030 con 0x55ef8eb59480 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.619+0000 7fbe22815740 10 client.17013 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.619+0000 7fbe22815740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.620+0000 7fbe22815740 20 client.17013 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.620+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8eb72030 0x55ef8eb74490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.620+0000 7fbe22815740 4 client.17013 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.620+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbe080c27e0 con 0x55ef8eb72030 2024-04-22T00:06:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.621+0000 7fbe1d8eb640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8eb72030 0x55ef8eb74490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.625+0000 7fbe1d8eb640 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8eb72030 0x55ef8eb74490 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bb18b043b77d7b server_cookie=27b7a68d75513971 in_seq=0 out_seq=0 2024-04-22T00:06:07.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.625+0000 7fbe017fa640 10 client.17013 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.629+0000 7fbe017fa640 1 -- 172.21.15.38:0/2147465831 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbe080c27e0 con 0x55ef8eb72030 2024-04-22T00:06:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.629+0000 7fbe017fa640 10 client.17013 handle_command_reply: tid=0 2024-04-22T00:06:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.629+0000 7fbe22815740 1 client.17013 shutdown 2024-04-22T00:06:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.629+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 msgr2=0x55ef8eb5f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.629+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbe080c6b70 tx=0x7fbe080bec60 comp rx=0 tx=0).stop 2024-04-22T00:06:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.630+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 shutdown_connections 2024-04-22T00:06:07.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb62250 0x55ef8eb5fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8eb72030 0x55ef8eb74490 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5f910 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe22815740 1 --2- 172.21.15.38:0/2147465831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb58b60 0x55ef8eb5f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 >> 172.21.15.38:0/2147465831 conn(0x55ef8eb53480 msgr2=0x55ef8eb551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:07.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.631+0000 7fbe017fa640 0 client.17013 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:07.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.633+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 shutdown_connections 2024-04-22T00:06:07.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.633+0000 7fbe22815740 1 -- 172.21.15.38:0/2147465831 wait complete. 2024-04-22T00:06:07.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.634+0000 7fbe22815740 20 client.17013 trim_cache size 0 max 16384 2024-04-22T00:06:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.659+0000 7fbe22815740 1 Processor -- start 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.659+0000 7fbe22815740 1 -- start start 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb6da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb62250 0x55ef8eb522a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181bd820 con 0x55ef8eb59480 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181bf360 con 0x55ef8eb58b60 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe22815740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181bdef0 con 0x55ef8eb62250 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58956/0 (socket says 172.21.15.38:58956) 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/3779908258 learned_addr learned my addr 172.21.15.38:0/3779908258 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d8eb640 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb6da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb62250 0x55ef8eb522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/3779908258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb62250 msgr2=0x55ef8eb522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb62250 0x55ef8eb522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/3779908258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 msgr2=0x55ef8eb6da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb6da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.660+0000 7fbe1d0ea640 1 -- 172.21.15.38:0/3779908258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef8eb601b0 con 0x55ef8eb59480 2024-04-22T00:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.661+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbe080bc6e0 tx=0x7fbe080be330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3a721c7b4bcd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.661+0000 7fbe037fe640 1 -- 172.21.15.38:0/3779908258 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080d7020 con 0x55ef8eb59480 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.661+0000 7fbe037fe640 1 -- 172.21.15.38:0/3779908258 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe080bd880 con 0x55ef8eb59480 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.661+0000 7fbe037fe640 1 -- 172.21.15.38:0/3779908258 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080d1070 con 0x55ef8eb59480 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 -- 172.21.15.38:0/3779908258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 msgr2=0x55ef8eb5fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fbe080bc6e0 tx=0x7fbe080be330 comp rx=0 tx=0).stop 2024-04-22T00:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 -- 172.21.15.38:0/3779908258 shutdown_connections 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb62250 0x55ef8eb522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb59480 0x55ef8eb5fc70 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 --2- 172.21.15.38:0/3779908258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb6da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 -- 172.21.15.38:0/3779908258 >> 172.21.15.38:0/3779908258 conn(0x55ef8eb53480 msgr2=0x55ef8eb51b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 -- 172.21.15.38:0/3779908258 shutdown_connections 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.662+0000 7fbe22815740 1 -- 172.21.15.38:0/3779908258 wait complete. 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.663+0000 7fbe22815740 1 -- start start 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.663+0000 7fbe22815740 1 Processor -- start 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.663+0000 7fbe22815740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb69eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.663+0000 7fbe22815740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb59480 0x55ef8eb6a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe22815740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 0x55ef8eb5b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe22815740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181bec90 con 0x55ef8eb62250 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe22815740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181be5c0 con 0x55ef8eb58b60 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe22815740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbe181bb800 con 0x55ef8eb59480 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb69eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1d8eb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb69eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60398/0 (socket says 172.21.15.38:60398) 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1d8eb640 1 -- 172.21.15.38:0/914109434 learned_addr learned my addr 172.21.15.38:0/914109434 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 0x55ef8eb5b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1d0ea640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb59480 0x55ef8eb6a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe22815740 10 client.0 _ll_register_callbacks cb 0x7fbe080d5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 -- 172.21.15.38:0/914109434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb59480 msgr2=0x55ef8eb6a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb59480 0x55ef8eb6a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 -- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 msgr2=0x55ef8eb69eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb69eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.664+0000 7fbe1e0ec640 1 -- 172.21.15.38:0/914109434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef8eb5bde0 con 0x55ef8eb62250 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe1e0ec640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 0x55ef8eb5b7a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x55ef8eb5dbf0 tx=0x7fbe1400d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd6ab5e0fbef1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe017fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14004020 con 0x55ef8eb62250 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fbe140041c0 con 0x55ef8eb62250 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe080bd880 con 0x55ef8eb62250 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbe14034050 con 0x55ef8eb62250 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe22815740 10 client.17019 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.665+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ef8ebd7ec0 con 0x55ef8eb62250 2024-04-22T00:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe1401c040 con 0x55ef8eb62250 2024-04-22T00:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbe1402c440 con 0x55ef8eb62250 2024-04-22T00:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 10 client.17019 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 20 client.17019 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8ebd8630 0x55ef8ebdaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 4 client.17019 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fbe080bd880 con 0x55ef8ebd8630 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe1d8eb640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8ebd8630 0x55ef8ebdaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.666+0000 7fbe017fa640 10 client.17019 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:07.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:07.667+0000 7fbe1d8eb640 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8ebd8630 0x55ef8ebdaa50 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f7a8ad56e9fd5c server_cookie=ec7427e0c9ff2d12 in_seq=0 out_seq=0 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.389+0000 7fbe017fa640 1 -- 172.21.15.38:0/914109434 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fbe080bd880 con 0x55ef8ebd8630 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.389+0000 7fbe017fa640 10 client.17019 handle_command_reply: tid=0 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.389+0000 7fbe22815740 1 client.17019 shutdown 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 msgr2=0x55ef8eb5b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 0x55ef8eb5b7a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x55ef8eb5dbf0 tx=0x7fbe1400d040 comp rx=0 tx=0).stop 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 shutdown_connections 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ef8ebd8630 0x55ef8ebdaa50 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe017fa640 0 client.17019 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ef8eb62250 0x55ef8eb5b7a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef8eb59480 0x55ef8eb6a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 --2- 172.21.15.38:0/914109434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef8eb58b60 0x55ef8eb69eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 >> 172.21.15.38:0/914109434 conn(0x55ef8eb53480 msgr2=0x55ef8eb52420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 shutdown_connections 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 1 -- 172.21.15.38:0/914109434 wait complete. 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.390+0000 7fbe22815740 20 client.17019 trim_cache size 0 max 16384 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:06:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:06:08.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:06:08.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:06:08.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 msgr2=0x7fbe040a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 0x7fbe040a5410 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400be30 tx=0x7fbe14004730 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 msgr2=0x7fbe181a7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 0x7fbe181a7320 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbe080347e0 tx=0x7fbe0800e890 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 shutdown_connections 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbe040a2f50 0x7fbe040a5410 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbe18106660 0x7fbe181ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe18101060 0x7fbe181a7320 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 --2- 172.21.15.38:0/2298490275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe181006a0 0x7fbe181a6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 >> 172.21.15.38:0/2298490275 conn(0x7fbe18077790 msgr2=0x7fbe18103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 shutdown_connections 2024-04-22T00:06:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:08.391+0000 7fbe0cbff640 1 -- 172.21.15.38:0/2298490275 wait complete. 2024-04-22T00:06:08.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:08 smithi038 ceph-mon[27972]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-22T00:06:08.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:08 smithi203 ceph-mon[31431]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-22T00:06:08.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:08 smithi111 ceph-mon[31543]: pgmap v124: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 13 MiB/s wr, 3.33k op/s 2024-04-22T00:06:10.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:10 smithi038 ceph-mon[27972]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 10 MiB/s wr, 2.52k op/s 2024-04-22T00:06:10.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:10 smithi203 ceph-mon[31431]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 10 MiB/s wr, 2.52k op/s 2024-04-22T00:06:10.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:10 smithi111 ceph-mon[31543]: pgmap v125: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 10 MiB/s wr, 2.52k op/s 2024-04-22T00:06:11.108 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:06:11.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:06:11.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 -- 172.21.15.38:0/3462865388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac1066b0 msgr2=0x7efeac106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/3462865388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac1066b0 0x7efeac106a90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7efea8009930 tx=0x7efea802bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 -- 172.21.15.38:0/3462865388 shutdown_connections 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/3462865388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac100fd0 0x7efeac10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/3462865388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 0x7efeac100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/3462865388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac1066b0 0x7efeac106a90 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 -- 172.21.15.38:0/3462865388 >> 172.21.15.38:0/3462865388 conn(0x7efeac077740 msgr2=0x7efeac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 -- 172.21.15.38:0/3462865388 shutdown_connections 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.227+0000 7efeb2cd4640 1 -- 172.21.15.38:0/3462865388 wait complete. 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 Processor -- start 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 -- start start 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 0x7efeac197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac1066b0 0x7efeac19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeac110a80 con 0x7efeac100fd0 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeac110900 con 0x7efeac100610 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb2cd4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeac110c00 con 0x7efeac1066b0 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58994/0 (socket says 172.21.15.38:58994) 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 -- 172.21.15.38:0/2358744641 learned_addr learned my addr 172.21.15.38:0/2358744641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb0a70640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 0x7efeac197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efeb1271640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac1066b0 0x7efeac19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 -- 172.21.15.38:0/2358744641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac1066b0 msgr2=0x7efeac19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac1066b0 0x7efeac19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 -- 172.21.15.38:0/2358744641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 msgr2=0x7efeac197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 0x7efeac197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 -- 172.21.15.38:0/2358744641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac19cf60 con 0x7efeac100fd0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.228+0000 7efea3fff640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00c920 tx=0x7efe9c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.229+0000 7efea1ffb640 1 -- 172.21.15.38:0/2358744641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c007c70 con 0x7efeac100fd0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.229+0000 7efea1ffb640 1 -- 172.21.15.38:0/2358744641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe9c007e10 con 0x7efeac100fd0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.229+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac19d250 con 0x7efeac100fd0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.229+0000 7efea1ffb640 1 -- 172.21.15.38:0/2358744641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c018890 con 0x7efeac100fd0 2024-04-22T00:06:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.229+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac1a4ae0 con 0x7efeac100fd0 2024-04-22T00:06:11.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efea1ffb640 1 -- 172.21.15.38:0/2358744641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efe9c018a70 con 0x7efeac100fd0 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 Processor -- start 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 -- start start 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d054083ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d054084430 0x55d054086820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d053fdf560 con 0x55d05407af30 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d053f87730 con 0x55d05407b850 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb599d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d053fdb520 con 0x55d054084430 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efea1ffb640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 0x7efe940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb1271640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb1271640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59002/0 (socket says 172.21.15.38:59002) 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb1271640 1 -- 172.21.15.38:0/261809430 learned_addr learned my addr 172.21.15.38:0/261809430 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efea3fff640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d054084430 0x55d054086820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.232+0000 7efeb0a70640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d054083ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 0x7efe940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efea1ffb640 1 -- 172.21.15.38:0/2358744641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efe9c0cf830 con 0x7efeac100fd0 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 -- 172.21.15.38:0/261809430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d054084430 msgr2=0x55d054086820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d054084430 0x55d054086820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 -- 172.21.15.38:0/261809430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 msgr2=0x55d054083ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d054083ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 -- 172.21.15.38:0/261809430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d053dbd7d0 con 0x55d05407af30 2024-04-22T00:06:11.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb0a70640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d054083ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efea400a990 tx=0x7efea400ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e469595def56d41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efe92ffd640 1 -- 172.21.15.38:0/261809430 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea401d030 con 0x55d05407af30 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efe92ffd640 1 -- 172.21.15.38:0/261809430 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea4015c60 con 0x55d05407af30 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efe92ffd640 1 -- 172.21.15.38:0/261809430 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea40213c0 con 0x55d05407af30 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.233+0000 7efeb1271640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 0x7efe940a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efea4020000 tx=0x7efea401e000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- 172.21.15.38:0/261809430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 msgr2=0x55d05407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efea400a990 tx=0x7efea400ae50 comp rx=0 tx=0).stop 2024-04-22T00:06:11.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- 172.21.15.38:0/261809430 shutdown_connections 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d054084430 0x55d054086820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d054083ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 --2- 172.21.15.38:0/261809430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407af30 0x55d05407b310 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- 172.21.15.38:0/261809430 >> 172.21.15.38:0/261809430 conn(0x55d054075660 msgr2=0x55d054075a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- 172.21.15.38:0/261809430 shutdown_connections 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- 172.21.15.38:0/261809430 wait complete. 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 -- start start 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.234+0000 7efeb599d740 1 Processor -- start 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d0540814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d054081f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d054087220 con 0x55d05407b850 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d053f9ae00 con 0x55d054084430 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0540873a0 con 0x55d05407af30 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 10 client.0 _ll_register_callbacks cb 0x55d053fd5eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59018/0 (socket says 172.21.15.38:59018) 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 -- 172.21.15.38:0/1067425534 learned_addr learned my addr 172.21.15.38:0/1067425534 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb1271640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d054081f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb0a70640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d0540814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 -- 172.21.15.38:0/1067425534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 msgr2=0x55d0540814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d0540814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 -- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 msgr2=0x55d054081f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d054081f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 -- 172.21.15.38:0/1067425534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d053cf33f0 con 0x55d05407b850 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb1271640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d054081f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efea3fff640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0ccca0 tx=0x7efe9c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc20b45bbdd8a64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efe8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c09fdc0 con 0x55d05407b850 2024-04-22T00:06:11.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efe9c098080 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.235+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea4015c60 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efe9c0a4680 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 10 client.17031 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d053cb30f0 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea4015c60 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efe9c0cfce0 con 0x55d05407b850 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 10 client.17031 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 20 client.17031 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:11.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054094870 0x55d054096cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 4 client.17031 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d053fecaa0 con 0x55d054094870 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.236+0000 7efeb0a70640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054094870 0x55d054096cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.237+0000 7efe8bfff640 10 client.17031 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.237+0000 7efeb0a70640 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054094870 0x55d054096cd0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67c83a6575eb7f2 server_cookie=3158c540b099bc0f in_seq=0 out_seq=0 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efe8bfff640 1 -- 172.21.15.38:0/1067425534 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d053fecaa0 con 0x55d054094870 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efe8bfff640 10 client.17031 handle_command_reply: tid=0 2024-04-22T00:06:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 client.17031 shutdown 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 msgr2=0x55d054081a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0ccca0 tx=0x7efe9c000bc0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 shutdown_connections 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d054081f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054094870 0x55d054096cd0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d054081a00 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 --2- 172.21.15.38:0/1067425534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d0540814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 >> 172.21.15.38:0/1067425534 conn(0x55d054075660 msgr2=0x55d0540773b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efe8bfff640 0 client.17031 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:11.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 shutdown_connections 2024-04-22T00:06:11.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.239+0000 7efeb599d740 1 -- 172.21.15.38:0/1067425534 wait complete. 2024-04-22T00:06:11.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.240+0000 7efeb599d740 20 client.17031 trim_cache size 0 max 16384 2024-04-22T00:06:11.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 Processor -- start 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 -- start start 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d054083bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d0540900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1ae960 con 0x55d05407b850 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1b04a0 con 0x55d054084430 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb599d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1af030 con 0x55d05407af30 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efea3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59024/0 (socket says 172.21.15.38:59024) 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efea3fff640 1 -- 172.21.15.38:0/484508504 learned_addr learned my addr 172.21.15.38:0/484508504 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb0a70640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d054083bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.261+0000 7efeb1271640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d0540900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 -- 172.21.15.38:0/484508504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 msgr2=0x55d0540900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d0540900e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 -- 172.21.15.38:0/484508504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 msgr2=0x55d05407ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 -- 172.21.15.38:0/484508504 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d054080470 con 0x55d05407af30 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efea3fff640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efeb0a70640 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d054083bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efea8009230 tx=0x7efea8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4432ce0db649013 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efe927fc640 1 -- 172.21.15.38:0/484508504 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8004380 con 0x55d05407af30 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efe927fc640 1 -- 172.21.15.38:0/484508504 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea804e030 con 0x55d05407af30 2024-04-22T00:06:11.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.262+0000 7efe927fc640 1 -- 172.21.15.38:0/484508504 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea804a330 con 0x55d05407af30 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 -- 172.21.15.38:0/484508504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 msgr2=0x55d054083bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d054083bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efea8009230 tx=0x7efea8009260 comp rx=0 tx=0).stop 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 -- 172.21.15.38:0/484508504 shutdown_connections 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d054084430 0x55d0540900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d05407b850 0x55d05407ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 --2- 172.21.15.38:0/484508504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d054083bf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 -- 172.21.15.38:0/484508504 >> 172.21.15.38:0/484508504 conn(0x55d054075660 msgr2=0x55d054073e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 -- 172.21.15.38:0/484508504 shutdown_connections 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.263+0000 7efeb599d740 1 -- 172.21.15.38:0/484508504 wait complete. 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 -- start start 2024-04-22T00:06:11.260 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 Processor -- start 2024-04-22T00:06:11.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d05408c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d054084430 0x55d0540895c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1afdd0 con 0x55d054084430 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1af700 con 0x55d05407b850 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1ac940 con 0x55d05407af30 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.264+0000 7efeb599d740 10 client.0 _ll_register_callbacks cb 0x7efe9c084f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60440/0 (socket says 172.21.15.38:60440) 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 -- 172.21.15.38:0/4256132825 learned_addr learned my addr 172.21.15.38:0/4256132825 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efeb1271640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d054084430 0x55d0540895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efeb0a70640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d05408c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 -- 172.21.15.38:0/4256132825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 msgr2=0x55d05408c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d05408c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 -- 172.21.15.38:0/4256132825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d054084430 msgr2=0x55d0540895c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d054084430 0x55d0540895c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 -- 172.21.15.38:0/4256132825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05407b680 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efea3fff640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0983e0 tx=0x7efe9c0ccd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ceee0fdda4bfa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efe8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.265+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c09fdc0 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea804e030 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efe9c097760 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efe9c0979a0 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efeb599d740 10 client.26438 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d0540ff8b0 con 0x55d05407b850 2024-04-22T00:06:11.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea804e030 con 0x55d05407b850 2024-04-22T00:06:11.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.266+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efe9c0cfce0 con 0x55d05407b850 2024-04-22T00:06:11.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 10 client.26438 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:11.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 10 client.26438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:11.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 20 client.26438 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:11.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054100030 0x55d054102450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 4 client.26438 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d053fe6df0 con 0x55d054100030 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb0a70640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054100030 0x55d054102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efe8bfff640 10 client.26438 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.267+0000 7efeb0a70640 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054100030 0x55d054102450 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22902b36c3ca0801 server_cookie=6fe31a008df580a8 in_seq=0 out_seq=0 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efe8bfff640 1 -- 172.21.15.38:0/4256132825 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d053fe6df0 con 0x55d054100030 2024-04-22T00:06:11.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efe8bfff640 10 client.26438 handle_command_reply: tid=0 2024-04-22T00:06:11.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 client.26438 shutdown 2024-04-22T00:06:11.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 msgr2=0x55d05408cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0983e0 tx=0x7efe9c0ccd10 comp rx=0 tx=0).stop 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 shutdown_connections 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d054100030 0x55d054102450 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d054084430 0x55d0540895c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d05407b850 0x55d05408cb30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 --2- 172.21.15.38:0/4256132825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d05407af30 0x55d05408c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 >> 172.21.15.38:0/4256132825 conn(0x55d054075660 msgr2=0x55d0540746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efe8bfff640 0 client.26438 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 shutdown_connections 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 1 -- 172.21.15.38:0/4256132825 wait complete. 2024-04-22T00:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.268+0000 7efeb599d740 20 client.26438 trim_cache size 0 max 16384 2024-04-22T00:06:11.272 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:11.272 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:06:11.272 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:06:11.273 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:11.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 msgr2=0x7efe940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 0x7efe940a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efea4020000 tx=0x7efea401e000 comp rx=0 tx=0).stop 2024-04-22T00:06:11.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 msgr2=0x7efeac1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00c920 tx=0x7efe9c00cde0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 shutdown_connections 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe940a2d00 0x7efe940a51c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeac1066b0 0x7efeac19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeac100fd0 0x7efeac1984b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 --2- 172.21.15.38:0/2358744641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeac100610 0x7efeac197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 >> 172.21.15.38:0/2358744641 conn(0x7efeac077740 msgr2=0x7efeac104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 shutdown_connections 2024-04-22T00:06:11.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.269+0000 7efeb2cd4640 1 -- 172.21.15.38:0/2358744641 wait complete. 2024-04-22T00:06:11.281 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:06:11.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:06:11.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.407+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/198403021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 msgr2=0x7f3d28106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.407+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/198403021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d28106a10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009930 tx=0x7f3d1802bc60 comp rx=0 tx=0).stop 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/198403021 shutdown_connections 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/198403021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100fe0 0x7f3d2810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/198403021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100620 0x7f3d28100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/198403021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d28106a10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/198403021 >> 172.21.15.38:0/198403021 conn(0x7f3d28077740 msgr2=0x7f3d28077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/198403021 shutdown_connections 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.408+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/198403021 wait complete. 2024-04-22T00:06:11.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 Processor -- start 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 -- start start 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100620 0x7f3d28197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100fe0 0x7f3d28197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3d28110b50 con 0x7f3d28106630 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3d281109d0 con 0x7f3d28100620 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2cfd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d28110cd0 con 0x7f3d28100fe0 2024-04-22T00:06:11.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d26d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d26d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59050/0 (socket says 172.21.15.38:59050) 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d26d9d640 1 -- 172.21.15.38:0/1564575861 learned_addr learned my addr 172.21.15.38:0/1564575861 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.409+0000 7f3d2659c640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100620 0x7f3d28197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d25d9b640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100fe0 0x7f3d28197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 -- 172.21.15.38:0/1564575861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100fe0 msgr2=0x7f3d28197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100fe0 0x7f3d28197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 -- 172.21.15.38:0/1564575861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100620 msgr2=0x7f3d28197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100620 0x7f3d28197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 -- 172.21.15.38:0/1564575861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2819c040 con 0x7f3d28106630 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d26d9d640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b990 tx=0x7f3d2000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.410+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20019070 con 0x7f3d28106630 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.411+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2000cba0 con 0x7f3d28106630 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.411+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20007650 con 0x7f3d28106630 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.411+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2819c330 con 0x7f3d28106630 2024-04-22T00:06:11.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.411+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d280688d0 con 0x7f3d28106630 2024-04-22T00:06:11.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.412+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d28068560 con 0x7f3d28106630 2024-04-22T00:06:11.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.414+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3d2001c070 con 0x7f3d28106630 2024-04-22T00:06:11.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.415+0000 7f3d177fe640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 0x7f3d100a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.415+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3d200cec10 con 0x7f3d28106630 2024-04-22T00:06:11.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.416+0000 7f3d2659c640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 0x7f3d100a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.416+0000 7f3d2659c640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 0x7f3d100a75d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3d1802efd0 tx=0x7f3d18000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.423+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3d2009af50 con 0x7f3d28106630 2024-04-22T00:06:11.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.771+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d28069350 con 0x7f3d28106630 2024-04-22T00:06:11.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.772+0000 7f3d177fe640 1 -- 172.21.15.38:0/1564575861 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7f3d2009f250 con 0x7f3d28106630 2024-04-22T00:06:11.767 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:06:11.767 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:06:11.767 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:06:11.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.775+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 msgr2=0x7f3d100a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.775+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 0x7f3d100a75d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3d1802efd0 tx=0x7f3d18000bc0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.775+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 msgr2=0x7f3d2819b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.775+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b990 tx=0x7f3d2000be50 comp rx=0 tx=0).stop 2024-04-22T00:06:11.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 shutdown_connections 2024-04-22T00:06:11.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3d100a5110 0x7f3d100a75d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d28106630 0x7f3d2819b910 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d28100fe0 0x7f3d28197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 --2- 172.21.15.38:0/1564575861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d28100620 0x7f3d28197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 >> 172.21.15.38:0/1564575861 conn(0x7f3d28077740 msgr2=0x7f3d28102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 shutdown_connections 2024-04-22T00:06:11.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.776+0000 7f3d2cfd7640 1 -- 172.21.15.38:0/1564575861 wait complete. 2024-04-22T00:06:11.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:06:11.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 -- 172.21.15.38:0/3524060616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb411af20 msgr2=0x7f2eb411d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 --2- 172.21.15.38:0/3524060616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb411af20 0x7f2eb411d310 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009ca0 tx=0x7f2eac02c100 comp rx=0 tx=0).stop 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 -- 172.21.15.38:0/3524060616 shutdown_connections 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 --2- 172.21.15.38:0/3524060616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb411af20 0x7f2eb411d310 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 --2- 172.21.15.38:0/3524060616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb410dfc0 0x7f2eb410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 --2- 172.21.15.38:0/3524060616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eb410ff90 0x7f2eb410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.970+0000 7f2eba639640 1 -- 172.21.15.38:0/3524060616 >> 172.21.15.38:0/3524060616 conn(0x7f2eb406c850 msgr2=0x7f2eb406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.971+0000 7f2eba639640 1 -- 172.21.15.38:0/3524060616 shutdown_connections 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.972+0000 7f2eba639640 1 -- 172.21.15.38:0/3524060616 wait complete. 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.972+0000 7f2eba639640 1 Processor -- start 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.974+0000 7f2eba639640 1 -- start start 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eb410dfc0 0x7f2eb411a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 0x7f2eb41146e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4111130 con 0x7f2eb411af20 2024-04-22T00:06:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4110fb0 con 0x7f2eb410dfc0 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eba639640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41112b0 con 0x7f2eb410ff90 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eb8bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eb8bd6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59078/0 (socket says 172.21.15.38:59078) 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eb8bd6640 1 -- 172.21.15.38:0/279122732 learned_addr learned my addr 172.21.15.38:0/279122732 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.975+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 0x7f2eb41146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb37fe640 1 -- 172.21.15.38:0/279122732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eb410dfc0 msgr2=0x7f2eb411a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eb410dfc0 0x7f2eb411a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb37fe640 1 -- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 msgr2=0x7f2eb4114c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb37fe640 1 -- 172.21.15.38:0/279122732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb4115350 con 0x7f2eb410ff90 2024-04-22T00:06:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.976+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:11.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.980+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 0x7f2eb41146e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b3f0 tx=0x7f2ea400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.981+0000 7f2eb17fa640 1 -- 172.21.15.38:0/279122732 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4015020 con 0x7f2eb410ff90 2024-04-22T00:06:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.981+0000 7f2eb17fa640 1 -- 172.21.15.38:0/279122732 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea4011d10 con 0x7f2eb410ff90 2024-04-22T00:06:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.981+0000 7f2eb17fa640 1 -- 172.21.15.38:0/279122732 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400f2e0 con 0x7f2eb410ff90 2024-04-22T00:06:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.981+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb41bbc20 con 0x7f2eb410ff90 2024-04-22T00:06:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.982+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb41bc030 con 0x7f2eb410ff90 2024-04-22T00:06:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.986+0000 7f2ebd223740 1 Processor -- start 2024-04-22T00:06:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.986+0000 7f2eb17fa640 1 -- 172.21.15.38:0/279122732 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2ea4018020 con 0x7f2eb410ff90 2024-04-22T00:06:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2eb17fa640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 0x7f2e9c0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2eb17fa640 1 -- 172.21.15.38:0/279122732 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2ea40cee00 con 0x7f2eb410ff90 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 -- start start 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bdb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96be4520 0x560f96be6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f96b3f530 con 0x560f96be4520 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560f96ae7830 con 0x560f96bdb020 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.987+0000 7f2ebd223740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560f96b3b4f0 con 0x560f96bdb940 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.988+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 0x7f2e9c0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.988+0000 7f2eb37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.988+0000 7f2eb37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59196/0 (socket says 172.21.15.38:59196) 2024-04-22T00:06:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.988+0000 7f2eb37fe640 1 -- 172.21.15.38:0/2848075800 learned_addr learned my addr 172.21.15.38:0/2848075800 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb37fe640 1 -- 172.21.15.38:0/2848075800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 msgr2=0x560f96bdb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bdb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb37fe640 1 -- 172.21.15.38:0/2848075800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96be4520 msgr2=0x560f96be6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96be4520 0x560f96be6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb37fe640 1 -- 172.21.15.38:0/2848075800 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f96b0b2c0 con 0x560f96bdb940 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.989+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 0x7f2e9c0a52c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2eac005fe0 tx=0x7f2eac005f70 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.990+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40a37d0 tx=0x7f2ea40092c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132ef76c14301e8c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.992+0000 7f2e9a7fc640 1 -- 172.21.15.38:0/2848075800 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea409b090 con 0x560f96bdb940 2024-04-22T00:06:11.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.992+0000 7f2e9a7fc640 1 -- 172.21.15.38:0/2848075800 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea409f440 con 0x560f96bdb940 2024-04-22T00:06:11.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.993+0000 7f2ebd223740 1 -- 172.21.15.38:0/2848075800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 msgr2=0x560f96be3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.993+0000 7f2ebd223740 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40a37d0 tx=0x7f2ea40092c0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 -- 172.21.15.38:0/2848075800 shutdown_connections 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96be4520 0x560f96be6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be3fe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 --2- 172.21.15.38:0/2848075800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bdb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 -- 172.21.15.38:0/2848075800 >> 172.21.15.38:0/2848075800 conn(0x560f96bd5750 msgr2=0x560f96bd5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 -- 172.21.15.38:0/2848075800 shutdown_connections 2024-04-22T00:06:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.995+0000 7f2ebd223740 1 -- 172.21.15.38:0/2848075800 wait complete. 2024-04-22T00:06:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 -- start start 2024-04-22T00:06:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 Processor -- start 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb020 0x560f96be3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f96be7300 con 0x560f96bdb020 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560f96afaf10 con 0x560f96be4520 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560f96be7480 con 0x560f96bdb940 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.996+0000 7f2ebd223740 10 client.0 _ll_register_callbacks cb 0x560f96852be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59210/0 (socket says 172.21.15.38:59210) 2024-04-22T00:06:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 -- 172.21.15.38:0/441608034 learned_addr learned my addr 172.21.15.38:0/441608034 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 -- 172.21.15.38:0/441608034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 msgr2=0x560f96be1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 -- 172.21.15.38:0/441608034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb020 msgr2=0x560f96be3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb020 0x560f96be3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.997+0000 7f2eb37fe640 1 -- 172.21.15.38:0/441608034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f96accc80 con 0x560f96bdb940 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.998+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40103c0 tx=0x7f2ea40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a21fee01340fb44 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.998+0000 7f2e937fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.998+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40cc430 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.998+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2ea40cc5d0 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.998+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ea409f440 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.999+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2ea409a4e0 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.999+0000 7f2ebd223740 10 client.26311 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:11.999+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560f968533f0 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea409f440 con 0x560f96bdb940 2024-04-22T00:06:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2ea4099f50 con 0x560f96bdb940 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 10 client.26311 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 10 client.26311 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 20 client.26311 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96bf4290 0x560f96bf6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 4 client.26311 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.000+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f96b4ca70 con 0x560f96bf4290 2024-04-22T00:06:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.001+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96bf4290 0x560f96bf6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:11.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.002+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96bf4290 0x560f96bf6680 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6de778d337fc6c4 server_cookie=b414af5523b5f37a in_seq=0 out_seq=0 2024-04-22T00:06:11.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.002+0000 7f2e937fe640 10 client.26311 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:11.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2e937fe640 1 -- 172.21.15.38:0/441608034 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f96b4ca70 con 0x560f96bf4290 2024-04-22T00:06:11.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2e937fe640 10 client.26311 handle_command_reply: tid=0 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 client.26311 shutdown 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 msgr2=0x560f96be1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40103c0 tx=0x7f2ea40047d0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 shutdown_connections 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96bf4290 0x560f96bf6680 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb940 0x560f96be1270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 --2- 172.21.15.38:0/441608034 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb020 0x560f96be3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 >> 172.21.15.38:0/441608034 conn(0x560f96bd5750 msgr2=0x560f96bd74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.004+0000 7f2e937fe640 0 client.26311 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.005+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 shutdown_connections 2024-04-22T00:06:12.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.006+0000 7f2ebd223740 1 -- 172.21.15.38:0/441608034 wait complete. 2024-04-22T00:06:12.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.006+0000 7f2ebd223740 20 client.26311 trim_cache size 0 max 16384 2024-04-22T00:06:12.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.035+0000 7f2ebd223740 1 Processor -- start 2024-04-22T00:06:12.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.036+0000 7f2ebd223740 1 -- start start 2024-04-22T00:06:12.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.036+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.036+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96be39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59214/0 (socket says 172.21.15.38:59214) 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.036+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2eb3fff640 1 -- 172.21.15.38:0/1756940382 learned_addr learned my addr 172.21.15.38:0/1756940382 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96be39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c60d0 con 0x560f96bdb940 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c7c10 con 0x560f96be4520 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.037+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c67a0 con 0x560f96bdb020 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb3fff640 1 -- 172.21.15.38:0/1756940382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 msgr2=0x560f96be3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb3fff640 1 -- 172.21.15.38:0/1756940382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 msgr2=0x560f96be39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96be39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb3fff640 1 -- 172.21.15.38:0/1756940382 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f96be11e0 con 0x560f96bdb020 2024-04-22T00:06:12.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.038+0000 7f2eb8bd6640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:12.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.039+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009a80 tx=0x7f2ea80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67207b5f6e873a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.039+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96be39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:12.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.040+0000 7f2e99ffb640 1 -- 172.21.15.38:0/1756940382 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea801c030 con 0x560f96bdb020 2024-04-22T00:06:12.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.040+0000 7f2e99ffb640 1 -- 172.21.15.38:0/1756940382 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea8010df0 con 0x560f96bdb020 2024-04-22T00:06:12.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.040+0000 7f2e99ffb640 1 -- 172.21.15.38:0/1756940382 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea80035f0 con 0x560f96bdb020 2024-04-22T00:06:12.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.042+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 msgr2=0x560f96bf0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.036 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.042+0000 7f2ebd223740 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009a80 tx=0x7f2ea80046c0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.043+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 shutdown_connections 2024-04-22T00:06:12.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.043+0000 7f2ebd223740 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96be4520 0x560f96be3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.043+0000 7f2ebd223740 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96be39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.043+0000 7f2ebd223740 1 --2- 172.21.15.38:0/1756940382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96bdb020 0x560f96bf0210 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.043+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 >> 172.21.15.38:0/1756940382 conn(0x560f96bd5750 msgr2=0x560f96bd3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.044+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 shutdown_connections 2024-04-22T00:06:12.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.044+0000 7f2ebd223740 1 -- 172.21.15.38:0/1756940382 wait complete. 2024-04-22T00:06:12.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.044+0000 7f2ebd223740 1 -- start start 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.044+0000 7f2ebd223740 1 Processor -- start 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96bddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 0x560f96bde040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c7540 con 0x560f96bdb940 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c6e70 con 0x560f96bdb020 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41c40b0 con 0x560f96be4520 2024-04-22T00:06:12.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2ebd223740 10 client.0 _ll_register_callbacks cb 0x7f2ea409a9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.045+0000 7f2eb8bd6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 0x560f96bde040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb8bd6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 0x560f96bde040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59220/0 (socket says 172.21.15.38:59220) 2024-04-22T00:06:12.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb8bd6640 1 -- 172.21.15.38:0/3488614505 learned_addr learned my addr 172.21.15.38:0/3488614505 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96bddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 -- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 msgr2=0x560f96bde040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 0x560f96bde040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 -- 172.21.15.38:0/3488614505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 msgr2=0x560f96bddb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96bddb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.046+0000 7f2eb3fff640 1 -- 172.21.15.38:0/3488614505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f96bdb770 con 0x560f96bdb020 2024-04-22T00:06:12.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.048+0000 7f2eb3fff640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bec910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8005d00 tx=0x7f2ea8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d384b6b813167 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.048+0000 7f2e937fe640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:12.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.048+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8026020 con 0x560f96bdb020 2024-04-22T00:06:12.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.049+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ea8010df0 con 0x560f96bdb020 2024-04-22T00:06:12.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.049+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2ea8004440 con 0x560f96bdb020 2024-04-22T00:06:12.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.049+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2ea8031050 con 0x560f96bdb020 2024-04-22T00:06:12.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.049+0000 7f2ebd223740 10 client.26450 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.049+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560f9691d7d0 con 0x560f96bdb020 2024-04-22T00:06:12.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.050+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea8010df0 con 0x560f96bdb020 2024-04-22T00:06:12.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2ea8003b20 con 0x560f96bdb020 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 10 client.26450 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 10 client.26450 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 20 client.26450 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96c5a610 0x560f96c5ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 4 client.26450 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f96b46dc0 con 0x560f96c5a610 2024-04-22T00:06:12.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.051+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96c5a610 0x560f96c5ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2eb37fe640 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96c5a610 0x560f96c5ca30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd2d6104e80373 server_cookie=75ed1b126b660c0e in_seq=0 out_seq=0 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2e937fe640 10 client.26450 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2e937fe640 1 -- 172.21.15.38:0/3488614505 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f96b46dc0 con 0x560f96c5a610 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2e937fe640 10 client.26450 handle_command_reply: tid=0 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 client.26450 shutdown 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 msgr2=0x560f96bec910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bec910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8005d00 tx=0x7f2ea8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 shutdown_connections 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560f96c5a610 0x560f96c5ca30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560f96be4520 0x560f96bde040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f96bdb940 0x560f96bddb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 --2- 172.21.15.38:0/3488614505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560f96bdb020 0x560f96bec910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 >> 172.21.15.38:0/3488614505 conn(0x560f96bd5750 msgr2=0x560f96bd4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.049 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.053+0000 7f2e937fe640 0 client.26450 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:12.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 shutdown_connections 2024-04-22T00:06:12.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2ebd223740 1 -- 172.21.15.38:0/3488614505 wait complete. 2024-04-22T00:06:12.050 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2ebd223740 20 client.26450 trim_cache size 0 max 16384 2024-04-22T00:06:12.050 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 msgr2=0x7f2e9c0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 0x7f2e9c0a52c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2eac005fe0 tx=0x7f2eac005f70 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 msgr2=0x7f2eb41146e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.056+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 0x7f2eb41146e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b3f0 tx=0x7f2ea400b8b0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 shutdown_connections 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e9c0a2e00 0x7f2e9c0a52c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eb411af20 0x7f2eb4114c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eb410ff90 0x7f2eb41146e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 --2- 172.21.15.38:0/279122732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eb410dfc0 0x7f2eb411a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 >> 172.21.15.38:0/279122732 conn(0x7f2eb406c850 msgr2=0x7f2eb411c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.057+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 shutdown_connections 2024-04-22T00:06:12.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.058+0000 7f2eba639640 1 -- 172.21.15.38:0/279122732 wait complete. 2024-04-22T00:06:12.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:06:12.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- 172.21.15.38:0/2872900303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 msgr2=0x7f5220103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 --2- 172.21.15.38:0/2872900303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f5220103e00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5210009930 tx=0x7f521002bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- 172.21.15.38:0/2872900303 shutdown_connections 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 --2- 172.21.15.38:0/2872900303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220104340 0x7f5220110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 --2- 172.21.15.38:0/2872900303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f5220103e00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 --2- 172.21.15.38:0/2872900303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220109990 0x7f5220109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- 172.21.15.38:0/2872900303 >> 172.21.15.38:0/2872900303 conn(0x7f52200ff4d0 msgr2=0x7f52201018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- 172.21.15.38:0/2872900303 shutdown_connections 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- 172.21.15.38:0/2872900303 wait complete. 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 Processor -- start 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.192+0000 7f52285de640 1 -- start start 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220104340 0x7f522019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220109990 0x7f522019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113da0 con 0x7f5220103980 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113c20 con 0x7f5220104340 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f52285de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5220113f20 con 0x7f5220109990 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59144/0 (socket says 172.21.15.38:59144) 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 -- 172.21.15.38:0/3754754365 learned_addr learned my addr 172.21.15.38:0/3754754365 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f5226b7b640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220109990 0x7f522019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f5225b79640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220104340 0x7f522019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 -- 172.21.15.38:0/3754754365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220109990 msgr2=0x7f522019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220109990 0x7f522019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 -- 172.21.15.38:0/3754754365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220104340 msgr2=0x7f522019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220104340 0x7f522019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 -- 172.21.15.38:0/3754754365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522019f2f0 con 0x7f5220103980 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.193+0000 7f522637a640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f521800ebc0 tx=0x7f521800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.194+0000 7f52177fe640 1 -- 172.21.15.38:0/3754754365 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218010630 con 0x7f5220103980 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.194+0000 7f52177fe640 1 -- 172.21.15.38:0/3754754365 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5218010cb0 con 0x7f5220103980 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.194+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522019f5e0 con 0x7f5220103980 2024-04-22T00:06:12.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.194+0000 7f52177fe640 1 -- 172.21.15.38:0/3754754365 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218018bd0 con 0x7f5220103980 2024-04-22T00:06:12.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.194+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201a6e70 con 0x7f5220103980 2024-04-22T00:06:12.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.196+0000 7f52177fe640 1 -- 172.21.15.38:0/3754754365 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f52180107d0 con 0x7f5220103980 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 Processor -- start 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 -- start start 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x565342360fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x565342363910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5653422bc530 con 0x565342361520 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565342264830 con 0x565342358940 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522b1ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5653422b84f0 con 0x565342358020 2024-04-22T00:06:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f5226b7b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f5226b7b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59248/0 (socket says 172.21.15.38:59248) 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f5226b7b640 1 -- 172.21.15.38:0/835776625 learned_addr learned my addr 172.21.15.38:0/835776625 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.197+0000 7f522637a640 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x565342360fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5225b79640 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x565342363910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f52177fe640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 0x7f520c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 -- 172.21.15.38:0/835776625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 msgr2=0x565342360fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x565342360fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 -- 172.21.15.38:0/835776625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 msgr2=0x565342363910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x565342363910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 -- 172.21.15.38:0/835776625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653422882c0 con 0x565342358020 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f52177fe640 1 -- 172.21.15.38:0/3754754365 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52180cfcf0 con 0x7f5220103980 2024-04-22T00:06:12.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f521c009ca0 tx=0x7f521c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fcd53910359dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5226b7b640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 0x7f520c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5214bfe640 1 -- 172.21.15.38:0/835776625 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c01e070 con 0x565342358020 2024-04-22T00:06:12.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.198+0000 7f5214bfe640 1 -- 172.21.15.38:0/835776625 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521c019df0 con 0x565342358020 2024-04-22T00:06:12.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.199+0000 7f5214bfe640 1 -- 172.21.15.38:0/835776625 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c018590 con 0x565342358020 2024-04-22T00:06:12.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.199+0000 7f522b1ca740 1 -- 172.21.15.38:0/835776625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 msgr2=0x565342358400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.199+0000 7f5226b7b640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 0x7f520c0a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f521c01b000 tx=0x7f521c018be0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.199+0000 7f522b1ca740 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f521c009ca0 tx=0x7f521c011810 comp rx=0 tx=0).stop 2024-04-22T00:06:12.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 -- 172.21.15.38:0/835776625 shutdown_connections 2024-04-22T00:06:12.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x565342363910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x565342360fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 --2- 172.21.15.38:0/835776625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x565342358400 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 -- 172.21.15.38:0/835776625 >> 172.21.15.38:0/835776625 conn(0x565342352750 msgr2=0x565342352b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 -- 172.21.15.38:0/835776625 shutdown_connections 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 -- 172.21.15.38:0/835776625 wait complete. 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 -- start start 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.200+0000 7f522b1ca740 1 Processor -- start 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x56534235e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x565342364310 con 0x565342358020 2024-04-22T00:06:12.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x565342277f10 con 0x565342358940 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x565342364490 con 0x565342361520 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522b1ca740 10 client.0 _ll_register_callbacks cb 0x565341fcfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59170/0 (socket says 172.21.15.38:59170) 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 -- 172.21.15.38:0/1036312321 learned_addr learned my addr 172.21.15.38:0/1036312321 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f5226b7b640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x56534235e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 -- 172.21.15.38:0/1036312321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 msgr2=0x56534235e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x56534235e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 -- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 msgr2=0x56534235e1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.201+0000 7f522637a640 1 -- 172.21.15.38:0/1036312321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565342249c80 con 0x565342358020 2024-04-22T00:06:12.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f522637a640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f521800eb90 tx=0x7f521800ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54412e0bb5b2ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f5209ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218010e60 con 0x565342358020 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f521809b3d0 con 0x565342358020 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f521c019df0 con 0x565342358020 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5218098220 con 0x565342358020 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f522b1ca740 10 client.17061 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565341fd03f0 con 0x565342358020 2024-04-22T00:06:12.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.202+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521c019df0 con 0x565342358020 2024-04-22T00:06:12.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5218097cd0 con 0x565342358020 2024-04-22T00:06:12.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 10 client.17061 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 10 client.17061 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 20 client.17061 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x565342379dc0 0x56534237c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 4 client.17061 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.203+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653422c9a70 con 0x565342379dc0 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.204+0000 7f5225b79640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x565342379dc0 0x56534237c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.204+0000 7f5209ffb640 10 client.17061 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.204+0000 7f5225b79640 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x565342379dc0 0x56534237c250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f087fb1f1ceb997 server_cookie=6fa4b91710560069 in_seq=0 out_seq=0 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f5209ffb640 1 -- 172.21.15.38:0/1036312321 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653422c9a70 con 0x565342379dc0 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f5209ffb640 10 client.17061 handle_command_reply: tid=0 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 client.17061 shutdown 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 msgr2=0x565342360df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f521800eb90 tx=0x7f521800ef80 comp rx=0 tx=0).stop 2024-04-22T00:06:12.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 shutdown_connections 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x56534235e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x565342379dc0 0x56534237c250 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 --2- 172.21.15.38:0/1036312321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358020 0x565342360df0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f5209ffb640 0 client.17061 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 >> 172.21.15.38:0/1036312321 conn(0x565342352750 msgr2=0x5653423544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 shutdown_connections 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 1 -- 172.21.15.38:0/1036312321 wait complete. 2024-04-22T00:06:12.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.205+0000 7f522b1ca740 20 client.17061 trim_cache size 0 max 16384 2024-04-22T00:06:12.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.222+0000 7f522b1ca740 1 Processor -- start 2024-04-22T00:06:12.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.222+0000 7f522b1ca740 1 -- start start 2024-04-22T00:06:12.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.222+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 0x56534236d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.222+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.222+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x565342381b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522b1ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52201ac970 con 0x565342358940 2024-04-22T00:06:12.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522b1ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52201ae4b0 con 0x565342358020 2024-04-22T00:06:12.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522b1ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52201ad040 con 0x565342361520 2024-04-22T00:06:12.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5225b79640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5225b79640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59178/0 (socket says 172.21.15.38:59178) 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5225b79640 1 -- 172.21.15.38:0/962679087 learned_addr learned my addr 172.21.15.38:0/962679087 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 0x56534236d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5226b7b640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x565342381b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 -- 172.21.15.38:0/962679087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 msgr2=0x565342381b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x565342381b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 -- 172.21.15.38:0/962679087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 msgr2=0x565342360920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 -- 172.21.15.38:0/962679087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56534209a7d0 con 0x565342358020 2024-04-22T00:06:12.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5226b7b640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x565342381b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:12.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f5225b79640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f522637a640 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 0x56534236d1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52180983e0 tx=0x7f5218018670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c6c5b7aaa4b878 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f520bfff640 1 -- 172.21.15.38:0/962679087 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218010e60 con 0x565342358020 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f520bfff640 1 -- 172.21.15.38:0/962679087 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52180d3530 con 0x565342358020 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.223+0000 7f520bfff640 1 -- 172.21.15.38:0/962679087 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521809b8e0 con 0x565342358020 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- 172.21.15.38:0/962679087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 msgr2=0x56534236d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 0x56534236d1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52180983e0 tx=0x7f5218018670 comp rx=0 tx=0).stop 2024-04-22T00:06:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- 172.21.15.38:0/962679087 shutdown_connections 2024-04-22T00:06:12.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342361520 0x565342381b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342358940 0x565342360920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 --2- 172.21.15.38:0/962679087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358020 0x56534236d1d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- 172.21.15.38:0/962679087 >> 172.21.15.38:0/962679087 conn(0x565342352750 msgr2=0x565342354410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- 172.21.15.38:0/962679087 shutdown_connections 2024-04-22T00:06:12.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- 172.21.15.38:0/962679087 wait complete. 2024-04-22T00:06:12.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 -- start start 2024-04-22T00:06:12.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.224+0000 7f522b1ca740 1 Processor -- start 2024-04-22T00:06:12.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x56534235b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52201adde0 con 0x565342361520 2024-04-22T00:06:12.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52201ad710 con 0x565342358940 2024-04-22T00:06:12.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52201aa950 con 0x565342358020 2024-04-22T00:06:12.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522b1ca740 10 client.0 _ll_register_callbacks cb 0x7f52180099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59188/0 (socket says 172.21.15.38:59188) 2024-04-22T00:06:12.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 -- 172.21.15.38:0/792369052 learned_addr learned my addr 172.21.15.38:0/792369052 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f522637a640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x56534235b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5225b79640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 -- 172.21.15.38:0/792369052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 msgr2=0x56534235b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x56534235b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 -- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 msgr2=0x56534235b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 -- 172.21.15.38:0/792369052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56534235bf80 con 0x565342361520 2024-04-22T00:06:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5225b79640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5226b7b640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f521c009c40 tx=0x7f521c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e1682d7579cbf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5209ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.225+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c036060 con 0x565342361520 2024-04-22T00:06:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f521c019830 con 0x565342361520 2024-04-22T00:06:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52180d3530 con 0x565342361520 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f521c03d050 con 0x565342361520 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 10 client.17067 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5653423e09c0 con 0x565342361520 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52180d3530 con 0x565342361520 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f521c005040 con 0x565342361520 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 10 client.17067 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 10 client.17067 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:06:12.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 20 client.17067 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x5653423e0fa0 0x5653423e3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 4 client.17067 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56534235eac0 con 0x5653423e0fa0 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.226+0000 7f522637a640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x5653423e0fa0 0x5653423e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f5209ffb640 10 client.17067 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522637a640 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x5653423e0fa0 0x5653423e3390 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=edbcdcdddda1382b server_cookie=449447b2338e1c65 in_seq=0 out_seq=0 2024-04-22T00:06:12.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f5209ffb640 1 -- 172.21.15.38:0/792369052 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56534235eac0 con 0x5653423e0fa0 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f5209ffb640 10 client.17067 handle_command_reply: tid=0 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 client.17067 shutdown 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 msgr2=0x5653423df3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f521c009c40 tx=0x7f521c019000 comp rx=0 tx=0).stop 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 shutdown_connections 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x5653423e0fa0 0x5653423e3390 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x565342361520 0x5653423df3b0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565342358940 0x56534235b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f5209ffb640 0 client.17067 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 --2- 172.21.15.38:0/792369052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565342358020 0x56534235b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 >> 172.21.15.38:0/792369052 conn(0x565342352750 msgr2=0x565342358e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 shutdown_connections 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 1 -- 172.21.15.38:0/792369052 wait complete. 2024-04-22T00:06:12.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.227+0000 7f522b1ca740 20 client.17067 trim_cache size 0 max 16384 2024-04-22T00:06:12.244 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 msgr2=0x7f520c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 0x7f520c0a51c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f521c01b000 tx=0x7f521c018be0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 msgr2=0x7f522019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f521800ebc0 tx=0x7f521800c6a0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 shutdown_connections 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f520c0a2d00 0x7f520c0a51c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220109990 0x7f522019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5220104340 0x7f522019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 --2- 172.21.15.38:0/3754754365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5220103980 0x7f522019a300 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 >> 172.21.15.38:0/3754754365 conn(0x7f52200ff4d0 msgr2=0x7f52200ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 shutdown_connections 2024-04-22T00:06:12.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.228+0000 7f52285de640 1 -- 172.21.15.38:0/3754754365 wait complete. 2024-04-22T00:06:12.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:06:12.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 -- 172.21.15.38:0/2011272632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100620 msgr2=0x7fe81c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 --2- 172.21.15.38:0/2011272632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100620 0x7fe81c100aa0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe804009930 tx=0x7fe80402bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 -- 172.21.15.38:0/2011272632 shutdown_connections 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 --2- 172.21.15.38:0/2011272632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c100fe0 0x7fe81c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 --2- 172.21.15.38:0/2011272632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100620 0x7fe81c100aa0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 --2- 172.21.15.38:0/2011272632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c106630 0x7fe81c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 -- 172.21.15.38:0/2011272632 >> 172.21.15.38:0/2011272632 conn(0x7fe81c0776a0 msgr2=0x7fe81c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.361+0000 7fe820fa0640 1 -- 172.21.15.38:0/2011272632 shutdown_connections 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 -- 172.21.15.38:0/2011272632 wait complete. 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 Processor -- start 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 -- start start 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 0x7fe81c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 0x7fe81c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c110a90 con 0x7fe81c100fe0 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c110910 con 0x7fe81c100620 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.362+0000 7fe820fa0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c110c10 con 0x7fe81c106630 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 0x7fe81c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe81ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 0x7fe81c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59198/0 (socket says 172.21.15.38:59198) 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 -- 172.21.15.38:0/3844710128 learned_addr learned my addr 172.21.15.38:0/3844710128 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 -- 172.21.15.38:0/3844710128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 msgr2=0x7fe81c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 0x7fe81c1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 -- 172.21.15.38:0/3844710128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 msgr2=0x7fe81c196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 0x7fe81c196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 -- 172.21.15.38:0/3844710128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c1a64b0 con 0x7fe81c100fe0 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe81a59c640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 0x7fe81c196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 0x7fe81c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe813fff640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe804009900 tx=0x7fe804004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe8137fe640 1 -- 172.21.15.38:0/3844710128 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804043020 con 0x7fe81c100fe0 2024-04-22T00:06:12.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe8137fe640 1 -- 172.21.15.38:0/3844710128 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8040054e0 con 0x7fe81c100fe0 2024-04-22T00:06:12.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c1a6740 con 0x7fe81c100fe0 2024-04-22T00:06:12.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe8137fe640 1 -- 172.21.15.38:0/3844710128 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804003bb0 con 0x7fe81c100fe0 2024-04-22T00:06:12.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.363+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1a6c90 con 0x7fe81c100fe0 2024-04-22T00:06:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.365+0000 7fe8137fe640 1 -- 172.21.15.38:0/3844710128 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe804046020 con 0x7fe81c100fe0 2024-04-22T00:06:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 Processor -- start 2024-04-22T00:06:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 -- start start 2024-04-22T00:06:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe8137fe640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 0x7fe8080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661da5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562661cfe530 con 0x562661d9a020 2024-04-22T00:06:12.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562661ca6830 con 0x562661da3520 2024-04-22T00:06:12.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.366+0000 7fe823b81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562661cfa4f0 con 0x562661d9a940 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 0x7fe8080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59214/0 (socket says 172.21.15.38:59214) 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 -- 172.21.15.38:0/9810052 learned_addr learned my addr 172.21.15.38:0/9810052 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe813fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe8137fe640 1 -- 172.21.15.38:0/3844710128 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe8040f16a0 con 0x7fe81c100fe0 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661da5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 -- 172.21.15.38:0/9810052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 msgr2=0x562661da2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 -- 172.21.15.38:0/9810052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 msgr2=0x562661da5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661da5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 -- 172.21.15.38:0/9810052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562661cca2c0 con 0x562661d9a020 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe813fff640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661da5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81a59c640 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0114c0 tx=0x7fe80c011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302e5e12a78b5d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 0x7fe8080a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x562661d958e0 tx=0x7fe814005bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe810ff9640 1 -- 172.21.15.38:0/9810052 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00f5c0 con 0x562661d9a020 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe810ff9640 1 -- 172.21.15.38:0/9810052 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c00fc40 con 0x562661d9a020 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.367+0000 7fe810ff9640 1 -- 172.21.15.38:0/9810052 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c019cc0 con 0x562661d9a020 2024-04-22T00:06:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- 172.21.15.38:0/9810052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 msgr2=0x562661d9a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0114c0 tx=0x7fe80c011980 comp rx=0 tx=0).stop 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- 172.21.15.38:0/9810052 shutdown_connections 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661da5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 --2- 172.21.15.38:0/9810052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661d9a400 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- 172.21.15.38:0/9810052 >> 172.21.15.38:0/9810052 conn(0x562661d94750 msgr2=0x562661d94b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- 172.21.15.38:0/9810052 shutdown_connections 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- 172.21.15.38:0/9810052 wait complete. 2024-04-22T00:06:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.368+0000 7fe823b81740 1 -- start start 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 Processor -- start 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661d9a940 0x562661da0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661da3520 0x562661da07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562661da6310 con 0x562661d9a020 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562661cb9f10 con 0x562661d9a940 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562661da6490 con 0x562661da3520 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe823b81740 10 client.0 _ll_register_callbacks cb 0x562661a11be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59216/0 (socket says 172.21.15.38:59216) 2024-04-22T00:06:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 -- 172.21.15.38:0/2080005630 learned_addr learned my addr 172.21.15.38:0/2080005630 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661da3520 0x562661da07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe813fff640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661d9a940 0x562661da0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 -- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661da3520 msgr2=0x562661da07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661da3520 0x562661da07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 -- 172.21.15.38:0/2080005630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661d9a940 msgr2=0x562661da0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661d9a940 0x562661da0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 -- 172.21.15.38:0/2080005630 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562661c8bc80 con 0x562661d9a020 2024-04-22T00:06:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe81a59c640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005bc0 tx=0x7fe80c009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c476e4bf216f3639 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.369+0000 7fe801ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c026020 con 0x562661d9a020 2024-04-22T00:06:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe80c019df0 con 0x562661d9a020 2024-04-22T00:06:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80c00fc40 con 0x562661d9a020 2024-04-22T00:06:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe80c031050 con 0x562661d9a020 2024-04-22T00:06:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 10 client.17079 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562661a123f0 con 0x562661d9a020 2024-04-22T00:06:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c00fc40 con 0x562661d9a020 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe80c004750 con 0x562661d9a020 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 10 client.17079 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 10 client.17079 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 20 client.17079 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661db3590 0x562661db5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 4 client.17079 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562661d0ba70 con 0x562661db3590 2024-04-22T00:06:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.370+0000 7fe813fff640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661db3590 0x562661db5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.371+0000 7fe801ffb640 10 client.17079 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.371+0000 7fe813fff640 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661db3590 0x562661db5980 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9887ff36aeeb52e7 server_cookie=2f9ad1fcfc35de43 in_seq=0 out_seq=0 2024-04-22T00:06:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.371+0000 7fe801ffb640 1 -- 172.21.15.38:0/2080005630 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562661d0ba70 con 0x562661db3590 2024-04-22T00:06:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.371+0000 7fe801ffb640 10 client.17079 handle_command_reply: tid=0 2024-04-22T00:06:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.371+0000 7fe823b81740 1 client.17079 shutdown 2024-04-22T00:06:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 msgr2=0x562661da2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005bc0 tx=0x7fe80c009e00 comp rx=0 tx=0).stop 2024-04-22T00:06:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 shutdown_connections 2024-04-22T00:06:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661db3590 0x562661db5980 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661da3520 0x562661da07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661d9a940 0x562661da0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe801ffb640 0 client.17079 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 --2- 172.21.15.38:0/2080005630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661da2e30 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 >> 172.21.15.38:0/2080005630 conn(0x562661d94750 msgr2=0x562661d964a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 shutdown_connections 2024-04-22T00:06:12.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 1 -- 172.21.15.38:0/2080005630 wait complete. 2024-04-22T00:06:12.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.372+0000 7fe823b81740 20 client.17079 trim_cache size 0 max 16384 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.386+0000 7fe823b81740 1 Processor -- start 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.386+0000 7fe823b81740 1 -- start start 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661dbb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b0d00 con 0x562661d9a020 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b2840 con 0x562661da3520 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe823b81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b13d0 con 0x562661d9a940 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661dbb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59230/0 (socket says 172.21.15.38:59230) 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 -- 172.21.15.38:0/1481766166 learned_addr learned my addr 172.21.15.38:0/1481766166 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe813fff640 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 -- 172.21.15.38:0/1481766166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 msgr2=0x562661da2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 -- 172.21.15.38:0/1481766166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 msgr2=0x562661dbb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661dbb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 -- 172.21.15.38:0/1481766166 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562661adc7d0 con 0x562661d9a020 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe81a59c640 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe80c011e80 tx=0x7fe80c009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb54c351268cda9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.387+0000 7fe803fff640 1 -- 172.21.15.38:0/1481766166 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c026040 con 0x562661d9a020 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe803fff640 1 -- 172.21.15.38:0/1481766166 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c019970 con 0x562661d9a020 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe803fff640 1 -- 172.21.15.38:0/1481766166 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c005550 con 0x562661d9a020 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 -- 172.21.15.38:0/1481766166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 msgr2=0x562661daf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe80c011e80 tx=0x7fe80c009d90 comp rx=0 tx=0).stop 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 -- 172.21.15.38:0/1481766166 shutdown_connections 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661dbb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a940 0x562661da2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 --2- 172.21.15.38:0/1481766166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a020 0x562661daf1d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 -- 172.21.15.38:0/1481766166 >> 172.21.15.38:0/1481766166 conn(0x562661d94750 msgr2=0x562661d95910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 -- 172.21.15.38:0/1481766166 shutdown_connections 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.388+0000 7fe823b81740 1 -- 172.21.15.38:0/1481766166 wait complete. 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 -- start start 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 Processor -- start 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a020 0x562661da6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661e11a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b2170 con 0x562661d9a940 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1b1aa0 con 0x562661da3520 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c1aece0 con 0x562661d9a020 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe823b81740 10 client.0 _ll_register_callbacks cb 0x7fe804005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe81a59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a020 0x562661da6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe81ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661e11a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59234/0 (socket says 172.21.15.38:59234) 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 -- 172.21.15.38:0/4271112704 learned_addr learned my addr 172.21.15.38:0/4271112704 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 -- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a020 msgr2=0x562661da6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a020 0x562661da6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 -- 172.21.15.38:0/4271112704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 msgr2=0x562661e11a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661e11a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 -- 172.21.15.38:0/4271112704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562661e11ff0 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe81ad9d640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661e11a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.389+0000 7fe813fff640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe804000c00 tx=0x7fe8040035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5043a5e1d230bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8040bd990 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe804003d50 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80c019970 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe8040cf050 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 10 client.17085 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562661e19400 con 0x562661d9a940 2024-04-22T00:06:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c019970 con 0x562661d9a940 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe8040eed30 con 0x562661d9a940 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 10 client.17085 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 10 client.17085 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 20 client.17085 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661e19b60 0x562661e1bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 4 client.17085 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562661d05dc0 con 0x562661e19b60 2024-04-22T00:06:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.390+0000 7fe81a59c640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661e19b60 0x562661e1bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.391+0000 7fe801ffb640 10 client.17085 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.391+0000 7fe81a59c640 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661e19b60 0x562661e1bf80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d24c55a5da013dc server_cookie=5fdd156ad7df5adc in_seq=0 out_seq=0 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe801ffb640 1 -- 172.21.15.38:0/4271112704 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562661d05dc0 con 0x562661e19b60 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe801ffb640 10 client.17085 handle_command_reply: tid=0 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 client.17085 shutdown 2024-04-22T00:06:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 msgr2=0x562661da6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe804000c00 tx=0x7fe8040035f0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 shutdown_connections 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x562661e19b60 0x562661e1bf80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562661da3520 0x562661e11a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562661d9a940 0x562661da6ed0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe801ffb640 0 client.17085 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 --2- 172.21.15.38:0/4271112704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562661d9a020 0x562661da6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 >> 172.21.15.38:0/4271112704 conn(0x562661d94750 msgr2=0x562661d961a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 shutdown_connections 2024-04-22T00:06:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 1 -- 172.21.15.38:0/4271112704 wait complete. 2024-04-22T00:06:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.392+0000 7fe823b81740 20 client.17085 trim_cache size 0 max 16384 2024-04-22T00:06:12.406 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 msgr2=0x7fe8080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 0x7fe8080a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x562661d958e0 tx=0x7fe814005bd0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 msgr2=0x7fe81c197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe804009900 tx=0x7fe804004360 comp rx=0 tx=0).stop 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 shutdown_connections 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8080a2d00 0x7fe8080a51c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe81c106630 0x7fe81c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe81c100fe0 0x7fe81c197180 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 --2- 172.21.15.38:0/3844710128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c100620 0x7fe81c196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 >> 172.21.15.38:0/3844710128 conn(0x7fe81c0776a0 msgr2=0x7fe81c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 shutdown_connections 2024-04-22T00:06:12.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.393+0000 7fe820fa0640 1 -- 172.21.15.38:0/3844710128 wait complete. 2024-04-22T00:06:12.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:06:12.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.643+0000 7f2085ffb640 1 -- 172.21.15.38:0/3587506610 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207002cc80 con 0x7f2088100610 2024-04-22T00:06:12.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 -- 172.21.15.38:0/3587506610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2088100610 msgr2=0x7f2088100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 --2- 172.21.15.38:0/3587506610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2088100610 0x7f2088100a90 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2070009930 tx=0x7f207002bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:12.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 -- 172.21.15.38:0/3587506610 shutdown_connections 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 --2- 172.21.15.38:0/3587506610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 0x7f208810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 --2- 172.21.15.38:0/3587506610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2088100610 0x7f2088100a90 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 --2- 172.21.15.38:0/3587506610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20881066b0 0x7f2088106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 -- 172.21.15.38:0/3587506610 >> 172.21.15.38:0/3587506610 conn(0x7f2088077740 msgr2=0x7f2088077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 -- 172.21.15.38:0/3587506610 shutdown_connections 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.644+0000 7f208d66d640 1 -- 172.21.15.38:0/3587506610 wait complete. 2024-04-22T00:06:12.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.645+0000 7f208d66d640 1 Processor -- start 2024-04-22T00:06:12.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.646+0000 7f208d66d640 1 -- start start 2024-04-22T00:06:12.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.646+0000 7f208d66d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.646+0000 7f208d66d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 0x7f20881973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.646+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.646+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60662/0 (socket says 172.21.15.38:60662) 2024-04-22T00:06:12.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f2086ffd640 1 -- 172.21.15.38:0/3466474276 learned_addr learned my addr 172.21.15.38:0/3466474276 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f20867fc640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 0x7f20881973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20881066b0 0x7f208819b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2088110a40 con 0x7f20881066b0 2024-04-22T00:06:12.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f20881108c0 con 0x7f2088100610 2024-04-22T00:06:12.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.647+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2088110bc0 con 0x7f2088100fd0 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.648+0000 7f2086ffd640 1 -- 172.21.15.38:0/3466474276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 msgr2=0x7f20881973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.648+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 0x7f20881973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.648+0000 7f2086ffd640 1 -- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20881066b0 msgr2=0x7f208819b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.648+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20881066b0 0x7f208819b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.648+0000 7f2086ffd640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208819be80 con 0x7f2088100610 2024-04-22T00:06:12.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.649+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f208000d770 tx=0x7f208000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.649+0000 7f207ffff640 1 -- 172.21.15.38:0/3466474276 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208001b070 con 0x7f2088100610 2024-04-22T00:06:12.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.649+0000 7f207ffff640 1 -- 172.21.15.38:0/3466474276 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2080014e50 con 0x7f2088100610 2024-04-22T00:06:12.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.650+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208819c170 con 0x7f2088100610 2024-04-22T00:06:12.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.650+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20880688d0 con 0x7f2088100610 2024-04-22T00:06:12.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.651+0000 7f207ffff640 1 -- 172.21.15.38:0/3466474276 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20800035f0 con 0x7f2088100610 2024-04-22T00:06:12.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.653+0000 7f207ffff640 1 -- 172.21.15.38:0/3466474276 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2080004220 con 0x7f2088100610 2024-04-22T00:06:12.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.655+0000 7f209024e740 1 Processor -- start 2024-04-22T00:06:12.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.655+0000 7f209024e740 1 -- start start 2024-04-22T00:06:12.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a571c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a57a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55590a4cd530 con 0x55590a571830 2024-04-22T00:06:12.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55590a475830 con 0x55590a57ad30 2024-04-22T00:06:12.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f209024e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55590a4c94f0 con 0x55590a572150 2024-04-22T00:06:12.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f207ffff640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 0x7f20740a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.656+0000 7f207ffff640 1 -- 172.21.15.38:0/3466474276 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f20800d3400 con 0x7f2088100610 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f20877fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a571c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f20867fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a57a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f20867fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60668/0 (socket says 172.21.15.38:60668) 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f20867fc640 1 -- 172.21.15.38:0/64099918 learned_addr learned my addr 172.21.15.38:0/64099918 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.657+0000 7f20877fe640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 0x7f20740a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 -- 172.21.15.38:0/64099918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 msgr2=0x55590a57a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a57a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 -- 172.21.15.38:0/64099918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 msgr2=0x55590a571c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a571c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 -- 172.21.15.38:0/64099918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590a4992c0 con 0x55590a57ad30 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20877fe640 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a571c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.658+0000 7f20867fc640 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f207002cfd0 tx=0x7f2070005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8447c94efc3f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.659+0000 7f207d7fa640 1 -- 172.21.15.38:0/64099918 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070042070 con 0x55590a57ad30 2024-04-22T00:06:12.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.659+0000 7f207d7fa640 1 -- 172.21.15.38:0/64099918 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f207002ca80 con 0x55590a57ad30 2024-04-22T00:06:12.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.659+0000 7f207d7fa640 1 -- 172.21.15.38:0/64099918 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070004cd0 con 0x55590a57ad30 2024-04-22T00:06:12.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f2086ffd640 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a57a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 -- 172.21.15.38:0/64099918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 msgr2=0x55590a57d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f207002cfd0 tx=0x7f2070005c00 comp rx=0 tx=0).stop 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f20877fe640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 0x7f20740a5270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2078005fd0 tx=0x7f20780095e0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 -- 172.21.15.38:0/64099918 shutdown_connections 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a57ad30 0x55590a57d120 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a57a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 --2- 172.21.15.38:0/64099918 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a571c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.660+0000 7f209024e740 1 -- 172.21.15.38:0/64099918 >> 172.21.15.38:0/64099918 conn(0x55590a56bf60 msgr2=0x55590a56c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 -- 172.21.15.38:0/64099918 shutdown_connections 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 -- 172.21.15.38:0/64099918 wait complete. 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 -- start start 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 Processor -- start 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.661+0000 7f209024e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a572150 0x55590a577a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f209024e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a57ad30 0x55590a577f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f209024e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55590a58ead0 con 0x55590a572150 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f209024e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55590a488f10 con 0x55590a571830 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f209024e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55590a58ec50 con 0x55590a57ad30 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f20867fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a572150 0x55590a577a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f209024e740 10 client.0 _ll_register_callbacks cb 0x55590a1e0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60672/0 (socket says 172.21.15.38:60672) 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 -- 172.21.15.38:0/3515456485 learned_addr learned my addr 172.21.15.38:0/3515456485 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f20877fe640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a57ad30 0x55590a577f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 -- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a57ad30 msgr2=0x55590a577f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a57ad30 0x55590a577f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 -- 172.21.15.38:0/3515456485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a572150 msgr2=0x55590a577a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a572150 0x55590a577a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 -- 172.21.15.38:0/3515456485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590a45ac80 con 0x55590a571830 2024-04-22T00:06:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f2086ffd640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f208009bf50 tx=0x7f208009fdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84030f26b9a56795 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.662+0000 7f206e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:12.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.663+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080024040 con 0x55590a571830 2024-04-22T00:06:12.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.663+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f20800d0a30 con 0x55590a571830 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.663+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f207002ca80 con 0x55590a571830 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.664+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f20800a4e40 con 0x55590a571830 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.664+0000 7f209024e740 10 client.26495 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.664+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55590a1e13f0 con 0x55590a571830 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f207002ca80 con 0x55590a571830 2024-04-22T00:06:12.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f20800a7df0 con 0x55590a571830 2024-04-22T00:06:12.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 10 client.26495 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 10 client.26495 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:06:12.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 20 client.26495 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a5642b0 0x55590a566770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 4 client.26495 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55590a4daa70 con 0x55590a5642b0 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.665+0000 7f20867fc640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a5642b0 0x55590a566770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.666+0000 7f20867fc640 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a5642b0 0x55590a566770 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de1cd5c6f5be14d8 server_cookie=33fe0c9707e1a147 in_seq=0 out_seq=0 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.666+0000 7f206e7fc640 10 client.26495 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.667+0000 7f206e7fc640 1 -- 172.21.15.38:0/3515456485 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55590a4daa70 con 0x55590a5642b0 2024-04-22T00:06:12.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.667+0000 7f206e7fc640 10 client.26495 handle_command_reply: tid=0 2024-04-22T00:06:12.665 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.667+0000 7f209024e740 1 client.26495 shutdown 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 msgr2=0x55590a57a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f208009bf50 tx=0x7f208009fdb0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 shutdown_connections 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a5642b0 0x55590a566770 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f206e7fc640 0 client.26495 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a57ad30 0x55590a577f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a572150 0x55590a577a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 --2- 172.21.15.38:0/3515456485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a571830 0x55590a57a680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 >> 172.21.15.38:0/3515456485 conn(0x55590a56bf60 msgr2=0x55590a56dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 shutdown_connections 2024-04-22T00:06:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 1 -- 172.21.15.38:0/3515456485 wait complete. 2024-04-22T00:06:12.668 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.668+0000 7f209024e740 20 client.26495 trim_cache size 0 max 16384 2024-04-22T00:06:12.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.691+0000 7f209024e740 1 Processor -- start 2024-04-22T00:06:12.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.691+0000 7f209024e740 1 -- start start 2024-04-22T00:06:12.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a571830 0x55590a57a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 0x55590a57a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f206800fcd0 con 0x55590a5642b0 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2068011810 con 0x55590a572150 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f209024e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20680103a0 con 0x55590a571830 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f20877fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 0x55590a57a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f2086ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59286/0 (socket says 172.21.15.38:59286) 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f2086ffd640 1 -- 172.21.15.38:0/1501918829 learned_addr learned my addr 172.21.15.38:0/1501918829 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.692+0000 7f20877fe640 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 0x55590a57a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60678/0 (socket says 172.21.15.38:60678) 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 -- 172.21.15.38:0/1501918829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a571830 msgr2=0x55590a57a140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a571830 0x55590a57a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 -- 172.21.15.38:0/1501918829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 msgr2=0x55590a57a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 0x55590a57a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 -- 172.21.15.38:0/1501918829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590a2ab7d0 con 0x55590a5642b0 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.693+0000 7f2086ffd640 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f208009f510 tx=0x7f20800a3ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3159e71146878d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.694+0000 7f207cff9640 1 -- 172.21.15.38:0/1501918829 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080024070 con 0x55590a5642b0 2024-04-22T00:06:12.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.694+0000 7f207cff9640 1 -- 172.21.15.38:0/1501918829 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20800d3170 con 0x55590a5642b0 2024-04-22T00:06:12.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.694+0000 7f207cff9640 1 -- 172.21.15.38:0/1501918829 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20800a7d60 con 0x55590a5642b0 2024-04-22T00:06:12.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 -- 172.21.15.38:0/1501918829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 msgr2=0x55590a56da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f208009f510 tx=0x7f20800a3ab0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 -- 172.21.15.38:0/1501918829 shutdown_connections 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a572150 0x55590a57a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a571830 0x55590a57a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 --2- 172.21.15.38:0/1501918829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a5642b0 0x55590a56da50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.695+0000 7f209024e740 1 -- 172.21.15.38:0/1501918829 >> 172.21.15.38:0/1501918829 conn(0x55590a56bf60 msgr2=0x55590a56d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.696+0000 7f209024e740 1 -- 172.21.15.38:0/1501918829 shutdown_connections 2024-04-22T00:06:12.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.696+0000 7f209024e740 1 -- 172.21.15.38:0/1501918829 wait complete. 2024-04-22T00:06:12.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.697+0000 7f209024e740 1 -- start start 2024-04-22T00:06:12.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.697+0000 7f209024e740 1 Processor -- start 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a5642b0 0x55590a59c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a59cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a5f8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2068011140 con 0x55590a571830 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2068010a70 con 0x55590a5642b0 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f206800dcb0 con 0x55590a572150 2024-04-22T00:06:12.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f209024e740 10 client.0 _ll_register_callbacks cb 0x7f208000d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f20877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a5f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f20877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a5f8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:59382/0 (socket says 172.21.15.38:59382) 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f20877fe640 1 -- 172.21.15.38:0/255244551 learned_addr learned my addr 172.21.15.38:0/255244551 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.698+0000 7f20867fc640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a59cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f2086ffd640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a5642b0 0x55590a59c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f20867fc640 1 -- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 msgr2=0x55590a5f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f20867fc640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a5f8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f20867fc640 1 -- 172.21.15.38:0/255244551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a5642b0 msgr2=0x55590a59c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f20867fc640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a5642b0 0x55590a59c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.699+0000 7f20867fc640 1 -- 172.21.15.38:0/255244551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590a59d670 con 0x55590a571830 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.700+0000 7f20867fc640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a59cda0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2070009900 tx=0x7f2070005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82651cf76f0f15b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.700+0000 7f206e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:12.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.700+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070042040 con 0x55590a571830 2024-04-22T00:06:12.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.700+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f207002cbd0 con 0x55590a571830 2024-04-22T00:06:12.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.700+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20800d3170 con 0x55590a571830 2024-04-22T00:06:12.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.701+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f20700049b0 con 0x55590a571830 2024-04-22T00:06:12.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.701+0000 7f209024e740 10 client.17100 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.701+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55590a5f99f0 con 0x55590a571830 2024-04-22T00:06:12.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20800d3170 con 0x55590a571830 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2070005260 con 0x55590a571830 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 10 client.17100 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 10 client.17100 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 20 client.17100 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a57ad30 0x55590a5fbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 4 client.17100 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.702+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55590a4d4dc0 con 0x55590a57ad30 2024-04-22T00:06:12.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.703+0000 7f2086ffd640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a57ad30 0x55590a5fbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f2086ffd640 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a57ad30 0x55590a5fbd80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7208ab696664549 server_cookie=6abff1b2942bead7 in_seq=0 out_seq=0 2024-04-22T00:06:12.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f206e7fc640 10 client.17100 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:12.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f206e7fc640 1 -- 172.21.15.38:0/255244551 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55590a4d4dc0 con 0x55590a57ad30 2024-04-22T00:06:12.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f206e7fc640 10 client.17100 handle_command_reply: tid=0 2024-04-22T00:06:12.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f209024e740 1 client.17100 shutdown 2024-04-22T00:06:12.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 msgr2=0x55590a59cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.704+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a59cda0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2070009900 tx=0x7f2070005e80 comp rx=0 tx=0).stop 2024-04-22T00:06:12.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 shutdown_connections 2024-04-22T00:06:12.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55590a57ad30 0x55590a5fbd80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f206e7fc640 0 client.17100 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55590a572150 0x55590a5f8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55590a571830 0x55590a59cda0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 --2- 172.21.15.38:0/255244551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55590a5642b0 0x55590a59c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 >> 172.21.15.38:0/255244551 conn(0x55590a56bf60 msgr2=0x55590a5977c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 shutdown_connections 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.705+0000 7f209024e740 1 -- 172.21.15.38:0/255244551 wait complete. 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.706+0000 7f209024e740 20 client.17100 trim_cache size 0 max 16384 2024-04-22T00:06:12.700 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 msgr2=0x7f20740a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 0x7f20740a5270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2078005fd0 tx=0x7f20780095e0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 msgr2=0x7f2088196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f208000d770 tx=0x7f208000dc30 comp rx=0 tx=0).stop 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 shutdown_connections 2024-04-22T00:06:12.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f20740a2db0 0x7f20740a5270 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20881066b0 0x7f208819b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2088100fd0 0x7f20881973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 --2- 172.21.15.38:0/3466474276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2088100610 0x7f2088196e90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.711+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 >> 172.21.15.38:0/3466474276 conn(0x7f2088077740 msgr2=0x7f20880ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.713+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 shutdown_connections 2024-04-22T00:06:12.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.713+0000 7f208d66d640 1 -- 172.21.15.38:0/3466474276 wait complete. 2024-04-22T00:06:12.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:12 smithi038 ceph-mon[27972]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.5 MiB/s wr, 2.36k op/s 2024-04-22T00:06:12.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:12 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1564575861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:12.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:06:12.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:12 smithi203 ceph-mon[31431]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.5 MiB/s wr, 2.36k op/s 2024-04-22T00:06:12.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:12 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1564575861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:12.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:12 smithi111 ceph-mon[31543]: pgmap v126: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.5 MiB/s wr, 2.36k op/s 2024-04-22T00:06:12.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:12 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1564575861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:12.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 -- 172.21.15.38:0/4272748766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c01066b0 msgr2=0x7ff3c0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/4272748766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c01066b0 0x7ff3c0106a90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009930 tx=0x7ff3bc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:12.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 -- 172.21.15.38:0/4272748766 shutdown_connections 2024-04-22T00:06:12.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/4272748766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 0x7ff3c010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/4272748766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c0100610 0x7ff3c0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/4272748766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c01066b0 0x7ff3c0106a90 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 -- 172.21.15.38:0/4272748766 >> 172.21.15.38:0/4272748766 conn(0x7ff3c0077740 msgr2=0x7ff3c0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 -- 172.21.15.38:0/4272748766 shutdown_connections 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.868+0000 7ff3c83fe640 1 -- 172.21.15.38:0/4272748766 wait complete. 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 Processor -- start 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 -- start start 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 0x7ff3c010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 0x7ff3c010e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0111b30 con 0x7ff3c0100610 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01119b0 con 0x7ff3c01066b0 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c83fe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c0111cb0 con 0x7ff3c0100fd0 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59318/0 (socket says 172.21.15.38:59318) 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 -- 172.21.15.38:0/1489922597 learned_addr learned my addr 172.21.15.38:0/1489922597 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c5999640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 0x7ff3c010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c699b640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 0x7ff3c010e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 -- 172.21.15.38:0/1489922597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 msgr2=0x7ff3c010e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 0x7ff3c010e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 -- 172.21.15.38:0/1489922597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 msgr2=0x7ff3c010e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 0x7ff3c010e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c619a640 1 -- 172.21.15.38:0/1489922597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c01a6470 con 0x7ff3c0100610 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.869+0000 7ff3c699b640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 0x7ff3c010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3c619a640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009230 tx=0x7ff3bc02cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3b77fe640 1 -- 172.21.15.38:0/1489922597 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc041070 con 0x7ff3c0100610 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3b77fe640 1 -- 172.21.15.38:0/1489922597 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc004450 con 0x7ff3c0100610 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c01a6700 con 0x7ff3c0100610 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3b77fe640 1 -- 172.21.15.38:0/1489922597 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc003d90 con 0x7ff3c0100610 2024-04-22T00:06:12.870 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.870+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c01a6bf0 con 0x7ff3c0100610 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.873+0000 7ff3b77fe640 1 -- 172.21.15.38:0/1489922597 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff3bc044070 con 0x7ff3c0100610 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.873+0000 7ff3cafe1740 1 Processor -- start 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 -- start start 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 0x564031370fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031373910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5640312cc530 con 0x564031371520 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564031274830 con 0x564031368020 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5640312c84f0 con 0x564031368940 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3c699b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3c619a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 0x564031370fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3c5999640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031373910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3c699b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60702/0 (socket says 172.21.15.38:60702) 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.874+0000 7ff3c699b640 1 -- 172.21.15.38:0/2186654068 learned_addr learned my addr 172.21.15.38:0/2186654068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3b77fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 0x7ff3ac0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c699b640 1 -- 172.21.15.38:0/2186654068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 msgr2=0x564031370fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c699b640 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 0x564031370fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c699b640 1 -- 172.21.15.38:0/2186654068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 msgr2=0x564031373910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c699b640 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031373910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c699b640 1 -- 172.21.15.38:0/2186654068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640312982c0 con 0x564031368020 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3b77fe640 1 -- 172.21.15.38:0/1489922597 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3bc0f1200 con 0x7ff3c0100610 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.875+0000 7ff3c619a640 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 0x564031370fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.876+0000 7ff3c699b640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 0x7ff3ac0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.876+0000 7ff3c699b640 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009fd0 tx=0x7ff3b8010a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1a0bd8499755fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.876+0000 7ff3b4bfe640 1 -- 172.21.15.38:0/2186654068 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8019070 con 0x564031368020 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.876+0000 7ff3b4bfe640 1 -- 172.21.15.38:0/2186654068 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b80134e0 con 0x564031368020 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.876+0000 7ff3b4bfe640 1 -- 172.21.15.38:0/2186654068 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b801d7a0 con 0x564031368020 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.877+0000 7ff3cafe1740 1 -- 172.21.15.38:0/2186654068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 msgr2=0x564031368400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.877+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009fd0 tx=0x7ff3b8010a00 comp rx=0 tx=0).stop 2024-04-22T00:06:12.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.877+0000 7ff3c699b640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 0x7ff3ac0a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8020000 tx=0x7ff3b801ddf0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 -- 172.21.15.38:0/2186654068 shutdown_connections 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031373910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368940 0x564031370fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/2186654068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368020 0x564031368400 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 -- 172.21.15.38:0/2186654068 >> 172.21.15.38:0/2186654068 conn(0x564031362750 msgr2=0x564031362b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 -- 172.21.15.38:0/2186654068 shutdown_connections 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.878+0000 7ff3cafe1740 1 -- 172.21.15.38:0/2186654068 wait complete. 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 -- start start 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 Processor -- start 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368020 0x564031370e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031371520 0x56403136e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564031374310 con 0x564031368020 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564031287f10 con 0x564031368940 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564031374490 con 0x564031371520 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.879+0000 7ff3cafe1740 10 client.0 _ll_register_callbacks cb 0x564030fdfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:60718/0 (socket says 172.21.15.38:60718) 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 -- 172.21.15.38:0/51897968 learned_addr learned my addr 172.21.15.38:0/51897968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c699b640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031371520 0x56403136e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c619a640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368020 0x564031370e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 -- 172.21.15.38:0/51897968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031371520 msgr2=0x56403136e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031371520 0x56403136e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 -- 172.21.15.38:0/51897968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368020 msgr2=0x564031370e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368020 0x564031370e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.880+0000 7ff3c5999640 1 -- 172.21.15.38:0/51897968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564031259c80 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.881+0000 7ff3c5999640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0013710 tx=0x7ff3b0013bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b35358577e1cee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.881+0000 7ff3a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0011940 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3b0011ae0 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3b80134e0 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3b0011ce0 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3cafe1740 10 client.26510 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.882+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564030fe03f0 con 0x564031368940 2024-04-22T00:06:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b80134e0 con 0x564031368940 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff3b001a070 con 0x564031368940 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 10 client.26510 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 10 client.26510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 20 client.26510 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56403138a070 0x56403138c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 4 client.26510 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640312d9a70 con 0x56403138a070 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.883+0000 7ff3c619a640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56403138a070 0x56403138c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.884+0000 7ff3c619a640 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56403138a070 0x56403138c460 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa3389759269e84 server_cookie=39aaf1d225295ccd in_seq=0 out_seq=0 2024-04-22T00:06:12.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.884+0000 7ff3a9ffb640 10 client.26510 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.884+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/51897968 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640312d9a70 con 0x56403138a070 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.884+0000 7ff3a9ffb640 10 client.26510 handle_command_reply: tid=0 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 client.26510 shutdown 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 msgr2=0x56403136e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0013710 tx=0x7ff3b0013bd0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 shutdown_connections 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031371520 0x56403136e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x56403136e270 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56403138a070 0x56403138c460 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/51897968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368020 0x564031370e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 >> 172.21.15.38:0/51897968 conn(0x564031362750 msgr2=0x5640313644a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3a9ffb640 0 client.26510 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 shutdown_connections 2024-04-22T00:06:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 1 -- 172.21.15.38:0/51897968 wait complete. 2024-04-22T00:06:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.885+0000 7ff3cafe1740 20 client.26510 trim_cache size 0 max 16384 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 Processor -- start 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 -- start start 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x56403137d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x5640313700a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01ac8b0 con 0x564031371520 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01ae3f0 con 0x564031368940 2024-04-22T00:06:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01acf80 con 0x564031368020 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3c699b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3c699b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59350/0 (socket says 172.21.15.38:59350) 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3c699b640 1 -- 172.21.15.38:0/688691311 learned_addr learned my addr 172.21.15.38:0/688691311 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3c5999640 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x5640313700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.899+0000 7ff3c619a640 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x56403137d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 -- 172.21.15.38:0/688691311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 msgr2=0x56403137d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x56403137d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 -- 172.21.15.38:0/688691311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 msgr2=0x5640313700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x5640313700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 -- 172.21.15.38:0/688691311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640310aa7d0 con 0x564031371520 2024-04-22T00:06:12.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3c699b640 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8013e60 tx=0x7ff3b8028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ff00d021246e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3abfff640 1 -- 172.21.15.38:0/688691311 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8019050 con 0x564031371520 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3abfff640 1 -- 172.21.15.38:0/688691311 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b8036070 con 0x564031371520 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.900+0000 7ff3abfff640 1 -- 172.21.15.38:0/688691311 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b802d040 con 0x564031371520 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- 172.21.15.38:0/688691311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 msgr2=0x564031391ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8013e60 tx=0x7ff3b8028000 comp rx=0 tx=0).stop 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- 172.21.15.38:0/688691311 shutdown_connections 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031371520 0x564031391ba0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031368940 0x5640313700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/688691311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x56403137d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- 172.21.15.38:0/688691311 >> 172.21.15.38:0/688691311 conn(0x564031362750 msgr2=0x56403136f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- 172.21.15.38:0/688691311 shutdown_connections 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- 172.21.15.38:0/688691311 wait complete. 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 -- start start 2024-04-22T00:06:12.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.901+0000 7ff3cafe1740 1 Processor -- start 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x5640313750d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 0x564031375610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031371520 0x564031375b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01add20 con 0x564031368940 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c5999640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 0x564031375610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c5999640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 0x564031375610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59360/0 (socket says 172.21.15.38:59360) 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c5999640 1 -- 172.21.15.38:0/874976962 learned_addr learned my addr 172.21.15.38:0/874976962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01ad650 con 0x564031371520 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c01aa890 con 0x564031368020 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x5640313750d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3cafe1740 10 client.0 _ll_register_callbacks cb 0x7ff3b0009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c699b640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031371520 0x564031375b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 -- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031371520 msgr2=0x564031375b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031371520 0x564031375b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 -- 172.21.15.38:0/874976962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 msgr2=0x564031375610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 0x564031375610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640313ef410 con 0x564031368020 2024-04-22T00:06:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3c619a640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x5640313750d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0ee6c0 tx=0x7ff3bc0f35d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c86a23c03efe7d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.902+0000 7ff3a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc04b040 con 0x564031368020 2024-04-22T00:06:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3b8036070 con 0x564031368020 2024-04-22T00:06:12.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3bc0035f0 con 0x564031368020 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3bc0bd260 con 0x564031368020 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 10 client.26386 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5640313f06e0 con 0x564031368020 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b8036070 con 0x564031368020 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff3bc0ee830 con 0x564031368020 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 10 client.26386 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 10 client.26386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:12.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 20 client.26386 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640313f0ea0 0x5640313f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 4 client.26386 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.903+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640312d3dc0 con 0x5640313f0ea0 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3c5999640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640313f0ea0 0x5640313f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3a9ffb640 10 client.26386 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3c5999640 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640313f0ea0 0x5640313f32c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=692c1fdc7b10bf5d server_cookie=d6248be51a2a0b57 in_seq=0 out_seq=0 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3a9ffb640 1 -- 172.21.15.38:0/874976962 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640312d3dc0 con 0x5640313f0ea0 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3a9ffb640 10 client.26386 handle_command_reply: tid=0 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 client.26386 shutdown 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 msgr2=0x5640313750d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x5640313750d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0ee6c0 tx=0x7ff3bc0f35d0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 shutdown_connections 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564031371520 0x564031375b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640313f0ea0 0x5640313f32c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564031368940 0x564031375610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3a9ffb640 0 client.26386 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 --2- 172.21.15.38:0/874976962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564031368020 0x5640313750d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 >> 172.21.15.38:0/874976962 conn(0x564031362750 msgr2=0x564031370140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 shutdown_connections 2024-04-22T00:06:12.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 1 -- 172.21.15.38:0/874976962 wait complete. 2024-04-22T00:06:12.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.904+0000 7ff3cafe1740 20 client.26386 trim_cache size 0 max 16384 2024-04-22T00:06:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:12.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 msgr2=0x7ff3ac0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 0x7ff3ac0a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8020000 tx=0x7ff3b801ddf0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 msgr2=0x7ff3c010df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009230 tx=0x7ff3bc02cb20 comp rx=0 tx=0).stop 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 shutdown_connections 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3ac0a2d80 0x7ff3ac0a5240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3c01066b0 0x7ff3c010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3c0100fd0 0x7ff3c010e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 --2- 172.21.15.38:0/1489922597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3c0100610 0x7ff3c010df70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:12.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 >> 172.21.15.38:0/1489922597 conn(0x7ff3c0077740 msgr2=0x7ff3c00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:12.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 shutdown_connections 2024-04-22T00:06:12.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:12.905+0000 7ff3c83fe640 1 -- 172.21.15.38:0/1489922597 wait complete. 2024-04-22T00:06:13.916 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:06:13.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:06:13.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:06:14.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 -- 172.21.15.38:0/954007682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 msgr2=0x7f0e18100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 --2- 172.21.15.38:0/954007682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18100aa0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009930 tx=0x7f0e0402bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 -- 172.21.15.38:0/954007682 shutdown_connections 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 --2- 172.21.15.38:0/954007682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 0x7f0e1810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 --2- 172.21.15.38:0/954007682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18100aa0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 --2- 172.21.15.38:0/954007682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 0x7f0e18106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 -- 172.21.15.38:0/954007682 >> 172.21.15.38:0/954007682 conn(0x7f0e180776a0 msgr2=0x7f0e18077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 -- 172.21.15.38:0/954007682 shutdown_connections 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.067+0000 7f0e1f082640 1 -- 172.21.15.38:0/954007682 wait complete. 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 Processor -- start 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 -- start start 2024-04-22T00:06:14.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 0x7f0e18199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 0x7f0e181a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18110a40 con 0x7f0e18100620 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181108c0 con 0x7f0e18106630 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1f082640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e18110bc0 con 0x7f0e18100fe0 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54954/0 (socket says 172.21.15.38:54954) 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/2959433439 learned_addr learned my addr 172.21.15.38:0/2959433439 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e17fff640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 0x7f0e18199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 0x7f0e181a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/2959433439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 msgr2=0x7f0e18199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 0x7f0e18199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/2959433439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 msgr2=0x7f0e181a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 0x7f0e181a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.068+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/2959433439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e181a4400 con 0x7f0e18100620 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00d950 tx=0x7f0e0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e15ffb640 1 -- 172.21.15.38:0/2959433439 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c017030 con 0x7f0e18100620 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e181a45d0 con 0x7f0e18100620 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e15ffb640 1 -- 172.21.15.38:0/2959433439 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0c004020 con 0x7f0e18100620 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e15ffb640 1 -- 172.21.15.38:0/2959433439 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c0047a0 con 0x7f0e18100620 2024-04-22T00:06:14.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.069+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e181a4b40 con 0x7f0e18100620 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e15ffb640 1 -- 172.21.15.38:0/2959433439 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0e0c0041d0 con 0x7f0e18100620 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 Processor -- start 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 -- start start 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e15ffb640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 0x7f0e080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4ccdb0 0x56158e4cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56158e427170 con 0x56158e4c4720 2024-04-22T00:06:14.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56158e3cf7c0 con 0x56158e4ccdb0 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e21c60740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56158e423130 con 0x56158e4c3a40 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e17fff640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 0x7f0e080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54964/0 (socket says 172.21.15.38:54964) 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e1d61f640 1 -- 172.21.15.38:0/1578029303 learned_addr learned my addr 172.21.15.38:0/1578029303 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.072+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4ccdb0 0x56158e4cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e15ffb640 1 -- 172.21.15.38:0/2959433439 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e0c0cf070 con 0x7f0e18100620 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 -- 172.21.15.38:0/1578029303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 msgr2=0x56158e4c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 -- 172.21.15.38:0/1578029303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4ccdb0 msgr2=0x56158e4cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4ccdb0 0x56158e4cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 -- 172.21.15.38:0/1578029303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158e3e0e60 con 0x56158e4c4720 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e17fff640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 0x7f0e080a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005ab0 tx=0x7f0e04005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0e10013b60 tx=0x7f0e10009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71e3e00a87bdc70 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e02ffd640 1 -- 172.21.15.38:0/1578029303 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10013130 con 0x56158e4c4720 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e02ffd640 1 -- 172.21.15.38:0/1578029303 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e10013c50 con 0x56158e4c4720 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.073+0000 7f0e02ffd640 1 -- 172.21.15.38:0/1578029303 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10005440 con 0x56158e4c4720 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- 172.21.15.38:0/1578029303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 msgr2=0x56158e4c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0e10013b60 tx=0x7f0e10009b30 comp rx=0 tx=0).stop 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- 172.21.15.38:0/1578029303 shutdown_connections 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4ccdb0 0x56158e4cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 --2- 172.21.15.38:0/1578029303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4c3470 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- 172.21.15.38:0/1578029303 >> 172.21.15.38:0/1578029303 conn(0x56158e48a4f0 msgr2=0x56158e4be490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- 172.21.15.38:0/1578029303 shutdown_connections 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- 172.21.15.38:0/1578029303 wait complete. 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 -- start start 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.074+0000 7f0e21c60740 1 Processor -- start 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c4720 0x56158e4ca320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56158e4cfbc0 con 0x56158e4ccdb0 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56158e3e2d40 con 0x56158e4c4720 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56158e4cfd40 con 0x56158e4c3a40 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 10 client.0 _ll_register_callbacks cb 0x56158e41e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54972/0 (socket says 172.21.15.38:54972) 2024-04-22T00:06:14.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 -- 172.21.15.38:0/2418218672 learned_addr learned my addr 172.21.15.38:0/2418218672 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e17fff640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c4720 0x56158e4ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 -- 172.21.15.38:0/2418218672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 msgr2=0x56158e4c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 -- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c4720 msgr2=0x56158e4ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c4720 0x56158e4ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 -- 172.21.15.38:0/2418218672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158e2037e0 con 0x56158e4ccdb0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000fc90 tx=0x7f0e10004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bafb35b69920900 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0df7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e100053d0 con 0x56158e4ccdb0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e10005570 con 0x56158e4ccdb0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.075+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e10013c50 con 0x56158e4ccdb0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0e10030050 con 0x56158e4ccdb0 2024-04-22T00:06:14.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 10 client.17127 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56158e263f70 con 0x56158e4ccdb0 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e10013c50 con 0x56158e4ccdb0 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0e10010a10 con 0x56158e4ccdb0 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 10 client.17127 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 20 client.17127 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e4dcdd0 0x56158e4df1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 4 client.17127 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158e3fb450 con 0x56158e4dcdd0 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.076+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e4dcdd0 0x56158e4df1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0df7fff640 10 client.17127 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e4dcdd0 0x56158e4df1c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ab12882def2227e server_cookie=791083b3722cb0d0 in_seq=0 out_seq=0 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0df7fff640 1 -- 172.21.15.38:0/2418218672 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56158e3fb450 con 0x56158e4dcdd0 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0df7fff640 10 client.17127 handle_command_reply: tid=0 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 client.17127 shutdown 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 msgr2=0x56158e4ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000fc90 tx=0x7f0e10004cf0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 shutdown_connections 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e4dcdd0 0x56158e4df1c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4ca860 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0df7fff640 0 client.17127 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c4720 0x56158e4ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 --2- 172.21.15.38:0/2418218672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c3a40 0x56158e4c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.077+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 >> 172.21.15.38:0/2418218672 conn(0x56158e48a4f0 msgr2=0x56158e4bf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.078+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 shutdown_connections 2024-04-22T00:06:14.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.078+0000 7f0e21c60740 1 -- 172.21.15.38:0/2418218672 wait complete. 2024-04-22T00:06:14.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.078+0000 7f0e21c60740 20 client.17127 trim_cache size 0 max 16384 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 Processor -- start 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 -- start start 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c4720 0x56158e4bff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181aead0 con 0x56158e4ccdb0 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181b0610 con 0x56158e4c3a40 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e21c60740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181af1a0 con 0x56158e4c4720 2024-04-22T00:06:14.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.095+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54978/0 (socket says 172.21.15.38:54978) 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 -- 172.21.15.38:0/464137512 learned_addr learned my addr 172.21.15.38:0/464137512 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e17fff640 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c4720 0x56158e4bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 -- 172.21.15.38:0/464137512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c4720 msgr2=0x56158e4bff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c4720 0x56158e4bff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 -- 172.21.15.38:0/464137512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 msgr2=0x56158e4cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 -- 172.21.15.38:0/464137512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158e0fb170 con 0x56158e4ccdb0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009b70 tx=0x7f0e10005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4651954792fa0be5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e027fc640 1 -- 172.21.15.38:0/464137512 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10013440 con 0x56158e4ccdb0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e027fc640 1 -- 172.21.15.38:0/464137512 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e10017040 con 0x56158e4ccdb0 2024-04-22T00:06:14.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.096+0000 7f0e027fc640 1 -- 172.21.15.38:0/464137512 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e10013c50 con 0x56158e4ccdb0 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- 172.21.15.38:0/464137512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 msgr2=0x56158e4e48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009b70 tx=0x7f0e10005bd0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- 172.21.15.38:0/464137512 shutdown_connections 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4ccdb0 0x56158e4e48d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4c4720 0x56158e4bff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 --2- 172.21.15.38:0/464137512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- 172.21.15.38:0/464137512 >> 172.21.15.38:0/464137512 conn(0x56158e48a4f0 msgr2=0x56158e4bf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- 172.21.15.38:0/464137512 shutdown_connections 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- 172.21.15.38:0/464137512 wait complete. 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 -- start start 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.097+0000 7f0e21c60740 1 Processor -- start 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4ccdb0 0x56158e4d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181aff40 con 0x56158e4c4720 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181af870 con 0x56158e4c3a40 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e181acab0 con 0x56158e4ccdb0 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 10 client.0 _ll_register_callbacks cb 0x7f0e10013620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54988/0 (socket says 172.21.15.38:54988) 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e17fff640 1 -- 172.21.15.38:0/3691710352 learned_addr learned my addr 172.21.15.38:0/3691710352 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1d61f640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4ccdb0 0x56158e4d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/3691710352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4ccdb0 msgr2=0x56158e4d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4ccdb0 0x56158e4d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/3691710352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 msgr2=0x56158e4d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 -- 172.21.15.38:0/3691710352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158e542170 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e1ce1e640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4d0a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0cc480 tx=0x7f0e0c018f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49f9b32928dd1754 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0df7fff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c09b250 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e0c0978c0 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.098+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e10017040 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0e0c09f600 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 10 client.26534 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56158e543470 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e10017040 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0e0c0a3a30 con 0x56158e4c3a40 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 10 client.26534 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:14.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 10 client.26534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 20 client.26534 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e543b50 0x56158e545f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 4 client.26534 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56158e4346b0 con 0x56158e543b50 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e17fff640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e543b50 0x56158e545f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0df7fff640 10 client.26534 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.099+0000 7f0e17fff640 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e543b50 0x56158e545f90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58010d0cf9fccda0 server_cookie=3a12092583e59434 in_seq=0 out_seq=0 2024-04-22T00:06:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0df7fff640 1 -- 172.21.15.38:0/3691710352 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56158e4346b0 con 0x56158e543b50 2024-04-22T00:06:14.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0df7fff640 10 client.26534 handle_command_reply: tid=0 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 client.26534 shutdown 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 msgr2=0x56158e4d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4d0a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0cc480 tx=0x7f0e0c018f10 comp rx=0 tx=0).stop 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 shutdown_connections 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56158e4ccdb0 0x56158e4d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56158e543b50 0x56158e545f90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56158e4c4720 0x56158e4d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 --2- 172.21.15.38:0/3691710352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56158e4c3a40 0x56158e4d0a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0df7fff640 0 client.26534 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.100+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 >> 172.21.15.38:0/3691710352 conn(0x56158e48a4f0 msgr2=0x56158e4c0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 shutdown_connections 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e21c60740 1 -- 172.21.15.38:0/3691710352 wait complete. 2024-04-22T00:06:14.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e21c60740 20 client.26534 trim_cache size 0 max 16384 2024-04-22T00:06:14.102 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:06:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "4900014b-af7f-4a3a-8bd7-3a3ed008f558", 2024-04-22T00:06:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:06:14.102 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 msgr2=0x7f0e080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 0x7f0e080a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005ab0 tx=0x7f0e04005fd0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 msgr2=0x7f0e18198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00d950 tx=0x7f0e0c00de10 comp rx=0 tx=0).stop 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 shutdown_connections 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e080a2d00 0x7f0e080a51c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18106630 0x7f0e181a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e18100fe0 0x7f0e18199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 --2- 172.21.15.38:0/2959433439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e18100620 0x7f0e18198e40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 >> 172.21.15.38:0/2959433439 conn(0x7f0e180776a0 msgr2=0x7f0e181031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 shutdown_connections 2024-04-22T00:06:14.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.101+0000 7f0e1f082640 1 -- 172.21.15.38:0/2959433439 wait complete. 2024-04-22T00:06:14.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:06:14.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:06:14.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 -- 172.21.15.38:0/1016151406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 msgr2=0x7f7cb0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/1016151406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0100aa0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009930 tx=0x7f7ca402bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 -- 172.21.15.38:0/1016151406 shutdown_connections 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/1016151406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0100fe0 0x7f7cb010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/1016151406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0100aa0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/1016151406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0106630 0x7f7cb0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 -- 172.21.15.38:0/1016151406 >> 172.21.15.38:0/1016151406 conn(0x7f7cb00776a0 msgr2=0x7f7cb0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 -- 172.21.15.38:0/1016151406 shutdown_connections 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.218+0000 7f7cb4b25640 1 -- 172.21.15.38:0/1016151406 wait complete. 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 Processor -- start 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 -- start start 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0100fe0 0x7f7cb01993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 0x7f7cb01a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0110a40 con 0x7f7cb0100620 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01108c0 con 0x7f7cb0106630 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cb4b25640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0110bc0 con 0x7f7cb0100fe0 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55002/0 (socket says 172.21.15.38:55002) 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cae59c640 1 -- 172.21.15.38:0/2090602828 learned_addr learned my addr 172.21.15.38:0/2090602828 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 0x7f7cb01a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0100fe0 0x7f7cb01993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 -- 172.21.15.38:0/2090602828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0100fe0 msgr2=0x7f7cb01993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0100fe0 0x7f7cb01993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 -- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 msgr2=0x7f7cb0198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7caed9d640 1 -- 172.21.15.38:0/2090602828 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb01a4440 con 0x7f7cb0106630 2024-04-22T00:06:14.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.219+0000 7f7cae59c640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7caed9d640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 0x7f7cb01a3f00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fd0 tx=0x7f7ca800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7c9f7fe640 1 -- 172.21.15.38:0/2090602828 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ce20 con 0x7f7cb0106630 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb01a4610 con 0x7f7cb0106630 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7c9f7fe640 1 -- 172.21.15.38:0/2090602828 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca8005020 con 0x7f7cb0106630 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7c9f7fe640 1 -- 172.21.15.38:0/2090602828 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8010720 con 0x7f7cb0106630 2024-04-22T00:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.220+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb01a4ae0 con 0x7f7cb0106630 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.222+0000 7f7cb77ec740 1 Processor -- start 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.222+0000 7f7c9f7fe640 1 -- 172.21.15.38:0/2090602828 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7ca8005540 con 0x7f7cb0106630 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 -- start start 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece710820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562ece669560 con 0x562ece704f30 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562ece611730 con 0x562ece70e430 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ece665520 con 0x562ece705850 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55004/0 (socket says 172.21.15.38:55004) 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 -- 172.21.15.38:0/3108967302 learned_addr learned my addr 172.21.15.38:0/3108967302 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cae59c640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece710820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 -- 172.21.15.38:0/3108967302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 msgr2=0x562ece70def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 -- 172.21.15.38:0/3108967302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 msgr2=0x562ece710820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece710820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 -- 172.21.15.38:0/3108967302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ece4477d0 con 0x562ece704f30 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cae59c640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7c9f7fe640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 0x7f7c980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cae59c640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 0x7f7c980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.223+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ca402cfd0 tx=0x7f7ca4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972ed4b2aa637b5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7c9f7fe640 1 -- 172.21.15.38:0/2090602828 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7ca80cfef0 con 0x7f7cb0106630 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7c9cff9640 1 -- 172.21.15.38:0/3108967302 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4049070 con 0x562ece704f30 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7c9cff9640 1 -- 172.21.15.38:0/3108967302 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca402cb80 con 0x562ece704f30 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7c9cff9640 1 -- 172.21.15.38:0/3108967302 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4004f50 con 0x562ece704f30 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cae59c640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 0x7f7c980a51c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x562ece704a00 tx=0x7f7ca000b410 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3108967302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 msgr2=0x562ece705310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ca402cfd0 tx=0x7f7ca4005b50 comp rx=0 tx=0).stop 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3108967302 shutdown_connections 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece710820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3108967302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece705310 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.224+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3108967302 >> 172.21.15.38:0/3108967302 conn(0x562ece6ff660 msgr2=0x562ece6ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3108967302 shutdown_connections 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3108967302 wait complete. 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- start start 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 Processor -- start 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece70b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece70e430 0x562ece70bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562ece7111c0 con 0x562ece705850 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562ece624e00 con 0x562ece704f30 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ece711340 con 0x562ece70e430 2024-04-22T00:06:14.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cb77ec740 10 client.0 _ll_register_callbacks cb 0x562ece65feb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cadd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55006/0 (socket says 172.21.15.38:55006) 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cadd9b640 1 -- 172.21.15.38:0/736712863 learned_addr learned my addr 172.21.15.38:0/736712863 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.225+0000 7f7cae59c640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece70b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7caed9d640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece70e430 0x562ece70bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 -- 172.21.15.38:0/736712863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece70e430 msgr2=0x562ece70bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece70e430 0x562ece70bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 -- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 msgr2=0x562ece70b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece70b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 -- 172.21.15.38:0/736712863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ece37d3f0 con 0x562ece705850 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cae59c640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece70b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4036040 tx=0x7f7ca4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2217bb14002d36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4049070 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7ca4004240 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca402cb80 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7ca4056050 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cb77ec740 10 client.17145 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562ece33d0f0 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.226+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca402cb80 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7ca40039c0 con 0x562ece705850 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 10 client.17145 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 10 client.17145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 20 client.17145 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece71e670 0x562ece720a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 4 client.17145 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:14.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ece676aa0 con 0x562ece71e670 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cae59c640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece71e670 0x562ece720a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7c95ffb640 10 client.17145 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.227+0000 7f7cae59c640 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece71e670 0x562ece720a60 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d9facdca5af105 server_cookie=7dd5faafdf245032 in_seq=0 out_seq=0 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7c95ffb640 1 -- 172.21.15.38:0/736712863 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562ece676aa0 con 0x562ece71e670 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7c95ffb640 10 client.17145 handle_command_reply: tid=0 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 client.17145 shutdown 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 msgr2=0x562ece70ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4036040 tx=0x7f7ca4005fd0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 shutdown_connections 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece70e430 0x562ece70bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece71e670 0x562ece720a60 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece705850 0x562ece70ba10 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/736712863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece70b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7c95ffb640 0 client.17145 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 >> 172.21.15.38:0/736712863 conn(0x562ece6ff660 msgr2=0x562ece7013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 shutdown_connections 2024-04-22T00:06:14.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 1 -- 172.21.15.38:0/736712863 wait complete. 2024-04-22T00:06:14.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.228+0000 7f7cb77ec740 20 client.17145 trim_cache size 0 max 16384 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 Processor -- start 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 -- start start 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece723a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01ae900 con 0x562ece704f30 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01b0440 con 0x562ece70e430 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01aefd0 con 0x562ece705850 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55016/0 (socket says 172.21.15.38:55016) 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 -- 172.21.15.38:0/3653148280 learned_addr learned my addr 172.21.15.38:0/3653148280 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7caed9d640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece723a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 -- 172.21.15.38:0/3653148280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 msgr2=0x562ece70bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 -- 172.21.15.38:0/3653148280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 msgr2=0x562ece723a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece723a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cae59c640 1 -- 172.21.15.38:0/3653148280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ece6facf0 con 0x562ece704f30 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7caed9d640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece723a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.246+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cae59c640 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x562ece711400 tx=0x7f7ca0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af0cd08a2874d4d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7c97fff640 1 -- 172.21.15.38:0/3653148280 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0010d90 con 0x562ece704f30 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7c97fff640 1 -- 172.21.15.38:0/3653148280 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca002f070 con 0x562ece704f30 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7c97fff640 1 -- 172.21.15.38:0/3653148280 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca001c8f0 con 0x562ece704f30 2024-04-22T00:06:14.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3653148280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 msgr2=0x562ece70dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x562ece711400 tx=0x7f7ca0016000 comp rx=0 tx=0).stop 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3653148280 shutdown_connections 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece70e430 0x562ece723a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece70bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/3653148280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece704f30 0x562ece70dc00 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3653148280 >> 172.21.15.38:0/3653148280 conn(0x562ece6ff660 msgr2=0x562ece70b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3653148280 shutdown_connections 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.247+0000 7f7cb77ec740 1 -- 172.21.15.38:0/3653148280 wait complete. 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 -- start start 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 Processor -- start 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece723c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece720fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01afd70 con 0x562ece70e430 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01af6a0 con 0x562ece704f30 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb01ac8e0 con 0x562ece705850 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cb77ec740 10 client.0 _ll_register_callbacks cb 0x7f7ca4047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55032/0 (socket says 172.21.15.38:55032) 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 -- 172.21.15.38:0/1292784146 learned_addr learned my addr 172.21.15.38:0/1292784146 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cadd9b640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece720fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cae59c640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece723c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 -- 172.21.15.38:0/1292784146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 msgr2=0x562ece720fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece720fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 -- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 msgr2=0x562ece723c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece723c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7caed9d640 1 -- 172.21.15.38:0/1292784146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ece721b00 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.248+0000 7f7cae59c640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece723c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7caed9d640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8098a40 tx=0x7f7ca809c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19853d208f0182f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca80b4070 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ca002f070 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7ca80a00e0 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7ca80986e0 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 10 client.17151 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562ece784610 con 0x562ece70e430 2024-04-22T00:06:14.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca002f070 con 0x562ece70e430 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7ca809c360 con 0x562ece70e430 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 10 client.17151 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 20 client.17151 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece784d30 0x562ece787150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 4 client.17151 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.249+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562ece670df0 con 0x562ece784d30 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cae59c640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece784d30 0x562ece787150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cae59c640 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece784d30 0x562ece787150 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8c03fe46846ff1 server_cookie=4613f5a37878fa3e in_seq=0 out_seq=0 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7c95ffb640 10 client.17151 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7c95ffb640 1 -- 172.21.15.38:0/1292784146 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x562ece670df0 con 0x562ece784d30 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7c95ffb640 10 client.17151 handle_command_reply: tid=0 2024-04-22T00:06:14.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 client.17151 shutdown 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 msgr2=0x562ece721510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8098a40 tx=0x7f7ca809c070 comp rx=0 tx=0).stop 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 shutdown_connections 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562ece784d30 0x562ece787150 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562ece70e430 0x562ece721510 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7c95ffb640 0 client.17151 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ece705850 0x562ece720fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 --2- 172.21.15.38:0/1292784146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ece704f30 0x562ece723c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 >> 172.21.15.38:0/1292784146 conn(0x562ece6ff660 msgr2=0x562ece709970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 shutdown_connections 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 1 -- 172.21.15.38:0/1292784146 wait complete. 2024-04-22T00:06:14.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.250+0000 7f7cb77ec740 20 client.17151 trim_cache size 0 max 16384 2024-04-22T00:06:14.254 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:14.254 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (227 inodes in the stack)", 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "4900014b-af7f-4a3a-8bd7-3a3ed008f558": { 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "4900014b-af7f-4a3a-8bd7-3a3ed008f558", 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:06:14.255 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:14.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 msgr2=0x7f7c980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 0x7f7c980a51c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x562ece704a00 tx=0x7f7ca000b410 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 msgr2=0x7f7cb01a3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 0x7f7cb01a3f00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fd0 tx=0x7f7ca800ef10 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 shutdown_connections 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c980a2d00 0x7f7c980a51c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0106630 0x7f7cb01a3f00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb0100fe0 0x7f7cb01993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 --2- 172.21.15.38:0/2090602828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cb0100620 0x7f7cb0198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 >> 172.21.15.38:0/2090602828 conn(0x7f7cb00776a0 msgr2=0x7f7cb01031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 shutdown_connections 2024-04-22T00:06:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:14.251+0000 7f7cb4b25640 1 -- 172.21.15.38:0/2090602828 wait complete. 2024-04-22T00:06:14.258 INFO:tasks.cephfs.filesystem:scrub status for tag:4900014b-af7f-4a3a-8bd7-3a3ed008f558 - {'path': '/', 'tag': '4900014b-af7f-4a3a-8bd7-3a3ed008f558', 'options': 'recursive,force'} 2024-04-22T00:06:14.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:14 smithi038 ceph-mon[27972]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:14.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:14 smithi203 ceph-mon[31431]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:14.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:14 smithi111 ceph-mon[31543]: pgmap v127: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:15.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:15 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:06:15.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:15 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:15.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:15 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:15.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:15 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:06:15.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:15 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:15.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:15 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:15.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:15 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:06:15.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:15 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:15.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:15 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:16.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:16 smithi038 ceph-mon[27972]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:06:16.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:16 smithi203 ceph-mon[31431]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:06:16.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:16 smithi111 ceph-mon[31543]: pgmap v128: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 2.48k op/s 2024-04-22T00:06:18.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:18 smithi038 ceph-mon[27972]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-04-22T00:06:18.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:18 smithi203 ceph-mon[31431]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-04-22T00:06:18.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:18 smithi111 ceph-mon[31543]: pgmap v129: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 13 MiB/s wr, 3.31k op/s 2024-04-22T00:06:20.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:20 smithi038 ceph-mon[27972]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-04-22T00:06:20.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:20 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:06:20.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:20 smithi203 ceph-mon[31431]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-04-22T00:06:20.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:20 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:06:20.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:20 smithi111 ceph-mon[31543]: pgmap v130: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-04-22T00:06:20.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:20 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:06:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 -- 172.21.15.38:0/2922327951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 msgr2=0x7f07b810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 --2- 172.21.15.38:0/2922327951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 0x7f07b810d4f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009cd0 tx=0x7f07b002dff0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 -- 172.21.15.38:0/2922327951 shutdown_connections 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 --2- 172.21.15.38:0/2922327951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 0x7f07b810d4f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 --2- 172.21.15.38:0/2922327951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8100670 0x7f07b8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 --2- 172.21.15.38:0/2922327951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8106630 0x7f07b8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.544+0000 7f07bf09d640 1 -- 172.21.15.38:0/2922327951 >> 172.21.15.38:0/2922327951 conn(0x7f07b80777c0 msgr2=0x7f07b8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.547+0000 7f07bf09d640 1 -- 172.21.15.38:0/2922327951 shutdown_connections 2024-04-22T00:06:21.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.548+0000 7f07bf09d640 1 -- 172.21.15.38:0/2922327951 wait complete. 2024-04-22T00:06:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.549+0000 7f07bf09d640 1 Processor -- start 2024-04-22T00:06:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 -- start start 2024-04-22T00:06:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 0x7f07b81a4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 0x7f07b81a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 0x7f07b810beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810f3a0 con 0x7f07b8101030 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810f220 con 0x7f07b8100670 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bf09d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810f520 con 0x7f07b8106630 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bce39640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 0x7f07b81a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 0x7f07b810beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 0x7f07b810beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:55450/0 (socket says 172.21.15.38:55450) 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.551+0000 7f07bd63a640 1 -- 172.21.15.38:0/3668096005 learned_addr learned my addr 172.21.15.38:0/3668096005 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 -- 172.21.15.38:0/3668096005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 msgr2=0x7f07b810beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 0x7f07b810beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 -- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 msgr2=0x7f07b81a4860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:06:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 0x7f07b81a4860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 -- 172.21.15.38:0/3668096005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b810c680 con 0x7f07b8100670 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bce39640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 0x7f07b81a4320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c970 tx=0x7f07ac00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07b5ffb640 1 -- 172.21.15.38:0/3668096005 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac007bb0 con 0x7f07b8100670 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.552+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b810c970 con 0x7f07b8100670 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.553+0000 7f07b5ffb640 1 -- 172.21.15.38:0/3668096005 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ac007d50 con 0x7f07b8100670 2024-04-22T00:06:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.553+0000 7f07b5ffb640 1 -- 172.21.15.38:0/3668096005 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac00f450 con 0x7f07b8100670 2024-04-22T00:06:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.554+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b806b340 con 0x7f07b8100670 2024-04-22T00:06:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.557+0000 7f07c1d64740 1 Processor -- start 2024-04-22T00:06:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 -- start start 2024-04-22T00:06:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d09f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d15600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564af2c6db70 con 0x564af2d13210 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564af2c16470 con 0x564af2d09b20 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07c1d64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564af2c69b30 con 0x564af2d0a440 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.558+0000 7f07b5ffb640 1 -- 172.21.15.38:0/3668096005 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f07ac004f20 con 0x7f07b8100670 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:55454/0 (socket says 172.21.15.38:55454) 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 -- 172.21.15.38:0/3154015109 learned_addr learned my addr 172.21.15.38:0/3154015109 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 -- 172.21.15.38:0/3154015109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 msgr2=0x564af2d09f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d09f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 -- 172.21.15.38:0/3154015109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 msgr2=0x564af2d15600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:06:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d15600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 -- 172.21.15.38:0/3154015109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af2982210 con 0x564af2d0a440 2024-04-22T00:06:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.559+0000 7f07bce39640 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0cbf90 tx=0x7f07ac0d76d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9703e54ca57915b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.560+0000 7f07b5ffb640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 0x7f07a00a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.560+0000 7f07b5ffb640 1 -- 172.21.15.38:0/3668096005 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f07ac0ceca0 con 0x7f07b8100670 2024-04-22T00:06:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.563+0000 7f079effd640 1 -- 172.21.15.38:0/3154015109 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0de070 con 0x564af2d0a440 2024-04-22T00:06:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.563+0000 7f079effd640 1 -- 172.21.15.38:0/3154015109 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ac0dab60 con 0x564af2d0a440 2024-04-22T00:06:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.563+0000 7f07bd63a640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 0x7f07a00a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 -- 172.21.15.38:0/3154015109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 msgr2=0x564af2d12cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0cbf90 tx=0x7f07ac0d76d0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07bd63a640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 0x7f07a00a53b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07b000d4e0 tx=0x7f07b002e510 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 -- 172.21.15.38:0/3154015109 shutdown_connections 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d15600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d0a440 0x564af2d12cd0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0cbf90 tx=0x7f07ac0d76d0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3154015109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d09f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 -- 172.21.15.38:0/3154015109 >> 172.21.15.38:0/3154015109 conn(0x564af2d04440 msgr2=0x564af2d04820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 -- 172.21.15.38:0/3154015109 shutdown_connections 2024-04-22T00:06:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.564+0000 7f07c1d64740 1 -- 172.21.15.38:0/3154015109 wait complete. 2024-04-22T00:06:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.565+0000 7f07c1d64740 1 -- start start 2024-04-22T00:06:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.565+0000 7f07c1d64740 1 Processor -- start 2024-04-22T00:06:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d10410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d13210 0x564af2d10950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564af2d26e50 con 0x564af2d10ee0 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x564af2c29a10 con 0x564af2d13210 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564af2d26fd0 con 0x564af2d09b20 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07c1d64740 10 client.0 _ll_register_callbacks cb 0x564af2c64b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55056/0 (socket says 172.21.15.38:55056) 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07bd63a640 1 -- 172.21.15.38:0/3981211319 learned_addr learned my addr 172.21.15.38:0/3981211319 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.566+0000 7f07bce39640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d10410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 -- 172.21.15.38:0/3981211319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 msgr2=0x564af2d10410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d10410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 -- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d13210 msgr2=0x564af2d10950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d13210 0x564af2d10950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 -- 172.21.15.38:0/3981211319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af2a4e620 con 0x564af2d10ee0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f07bd63a640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009bf0 tx=0x7f07b00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684679e934c32b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.567+0000 7f0797fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.568+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0052060 con 0x564af2d10ee0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.568+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564af2c7b0b0 con 0x564af2d10ee0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.568+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f07b00047c0 con 0x564af2d10ee0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.568+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f07b0059050 con 0x564af2d10ee0 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.569+0000 7f07c1d64740 10 client.17157 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.569+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564af2d338b0 con 0x564af2d10ee0 2024-04-22T00:06:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.571+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564af2c7b0b0 con 0x564af2d10ee0 2024-04-22T00:06:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f07b000b040 con 0x564af2d10ee0 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 10 client.17157 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 10 client.17157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 20 client.17157 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d351e0 0x564af2d37660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 4 client.17157 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.572+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564af2c75400 con 0x564af2d351e0 2024-04-22T00:06:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.573+0000 7f07bce39640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d351e0 0x564af2d37660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.573+0000 7f07bce39640 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d351e0 0x564af2d37660 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0696b377082b1c server_cookie=e3a08b1c739b875d in_seq=0 out_seq=0 2024-04-22T00:06:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.574+0000 7f0797fff640 10 client.17157 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.574+0000 7f0797fff640 1 -- 172.21.15.38:0/3981211319 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564af2c75400 con 0x564af2d351e0 2024-04-22T00:06:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.574+0000 7f0797fff640 10 client.17157 handle_command_reply: tid=0 2024-04-22T00:06:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.574+0000 7f07c1d64740 1 client.17157 shutdown 2024-04-22T00:06:21.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.575+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 msgr2=0x564af2d30340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.575+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009bf0 tx=0x7f07b00095c0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 shutdown_connections 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d351e0 0x564af2d37660 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d30340 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d13210 0x564af2d10950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3981211319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d10410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 >> 172.21.15.38:0/3981211319 conn(0x564af2d04440 msgr2=0x564af2d06190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f0797fff640 0 client.17157 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:21.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 shutdown_connections 2024-04-22T00:06:21.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.576+0000 7f07c1d64740 1 -- 172.21.15.38:0/3981211319 wait complete. 2024-04-22T00:06:21.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.577+0000 7f07c1d64740 20 client.17157 trim_cache size 0 max 16384 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.611+0000 7f07c1d64740 1 Processor -- start 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.611+0000 7f07c1d64740 1 -- start start 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d3cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d3d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8114170 con 0x564af2d10ee0 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8115cb0 con 0x564af2d09b20 2024-04-22T00:06:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07c1d64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8114840 con 0x564af2d13210 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07bd63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:55476/0 (socket says 172.21.15.38:55476) 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07bd63a640 1 -- 172.21.15.38:0/115174528 learned_addr learned my addr 172.21.15.38:0/115174528 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07bce39640 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d3cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.612+0000 7f07b7fff640 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d3d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 -- 172.21.15.38:0/115174528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 msgr2=0x564af2d3cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d3cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 -- 172.21.15.38:0/115174528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 msgr2=0x564af2d3d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d3d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 -- 172.21.15.38:0/115174528 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af2d0a140 con 0x564af2d13210 2024-04-22T00:06:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.613+0000 7f07bd63a640 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07b00349f0 tx=0x7f07b0009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3bcb2940427c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.615+0000 7f079e7fc640 1 -- 172.21.15.38:0/115174528 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0052040 con 0x564af2d13210 2024-04-22T00:06:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.615+0000 7f079e7fc640 1 -- 172.21.15.38:0/115174528 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b003c040 con 0x564af2d13210 2024-04-22T00:06:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.615+0000 7f079e7fc640 1 -- 172.21.15.38:0/115174528 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0004780 con 0x564af2d13210 2024-04-22T00:06:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.616+0000 7f07c1d64740 1 -- 172.21.15.38:0/115174528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 msgr2=0x564af2d2fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.616+0000 7f07c1d64740 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07b00349f0 tx=0x7f07b0009a00 comp rx=0 tx=0).stop 2024-04-22T00:06:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.617+0000 7f07c1d64740 1 -- 172.21.15.38:0/115174528 shutdown_connections 2024-04-22T00:06:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.617+0000 7f07c1d64740 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d13210 0x564af2d2fd90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.617+0000 7f07c1d64740 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d10ee0 0x564af2d3d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.617+0000 7f07c1d64740 1 --2- 172.21.15.38:0/115174528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d09b20 0x564af2d3cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.617+0000 7f07c1d64740 1 -- 172.21.15.38:0/115174528 >> 172.21.15.38:0/115174528 conn(0x564af2d04440 msgr2=0x564af2d0ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.619+0000 7f07c1d64740 1 -- 172.21.15.38:0/115174528 shutdown_connections 2024-04-22T00:06:21.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.620+0000 7f07c1d64740 1 -- 172.21.15.38:0/115174528 wait complete. 2024-04-22T00:06:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.620+0000 7f07c1d64740 1 -- start start 2024-04-22T00:06:21.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.620+0000 7f07c1d64740 1 Processor -- start 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.625+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d10ee0 0x564af2d27ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d937a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07b81155e0 con 0x564af2d13210 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8114f10 con 0x564af2d10ee0 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f07b8073120 con 0x564af2d09b20 2024-04-22T00:06:21.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.626+0000 7f07c1d64740 10 client.0 _ll_register_callbacks cb 0x7f07b81a4f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.627+0000 7f07bce39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.627+0000 7f07bce39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:55490/0 (socket says 172.21.15.38:55490) 2024-04-22T00:06:21.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.627+0000 7f07bce39640 1 -- 172.21.15.38:0/3923216901 learned_addr learned my addr 172.21.15.38:0/3923216901 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:21.622 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.628+0000 7f07bce39640 1 -- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d10ee0 msgr2=0x564af2d27ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:06:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.628+0000 7f07bce39640 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d10ee0 0x564af2d27ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.628+0000 7f07bce39640 1 -- 172.21.15.38:0/3923216901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 msgr2=0x564af2d937a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:06:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.628+0000 7f07bce39640 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d937a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.628+0000 7f07bce39640 1 -- 172.21.15.38:0/3923216901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af2d93e60 con 0x564af2d09b20 2024-04-22T00:06:21.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.629+0000 7f07bce39640 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f07ac000c00 tx=0x7f07ac00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1dd2cb4f7d6dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:21.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.631+0000 7f0797fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:21.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.631+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0de040 con 0x564af2d09b20 2024-04-22T00:06:21.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.631+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f07ac0046c0 con 0x564af2d09b20 2024-04-22T00:06:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.633+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07b003c040 con 0x564af2d09b20 2024-04-22T00:06:21.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.634+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f07ac0970c0 con 0x564af2d09b20 2024-04-22T00:06:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.635+0000 7f07c1d64740 10 client.26431 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:21.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.635+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564af2d9b310 con 0x564af2d09b20 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07ac0d29a0 con 0x564af2d09b20 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f07ac0a3ce0 con 0x564af2d09b20 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 10 client.26431 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 10 client.26431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 20 client.26431 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d9b600 0x564af2d9d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 4 client.26431 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:06:21.630 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f07b003c040 con 0x564af2d9b600 2024-04-22T00:06:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.636+0000 7f07b7fff640 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d9b600 0x564af2d9d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:21.633 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.639+0000 7f07b7fff640 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d9b600 0x564af2d9d9f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6363c566c8dc3152 server_cookie=bdc3a8f57a07474e in_seq=0 out_seq=0 2024-04-22T00:06:21.633 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.639+0000 7f0797fff640 10 client.26431 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:21.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:21 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:06:21.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:21 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:06:21.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:21 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:06:21.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:21 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:06:21.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:21 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:06:21.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:21 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:06:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:21 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:06:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:21 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:06:21.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:21 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.961+0000 7f0797fff640 1 -- 172.21.15.38:0/3923216901 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f07b003c040 con 0x564af2d9b600 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.961+0000 7f0797fff640 10 client.26431 handle_command_reply: tid=0 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.961+0000 7f07c1d64740 1 client.26431 shutdown 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 msgr2=0x564af2d279a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f07ac000c00 tx=0x7f07ac00a340 comp rx=0 tx=0).stop 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 shutdown_connections 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x564af2d9b600 0x564af2d9d9f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564af2d13210 0x564af2d937a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f0797fff640 0 client.26431 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564af2d10ee0 0x564af2d27ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 --2- 172.21.15.38:0/3923216901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564af2d09b20 0x564af2d279a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.962+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 >> 172.21.15.38:0/3923216901 conn(0x564af2d04440 msgr2=0x564af2d02c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.963+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 shutdown_connections 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.963+0000 7f07c1d64740 1 -- 172.21.15.38:0/3923216901 wait complete. 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.963+0000 7f07c1d64740 20 client.26431 trim_cache size 0 max 16384 2024-04-22T00:06:21.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:06:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:06:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:06:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:06:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 msgr2=0x7f07a00a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 0x7f07a00a53b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f07b000d4e0 tx=0x7f07b002e510 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 msgr2=0x7f07b81a4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 0x7f07b81a4320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c970 tx=0x7f07ac00ce30 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 shutdown_connections 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f07a00a2ef0 0x7f07a00a53b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b8106630 0x7f07b810beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07b8101030 0x7f07b81a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 --2- 172.21.15.38:0/3668096005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b8100670 0x7f07b81a4320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 >> 172.21.15.38:0/3668096005 conn(0x7f07b80777c0 msgr2=0x7f07b8111240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 shutdown_connections 2024-04-22T00:06:21.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:21.964+0000 7f07bf09d640 1 -- 172.21.15.38:0/3668096005 wait complete. 2024-04-22T00:06:22.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:22 smithi038 ceph-mon[27972]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:06:22.780 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:22 smithi203 ceph-mon[31431]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:06:22.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:22 smithi111 ceph-mon[31543]: pgmap v131: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:06:24.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:24 smithi038 ceph-mon[27972]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-04-22T00:06:24.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:24 smithi203 ceph-mon[31431]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-04-22T00:06:24.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:24 smithi111 ceph-mon[31543]: pgmap v132: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-04-22T00:06:26.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:26 smithi038 ceph-mon[27972]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:06:26.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:26 smithi203 ceph-mon[31431]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:06:26.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:26 smithi111 ceph-mon[31543]: pgmap v133: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:06:28.328 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:06:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:06:27.944+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v348>,<0x100000001fb/client.0 v326>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v329>,<0x10000000001/sv_1 v350>,<0x10000000000/qa v331>,<0x1/volumes v317>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:06:28.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:28 smithi038 ceph-mon[27972]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 3.32k op/s 2024-04-22T00:06:28.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:28 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:28.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:28 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:28.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:28 smithi203 ceph-mon[31431]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 3.32k op/s 2024-04-22T00:06:28.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:28 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:28.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:28 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:28.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:28 smithi111 ceph-mon[31543]: pgmap v134: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 14 MiB/s wr, 3.32k op/s 2024-04-22T00:06:28.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:28 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:28.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:28 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:29.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:29 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:29.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:29 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:29.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:29 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:30.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:30 smithi203 ceph-mon[31431]: pgmap v135: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 11 MiB/s wr, 2.59k op/s 2024-04-22T00:06:30.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:30 smithi111 ceph-mon[31543]: pgmap v135: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 11 MiB/s wr, 2.59k op/s 2024-04-22T00:06:30.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:30 smithi038 ceph-mon[27972]: pgmap v135: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 11 MiB/s wr, 2.59k op/s 2024-04-22T00:06:32.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:32 smithi203 ceph-mon[31431]: pgmap v136: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.9 MiB/s wr, 2.43k op/s 2024-04-22T00:06:32.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:32 smithi111 ceph-mon[31543]: pgmap v136: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.9 MiB/s wr, 2.43k op/s 2024-04-22T00:06:32.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:32 smithi038 ceph-mon[27972]: pgmap v136: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.9 MiB/s wr, 2.43k op/s 2024-04-22T00:06:34.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:34 smithi203 ceph-mon[31431]: pgmap v137: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-04-22T00:06:34.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:34 smithi111 ceph-mon[31543]: pgmap v137: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-04-22T00:06:34.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:34 smithi038 ceph-mon[27972]: pgmap v137: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 13 MiB/s wr, 3.30k op/s 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3302917790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8101030 msgr2=0x7fc7f810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3302917790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8101030 0x7fc7f810d4f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009cd0 tx=0x7fc7f002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3302917790 shutdown_connections 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3302917790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8101030 0x7fc7f810d4f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3302917790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8100670 0x7fc7f8100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3302917790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8106630 0x7fc7f8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3302917790 >> 172.21.15.38:0/3302917790 conn(0x7fc7f80777c0 msgr2=0x7fc7f8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.057 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3302917790 shutdown_connections 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.060+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3302917790 wait complete. 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 Processor -- start 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 -- start start 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8100670 0x7fc7f80fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 0x7fc7f80fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8110a40 con 0x7fc7f8106630 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81108c0 con 0x7fc7f8101030 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7fd1be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8110bc0 con 0x7fc7f8100670 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.061+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57372/0 (socket says 172.21.15.38:57372) 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 -- 172.21.15.38:0/3500057175 learned_addr learned my addr 172.21.15.38:0/3500057175 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8100670 0x7fc7f80fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7effff640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 0x7fc7f80fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 -- 172.21.15.38:0/3500057175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8100670 msgr2=0x7fc7f80fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8100670 0x7fc7f80fd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 -- 172.21.15.38:0/3500057175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 msgr2=0x7fc7f80fdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 0x7fc7f80fdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 -- 172.21.15.38:0/3500057175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f81a64d0 con 0x7fc7f8106630 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7effff640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 0x7fc7f80fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:35.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f759e640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0004af0 tx=0x7fc7f0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f4d99640 1 -- 172.21.15.38:0/3500057175 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0041020 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f4d99640 1 -- 172.21.15.38:0/3500057175 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f0034cd0 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f81a6760 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7f4d99640 1 -- 172.21.15.38:0/3500057175 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0039040 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.062+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f81a6c50 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.064+0000 7fc7f4d99640 1 -- 172.21.15.38:0/3500057175 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc7f0044020 con 0x7fc7f8106630 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 Processor -- start 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 -- start start 2024-04-22T00:06:35.066 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 0x5639fa1eaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5639fa14eb70 con 0x5639fa1eb440 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5639fa0f7470 con 0x5639fa1eab20 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5639fa14ab30 con 0x5639fa1f4210 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57386/0 (socket says 172.21.15.38:57386) 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7f6d9d640 1 -- 172.21.15.38:0/3480289569 learned_addr learned my addr 172.21.15.38:0/3480289569 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7effff640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.065+0000 7fc7f4d99640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 0x7fc7e40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7effff640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 0x5639fa1eaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f759e640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 0x7fc7e40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.067 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 -- 172.21.15.38:0/3480289569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 msgr2=0x5639fa1f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 -- 172.21.15.38:0/3480289569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 msgr2=0x5639fa1eaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 0x5639fa1eaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 -- 172.21.15.38:0/3480289569 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639f9e63210 con 0x5639fa1eb440 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7effff640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 0x5639fa1eaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f4d99640 1 -- 172.21.15.38:0/3500057175 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc7f00f1410 con 0x7fc7f8106630 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800b710 tx=0x7fc7e8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f380fceb64b96a7c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7ed7fa640 1 -- 172.21.15.38:0/3480289569 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8003bd0 con 0x5639fa1eb440 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7f759e640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 0x7fc7e40a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00ee9e0 tx=0x7fc7f00a8230 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7ed7fa640 1 -- 172.21.15.38:0/3480289569 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e8015070 con 0x5639fa1eb440 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.066+0000 7fc7ed7fa640 1 -- 172.21.15.38:0/3480289569 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8004e40 con 0x5639fa1eb440 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3480289569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 msgr2=0x5639fa1f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800b710 tx=0x7fc7e8004610 comp rx=0 tx=0).stop 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3480289569 shutdown_connections 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1f3cd0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3480289569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eab20 0x5639fa1eaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3480289569 >> 172.21.15.38:0/3480289569 conn(0x5639fa1e5440 msgr2=0x5639fa1e5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3480289569 shutdown_connections 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3480289569 wait complete. 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 -- start start 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.067+0000 7fc7ffdab740 1 Processor -- start 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5639fa1f6e40 con 0x5639fa1eab20 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5639fa10aa10 con 0x5639fa1eb440 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5639fa1f6fc0 con 0x5639fa1f4210 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7ffdab740 10 client.0 _ll_register_callbacks cb 0x5639fa145b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57400/0 (socket says 172.21.15.38:57400) 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 -- 172.21.15.38:0/3602000624 learned_addr learned my addr 172.21.15.38:0/3602000624 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 -- 172.21.15.38:0/3602000624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 msgr2=0x5639fa1f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 -- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 msgr2=0x5639fa1f17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1f17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 -- 172.21.15.38:0/3602000624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639f9f2f620 con 0x5639fa1eab20 2024-04-22T00:06:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1f17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7effff640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0013b80 tx=0x7fc7e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b00ea7d0643c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.068+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e000ee80 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc7e000f630 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7e8015070 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc7e000f830 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 10 client.17172 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5639fa203160 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e8015070 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc7e001e020 con 0x5639fa1eab20 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 10 client.17172 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:35.071 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 20 client.17172 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa204620 0x5639fa206a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 4 client.17172 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.069+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639fa15c0b0 con 0x5639fa204620 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.070+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa204620 0x5639fa206a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.070+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa204620 0x5639fa206a70 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8d7d3753fb5e7f server_cookie=fa1f6a2e606e4fa4 in_seq=0 out_seq=0 2024-04-22T00:06:35.072 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.070+0000 7fc7de7fc640 10 client.17172 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.073+0000 7fc7de7fc640 1 -- 172.21.15.38:0/3602000624 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639fa15c0b0 con 0x5639fa204620 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.073+0000 7fc7de7fc640 10 client.17172 handle_command_reply: tid=0 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.073+0000 7fc7ffdab740 1 client.17172 shutdown 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.073+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 msgr2=0x5639fa1f12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.073+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0013b80 tx=0x7fc7e0004730 comp rx=0 tx=0).stop 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 shutdown_connections 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1f4210 0x5639fa1f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1f17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa204620 0x5639fa206a70 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/3602000624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eab20 0x5639fa1f12a0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 >> 172.21.15.38:0/3602000624 conn(0x5639fa1e5440 msgr2=0x5639fa1e7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7de7fc640 0 client.17172 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 shutdown_connections 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 1 -- 172.21.15.38:0/3602000624 wait complete. 2024-04-22T00:06:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.074+0000 7fc7ffdab740 20 client.17172 trim_cache size 0 max 16384 2024-04-22T00:06:35.099 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.103+0000 7fc7ffdab740 1 Processor -- start 2024-04-22T00:06:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.103+0000 7fc7ffdab740 1 -- start start 2024-04-22T00:06:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.103+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1f4210 0x5639fa206630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.100 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b0b90 con 0x5639fa1eb440 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b26d0 con 0x5639fa1f4210 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b1260 con 0x5639fa1eab20 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:58194/0 (socket says 172.21.15.38:58194) 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 -- 172.21.15.38:0/1573186803 learned_addr learned my addr 172.21.15.38:0/1573186803 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:35.101 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1f4210 0x5639fa206630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 -- 172.21.15.38:0/1573186803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1f4210 msgr2=0x5639fa206630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1f4210 0x5639fa206630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 -- 172.21.15.38:0/1573186803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 msgr2=0x5639fa1ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.102 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.104+0000 7fc7effff640 1 -- 172.21.15.38:0/1573186803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639fa1eb270 con 0x5639fa1eab20 2024-04-22T00:06:35.103 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.109+0000 7fc7effff640 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00138e0 tx=0x7fc7e00058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2b27e17de212d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.109+0000 7fc7ecff9640 1 -- 172.21.15.38:0/1573186803 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0035070 con 0x5639fa1eab20 2024-04-22T00:06:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.112+0000 7fc7ecff9640 1 -- 172.21.15.38:0/1573186803 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e000ee80 con 0x5639fa1eab20 2024-04-22T00:06:35.107 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.113+0000 7fc7ecff9640 1 -- 172.21.15.38:0/1573186803 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0020840 con 0x5639fa1eab20 2024-04-22T00:06:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.114+0000 7fc7ffdab740 1 -- 172.21.15.38:0/1573186803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 msgr2=0x5639fa1f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.108 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.114+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00138e0 tx=0x7fc7e00058b0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.116+0000 7fc7ffdab740 1 -- 172.21.15.38:0/1573186803 shutdown_connections 2024-04-22T00:06:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.116+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1f4210 0x5639fa206630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.116+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1eb440 0x5639fa1ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.116+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/1573186803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1f39d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.116+0000 7fc7ffdab740 1 -- 172.21.15.38:0/1573186803 >> 172.21.15.38:0/1573186803 conn(0x5639fa1e5440 msgr2=0x5639fa1efb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.117+0000 7fc7ffdab740 1 -- 172.21.15.38:0/1573186803 shutdown_connections 2024-04-22T00:06:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.117+0000 7fc7ffdab740 1 -- 172.21.15.38:0/1573186803 wait complete. 2024-04-22T00:06:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.118+0000 7fc7ffdab740 1 -- start start 2024-04-22T00:06:35.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.118+0000 7fc7ffdab740 1 Processor -- start 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1fc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1fc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1f4210 0x5639fa1ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b2000 con 0x5639fa1f4210 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b1930 con 0x5639fa1eb440 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1f4210 0x5639fa1ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1fc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:52578/0 (socket says 172.21.15.38:52578) 2024-04-22T00:06:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7f6d9d640 1 -- 172.21.15.38:0/2954427804 learned_addr learned my addr 172.21.15.38:0/2954427804 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.119+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81aeb70 con 0x5639fa1eab20 2024-04-22T00:06:35.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7ffdab740 10 client.0 _ll_register_callbacks cb 0x7fc7e0009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7effff640 1 -- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 msgr2=0x5639fa1fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7effff640 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7effff640 1 -- 172.21.15.38:0/2954427804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1f4210 msgr2=0x5639fa1ed5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7effff640 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1f4210 0x5639fa1ed5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.120+0000 7fc7effff640 1 -- 172.21.15.38:0/2954427804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639fa1edcc0 con 0x5639fa1eab20 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.121+0000 7fc7effff640 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1fc210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0013b80 tx=0x7fc7e0020440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad1d413882d78298 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.121+0000 7fc7de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.121+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0035040 con 0x5639fa1eab20 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.121+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc7e000fde0 con 0x5639fa1eab20 2024-04-22T00:06:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.122+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5639fa206e00 con 0x5639fa1eab20 2024-04-22T00:06:35.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.122+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc7e00114e0 con 0x5639fa1eab20 2024-04-22T00:06:35.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.122+0000 7fc7ffdab740 10 client.26449 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:35.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.122+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5639fa27b570 con 0x5639fa1eab20 2024-04-22T00:06:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5639fa206e00 con 0x5639fa1eab20 2024-04-22T00:06:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc7e003b040 con 0x5639fa1eab20 2024-04-22T00:06:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 10 client.26449 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 10 client.26449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 20 client.26449 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa27bb30 0x5639fa27df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 4 client.26449 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639fa156400 con 0x5639fa27bb30 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.123+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa27bb30 0x5639fa27df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.124+0000 7fc7f6d9d640 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa27bb30 0x5639fa27df50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d5ebcce78009ef8 server_cookie=6214f16f119b43f0 in_seq=0 out_seq=0 2024-04-22T00:06:35.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.125+0000 7fc7de7fc640 10 client.26449 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:35.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7de7fc640 1 -- 172.21.15.38:0/2954427804 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639fa156400 con 0x5639fa27bb30 2024-04-22T00:06:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7de7fc640 10 client.26449 handle_command_reply: tid=0 2024-04-22T00:06:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 client.26449 shutdown 2024-04-22T00:06:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 msgr2=0x5639fa1fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1fc210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0013b80 tx=0x7fc7e0020440 comp rx=0 tx=0).stop 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 shutdown_connections 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5639fa27bb30 0x5639fa27df50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5639fa1f4210 0x5639fa1ed5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5639fa1eb440 0x5639fa1fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7de7fc640 0 client.26449 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 --2- 172.21.15.38:0/2954427804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5639fa1eab20 0x5639fa1fc210 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.288+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 >> 172.21.15.38:0/2954427804 conn(0x5639fa1e5440 msgr2=0x5639fa1f1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 shutdown_connections 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7ffdab740 1 -- 172.21.15.38:0/2954427804 wait complete. 2024-04-22T00:06:35.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7ffdab740 20 client.26449 trim_cache size 0 max 16384 2024-04-22T00:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:06:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:06:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 msgr2=0x7fc7e40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 0x7fc7e40a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00ee9e0 tx=0x7fc7f00a8230 comp rx=0 tx=0).stop 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 msgr2=0x7fc7f80fe080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.289+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0004af0 tx=0x7fc7f0032ec0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 shutdown_connections 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc7e40a2da0 0x7fc7e40a5260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7f8106630 0x7fc7f80fe080 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7f8101030 0x7fc7f80fdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 --2- 172.21.15.38:0/3500057175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8100670 0x7fc7f80fd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 >> 172.21.15.38:0/3500057175 conn(0x7fc7f80777c0 msgr2=0x7fc7f8069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 shutdown_connections 2024-04-22T00:06:35.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:35.290+0000 7fc7fd1be640 1 -- 172.21.15.38:0/3500057175 wait complete. 2024-04-22T00:06:36.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:36 smithi203 ceph-mon[31431]: pgmap v138: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:06:36.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:36 smithi111 ceph-mon[31543]: pgmap v138: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:06:36.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:36 smithi038 ceph-mon[27972]: pgmap v138: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:06:38.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:38 smithi203 ceph-mon[31431]: pgmap v139: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 13 MiB/s wr, 3.24k op/s 2024-04-22T00:06:38.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:38 smithi111 ceph-mon[31543]: pgmap v139: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 13 MiB/s wr, 3.24k op/s 2024-04-22T00:06:38.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:38 smithi038 ceph-mon[27972]: pgmap v139: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 13 MiB/s wr, 3.24k op/s 2024-04-22T00:06:40.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:40 smithi203 ceph-mon[31431]: pgmap v140: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:40.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:40 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:40.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:40 smithi111 ceph-mon[31543]: pgmap v140: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:40.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:40 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:40.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:40 smithi038 ceph-mon[27972]: pgmap v140: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:40.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:40 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:41.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:41 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:06:41.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:41 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:06:41.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:41 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:06:41.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:41 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:06:41.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:41 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:06:41.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:41 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:06:42.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:42 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:06:42.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:42 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:06:42.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:42 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:06:42.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:42 smithi203 ceph-mon[31431]: pgmap v141: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:06:42.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:42 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:06:42.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:42 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:06:42.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:42 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:06:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:42 smithi111 ceph-mon[31543]: pgmap v141: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:06:42.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:42 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:06:42.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:42 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:06:42.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:42 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:06:42.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:42 smithi038 ceph-mon[27972]: pgmap v141: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:06:44.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:06:44.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:06:44.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 -- 172.21.15.38:0/3526289003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c1066b0 msgr2=0x7fef5c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 --2- 172.21.15.38:0/3526289003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c1066b0 0x7fef5c106a90 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fef44009930 tx=0x7fef4402bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 -- 172.21.15.38:0/3526289003 shutdown_connections 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 --2- 172.21.15.38:0/3526289003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100fd0 0x7fef5c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 --2- 172.21.15.38:0/3526289003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100610 0x7fef5c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 --2- 172.21.15.38:0/3526289003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c1066b0 0x7fef5c106a90 secure :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fef44009930 tx=0x7fef4402bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:44.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.449+0000 7fef61b93640 1 -- 172.21.15.38:0/3526289003 >> 172.21.15.38:0/3526289003 conn(0x7fef5c077740 msgr2=0x7fef5c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.452+0000 7fef61b93640 1 -- 172.21.15.38:0/3526289003 shutdown_connections 2024-04-22T00:06:44.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.452+0000 7fef61b93640 1 -- 172.21.15.38:0/3526289003 wait complete. 2024-04-22T00:06:44.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 Processor -- start 2024-04-22T00:06:44.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 -- start start 2024-04-22T00:06:44.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 0x7fef5c1162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 0x7fef5c110370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 0x7fef5c110d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10eae0 con 0x7fef5c110900 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10e960 con 0x7fef5c100610 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.453+0000 7fef61b93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10ec60 con 0x7fef5c100fd0 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 0x7fef5c1162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 0x7fef5c1162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55194/0 (socket says 172.21.15.38:55194) 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5b7fe640 1 -- 172.21.15.38:0/974343276 learned_addr learned my addr 172.21.15.38:0/974343276 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 0x7fef5c110370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 -- 172.21.15.38:0/974343276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 msgr2=0x7fef5c1162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 0x7fef5c1162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 -- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 msgr2=0x7fef5c110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5bfff640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 0x7fef5c110d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 0x7fef5c110d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5affd640 1 -- 172.21.15.38:0/974343276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c11da10 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.454+0000 7fef5bfff640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 0x7fef5c110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef5affd640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 0x7fef5c110370 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00e9f0 tx=0x7fef4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef58ff9640 1 -- 172.21.15.38:0/974343276 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c00ccc0 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c11dd00 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef58ff9640 1 -- 172.21.15.38:0/974343276 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef4c005020 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef58ff9640 1 -- 172.21.15.38:0/974343276 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0049e0 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.455+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c11e250 con 0x7fef5c100fd0 2024-04-22T00:06:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.458+0000 7fef58ff9640 1 -- 172.21.15.38:0/974343276 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fef4c005560 con 0x7fef5c100fd0 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef58ff9640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 0x7fef480a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 Processor -- start 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 -- start start 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef58ff9640 1 -- 172.21.15.38:0/974343276 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fef4c0cf6d0 con 0x7fef5c100fd0 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e1e392c560 con 0x55e1e39d1430 2024-04-22T00:06:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1e38d4730 con 0x55e1e39c7f30 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.459+0000 7fef64776740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e1e3928520 con 0x55e1e39c8850 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60726/0 (socket says 172.21.15.38:60726) 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 -- 172.21.15.38:0/924001510 learned_addr learned my addr 172.21.15.38:0/924001510 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 0x7fef480a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5affd640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 -- 172.21.15.38:0/924001510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 msgr2=0x55e1e39d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 -- 172.21.15.38:0/924001510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 msgr2=0x55e1e39c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5bfff640 1 -- 172.21.15.38:0/924001510 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1e370a7d0 con 0x55e1e39d1430 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.460+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.461+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 0x7fef480a5310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44005b50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.461+0000 7fef5bfff640 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fef54009f10 tx=0x7fef5400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3255ce067567f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.461+0000 7fef51ffb640 1 -- 172.21.15.38:0/924001510 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54003b70 con 0x55e1e39d1430 2024-04-22T00:06:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.461+0000 7fef51ffb640 1 -- 172.21.15.38:0/924001510 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef54003d10 con 0x55e1e39d1430 2024-04-22T00:06:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.461+0000 7fef51ffb640 1 -- 172.21.15.38:0/924001510 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54011a70 con 0x55e1e39d1430 2024-04-22T00:06:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.462+0000 7fef64776740 1 -- 172.21.15.38:0/924001510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 msgr2=0x55e1e39d3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.462+0000 7fef64776740 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fef54009f10 tx=0x7fef5400f730 comp rx=0 tx=0).stop 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 -- 172.21.15.38:0/924001510 shutdown_connections 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39d1430 0x55e1e39d3820 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 --2- 172.21.15.38:0/924001510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 -- 172.21.15.38:0/924001510 >> 172.21.15.38:0/924001510 conn(0x55e1e39c2660 msgr2=0x55e1e39c2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 -- 172.21.15.38:0/924001510 shutdown_connections 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 -- 172.21.15.38:0/924001510 wait complete. 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.463+0000 7fef64776740 1 -- start start 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.464+0000 7fef64776740 1 Processor -- start 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.464+0000 7fef64776740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.464+0000 7fef64776740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.464+0000 7fef64776740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 0x55e1e39cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef64776740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e1e39d4220 con 0x55e1e39c7f30 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef64776740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e1e38e7e00 con 0x55e1e39d1430 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef64776740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e1e39d43a0 con 0x55e1e39c8850 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef64776740 10 client.0 _ll_register_callbacks cb 0x55e1e3922eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60730/0 (socket says 172.21.15.38:60730) 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.465+0000 7fef5b7fe640 1 -- 172.21.15.38:0/2921841935 learned_addr learned my addr 172.21.15.38:0/2921841935 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 -- 172.21.15.38:0/2921841935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 msgr2=0x55e1e39ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5affd640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 0x55e1e39cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 -- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 msgr2=0x55e1e39cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 0x55e1e39cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 -- 172.21.15.38:0/2921841935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e36403f0 con 0x55e1e39c7f30 2024-04-22T00:06:44.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 0x55e1e39cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fef44035000 tx=0x7fef44005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29663dcac92d29cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.466+0000 7fef3affd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44052020 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef54003d10 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fef4403a040 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef44004c40 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef64776740 10 client.17184 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e1e36000f0 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.467+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef440035f0 con 0x55e1e39c7f30 2024-04-22T00:06:44.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef4404d450 con 0x55e1e39c7f30 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 10 client.17184 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 20 client.17184 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e39e3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 4 client.17184 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.468+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fef54003d10 con 0x55e1e39e1850 2024-04-22T00:06:44.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.471+0000 7fef5affd640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e39e3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.474+0000 7fef5affd640 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e39e3ca0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45fee6a72bb76fab server_cookie=6837deabb719d338 in_seq=0 out_seq=0 2024-04-22T00:06:44.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.474+0000 7fef3affd640 10 client.17184 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:44.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.475+0000 7fef3affd640 1 -- 172.21.15.38:0/2921841935 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fef54003d10 con 0x55e1e39e1850 2024-04-22T00:06:44.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.475+0000 7fef3affd640 10 client.17184 handle_command_reply: tid=0 2024-04-22T00:06:44.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.475+0000 7fef64776740 1 client.17184 shutdown 2024-04-22T00:06:44.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.475+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 msgr2=0x55e1e39ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.475+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fef44035000 tx=0x7fef44005fd0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.476+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 shutdown_connections 2024-04-22T00:06:44.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.476+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39d1430 0x55e1e39cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.476+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39c8850 0x55e1e39ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e39e3ca0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef3affd640 0 client.17184 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 1 --2- 172.21.15.38:0/2921841935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c7f30 0x55e1e39ce4b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 >> 172.21.15.38:0/2921841935 conn(0x55e1e39c2660 msgr2=0x55e1e39c43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 shutdown_connections 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 1 -- 172.21.15.38:0/2921841935 wait complete. 2024-04-22T00:06:44.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.477+0000 7fef64776740 20 client.17184 trim_cache size 0 max 16384 2024-04-22T00:06:44.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.513+0000 7fef64776740 1 Processor -- start 2024-04-22T00:06:44.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.513+0000 7fef64776740 1 -- start start 2024-04-22T00:06:44.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c1281b0 con 0x55e1e39c8850 2024-04-22T00:06:44.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c1d0a10 con 0x55e1e39c7f30 2024-04-22T00:06:44.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef64776740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c1cf5a0 con 0x55e1e39d1430 2024-04-22T00:06:44.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef5bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60736/0 (socket says 172.21.15.38:60736) 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.514+0000 7fef5affd640 1 -- 172.21.15.38:0/2978507835 learned_addr learned my addr 172.21.15.38:0/2978507835 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 -- 172.21.15.38:0/2978507835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 msgr2=0x55e1e39ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 -- 172.21.15.38:0/2978507835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 msgr2=0x55e1e39cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 -- 172.21.15.38:0/2978507835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1e39c7090 con 0x55e1e39c8850 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5affd640 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fef4c09b5b0 tx=0x7fef4c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d9a00c0fd232b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.515+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:44.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.518+0000 7fef517fa640 1 -- 172.21.15.38:0/2978507835 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c01c040 con 0x55e1e39c8850 2024-04-22T00:06:44.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.518+0000 7fef517fa640 1 -- 172.21.15.38:0/2978507835 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef4c099950 con 0x55e1e39c8850 2024-04-22T00:06:44.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.518+0000 7fef517fa640 1 -- 172.21.15.38:0/2978507835 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c004ca0 con 0x55e1e39c8850 2024-04-22T00:06:44.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 -- 172.21.15.38:0/2978507835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 msgr2=0x55e1e39dd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fef4c09b5b0 tx=0x7fef4c009f60 comp rx=0 tx=0).stop 2024-04-22T00:06:44.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 -- 172.21.15.38:0/2978507835 shutdown_connections 2024-04-22T00:06:44.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39dd0e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 --2- 172.21.15.38:0/2978507835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.519+0000 7fef64776740 1 -- 172.21.15.38:0/2978507835 >> 172.21.15.38:0/2978507835 conn(0x55e1e39c2660 msgr2=0x55e1e39c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.520+0000 7fef64776740 1 -- 172.21.15.38:0/2978507835 shutdown_connections 2024-04-22T00:06:44.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.520+0000 7fef64776740 1 -- 172.21.15.38:0/2978507835 wait complete. 2024-04-22T00:06:44.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.520+0000 7fef64776740 1 -- start start 2024-04-22T00:06:44.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.520+0000 7fef64776740 1 Processor -- start 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef64776740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef64776740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef64776740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef64776740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c1d0340 con 0x55e1e39c8850 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef5affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60750/0 (socket says 172.21.15.38:60750) 2024-04-22T00:06:44.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef5affd640 1 -- 172.21.15.38:0/2702268867 learned_addr learned my addr 172.21.15.38:0/2702268867 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.521+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c1cfc70 con 0x55e1e39c7f30 2024-04-22T00:06:44.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c126190 con 0x55e1e39d1430 2024-04-22T00:06:44.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef64776740 10 client.0 _ll_register_callbacks cb 0x7fef44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 -- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 msgr2=0x55e1e39d9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39d9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 -- 172.21.15.38:0/2702268867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 msgr2=0x55e1e39cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.522+0000 7fef5bfff640 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e3a46de0 con 0x55e1e39d1430 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.523+0000 7fef5bfff640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39cb6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef5400fc30 tx=0x7fef54004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a86c38c69058b6d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.523+0000 7fef3affd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.523+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54005240 con 0x55e1e39d1430 2024-04-22T00:06:44.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.523+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fef540053e0 con 0x55e1e39d1430 2024-04-22T00:06:44.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.523+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef4c099950 con 0x55e1e39d1430 2024-04-22T00:06:44.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef54004e40 con 0x55e1e39d1430 2024-04-22T00:06:44.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef64776740 10 client.26461 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:44.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e1e3a47f00 con 0x55e1e39d1430 2024-04-22T00:06:44.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef54024460 con 0x55e1e39d1430 2024-04-22T00:06:44.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef54011440 con 0x55e1e39d1430 2024-04-22T00:06:44.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef64776740 10 client.26461 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:44.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.524+0000 7fef64776740 10 client.26461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:44.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef64776740 20 client.26461 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e3a4a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef64776740 4 client.26461 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fef4c099950 con 0x55e1e39e1850 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e3a4a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef5b7fe640 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e3a4a4b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24efb1eb5555581d server_cookie=5e1a1d1a10726049 in_seq=0 out_seq=0 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.525+0000 7fef3affd640 10 client.26461 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef3affd640 1 -- 172.21.15.38:0/2702268867 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fef4c099950 con 0x55e1e39e1850 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef3affd640 10 client.26461 handle_command_reply: tid=0 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef64776740 1 client.26461 shutdown 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 msgr2=0x55e1e39cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39cb6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef5400fc30 tx=0x7fef54004a00 comp rx=0 tx=0).stop 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 shutdown_connections 2024-04-22T00:06:44.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.526+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1e39d1430 0x55e1e39cb6a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.527+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e1e39e1850 0x55e1e3a4a4b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.527+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e1e39c8850 0x55e1e39cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.527+0000 7fef3affd640 0 client.26461 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:44.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.527+0000 7fef64776740 1 --2- 172.21.15.38:0/2702268867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e1e39c7f30 0x55e1e39d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.527+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 >> 172.21.15.38:0/2702268867 conn(0x55e1e39c2660 msgr2=0x55e1e39c39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.529+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 shutdown_connections 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.529+0000 7fef64776740 1 -- 172.21.15.38:0/2702268867 wait complete. 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.529+0000 7fef64776740 20 client.26461 trim_cache size 0 max 16384 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:06:44.524 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:44.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 msgr2=0x7fef480a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 0x7fef480a5310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44005b50 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 msgr2=0x7fef5c110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 0x7fef5c110370 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00e9f0 tx=0x7fef4c00eeb0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 shutdown_connections 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fef480a2e50 0x7fef480a5310 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef5c110900 0x7fef5c110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef5c100fd0 0x7fef5c110370 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 --2- 172.21.15.38:0/974343276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef5c100610 0x7fef5c1162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.530+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 >> 172.21.15.38:0/974343276 conn(0x7fef5c077740 msgr2=0x7fef5c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.531+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 shutdown_connections 2024-04-22T00:06:44.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.531+0000 7fef61b93640 1 -- 172.21.15.38:0/974343276 wait complete. 2024-04-22T00:06:44.537 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:06:44.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:06:44.557 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:44 smithi038 ceph-mon[27972]: pgmap v142: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 14 MiB/s wr, 3.50k op/s 2024-04-22T00:06:44.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.648+0000 7f43ecb37640 1 -- 172.21.15.38:0/4274155754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 msgr2=0x7f43e810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.648+0000 7f43ecb37640 1 --2- 172.21.15.38:0/4274155754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e810d4f0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009cd0 tx=0x7f43e002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- 172.21.15.38:0/4274155754 shutdown_connections 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- 172.21.15.38:0/4274155754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e810d4f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- 172.21.15.38:0/4274155754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 0x7f43e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- 172.21.15.38:0/4274155754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 0x7f43e8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- 172.21.15.38:0/4274155754 >> 172.21.15.38:0/4274155754 conn(0x7f43e8077740 msgr2=0x7f43e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- 172.21.15.38:0/4274155754 shutdown_connections 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- 172.21.15.38:0/4274155754 wait complete. 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 Processor -- start 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- start start 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 0x7f43e8198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 0x7f43e81a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8110a40 con 0x7f43e8100fe0 2024-04-22T00:06:44.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f43e81108c0 con 0x7f43e8106630 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.649+0000 7f43ecb37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8110bc0 con 0x7f43e8100620 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60764/0 (socket says 172.21.15.38:60764) 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 -- 172.21.15.38:0/2794448833 learned_addr learned my addr 172.21.15.38:0/2794448833 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e6d9d640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 0x7f43e81a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e659c640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 0x7f43e8198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 -- 172.21.15.38:0/2794448833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 msgr2=0x7f43e8198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 0x7f43e8198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 -- 172.21.15.38:0/2794448833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 msgr2=0x7f43e81a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 0x7f43e81a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 -- 172.21.15.38:0/2794448833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e81a4440 con 0x7f43e8100fe0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43e5d9b640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e9f0 tx=0x7f43dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc00cd80 con 0x7f43e8100fe0 2024-04-22T00:06:44.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43dc005500 con 0x7f43e8100fe0 2024-04-22T00:06:44.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e81a4610 con 0x7f43e8100fe0 2024-04-22T00:06:44.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc010690 con 0x7f43e8100fe0 2024-04-22T00:06:44.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.650+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e81a4a80 con 0x7f43e8100fe0 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.651+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e8068560 con 0x7f43e8100fe0 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.652+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f43dc0046c0 con 0x7f43e8100fe0 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.653+0000 7f43d77fe640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 0x7f43d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.653+0000 7f43e659c640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 0x7f43d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.653+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f43dc0cfcf0 con 0x7f43e8100fe0 2024-04-22T00:06:44.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.654+0000 7f43e659c640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 0x7f43d00a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009a20 tx=0x7f43d8005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:44.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:44.658+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f43dc0d4070 con 0x7f43e8100fe0 2024-04-22T00:06:44.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:44 smithi203 ceph-mon[31431]: pgmap v142: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 14 MiB/s wr, 3.50k op/s 2024-04-22T00:06:44.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:44 smithi111 ceph-mon[31543]: pgmap v142: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 14 MiB/s wr, 3.50k op/s 2024-04-22T00:06:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.016+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43e802d170 con 0x7f43e8100fe0 2024-04-22T00:06:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.017+0000 7f43d77fe640 1 -- 172.21.15.38:0/2794448833 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7f43dc09c5d0 con 0x7f43e8100fe0 2024-04-22T00:06:45.011 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:06:45.011 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:06:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:06:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 msgr2=0x7f43d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 0x7f43d00a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009a20 tx=0x7f43d8005f90 comp rx=0 tx=0).stop 2024-04-22T00:06:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 msgr2=0x7f43e81993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00e9f0 tx=0x7f43dc00edf0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 shutdown_connections 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f43d00a2d00 0x7f43d00a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43e8106630 0x7f43e81a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43e8100fe0 0x7f43e81993c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 --2- 172.21.15.38:0/2794448833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43e8100620 0x7f43e8198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 >> 172.21.15.38:0/2794448833 conn(0x7f43e8077740 msgr2=0x7f43e8102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 shutdown_connections 2024-04-22T00:06:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.019+0000 7f43ecb37640 1 -- 172.21.15.38:0/2794448833 wait complete. 2024-04-22T00:06:45.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:06:45.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 -- 172.21.15.38:0/3108789490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 msgr2=0x7f9154100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 --2- 172.21.15.38:0/3108789490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154100a90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9148009930 tx=0x7f914802bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 -- 172.21.15.38:0/3108789490 shutdown_connections 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 --2- 172.21.15.38:0/3108789490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f915410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 --2- 172.21.15.38:0/3108789490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154100a90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 --2- 172.21.15.38:0/3108789490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f9154106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 -- 172.21.15.38:0/3108789490 >> 172.21.15.38:0/3108789490 conn(0x7f9154077740 msgr2=0x7f9154077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 -- 172.21.15.38:0/3108789490 shutdown_connections 2024-04-22T00:06:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.186+0000 7f9159684640 1 -- 172.21.15.38:0/3108789490 wait complete. 2024-04-22T00:06:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 Processor -- start 2024-04-22T00:06:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 -- start start 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f91541985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f915419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9154110a80 con 0x7f9154100610 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9154110900 con 0x7f9154100fd0 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9159684640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9154110c00 con 0x7f91541066b0 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f91527fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f91541985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f91537fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f915419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60776/0 (socket says 172.21.15.38:60776) 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.187+0000 7f9152ffd640 1 -- 172.21.15.38:0/2946483755 learned_addr learned my addr 172.21.15.38:0/2946483755 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 -- 172.21.15.38:0/2946483755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 msgr2=0x7f915419c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f915419c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 -- 172.21.15.38:0/2946483755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 msgr2=0x7f91541985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f91541985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 -- 172.21.15.38:0/2946483755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915419d010 con 0x7f9154100610 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f91527fc640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f91541985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f91537fe640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f915419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9152ffd640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f914400d950 tx=0x7f914400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f913bfff640 1 -- 172.21.15.38:0/2946483755 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144017070 con 0x7f9154100610 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f913bfff640 1 -- 172.21.15.38:0/2946483755 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9144009c30 con 0x7f9154100610 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915419d300 con 0x7f9154100610 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f913bfff640 1 -- 172.21.15.38:0/2946483755 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914400f600 con 0x7f9154100610 2024-04-22T00:06:45.185 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.188+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541a4b90 con 0x7f9154100610 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.190+0000 7f913bfff640 1 -- 172.21.15.38:0/2946483755 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f914401a070 con 0x7f9154100610 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 Processor -- start 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f913bfff640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 0x7f913c0a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 -- start start 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91537fe640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 0x7f913c0a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 0x559f69f3a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 0x559f69f42fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f913bfff640 1 -- 172.21.15.38:0/2946483755 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f91440cf180 con 0x7f9154100610 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f69e9e530 con 0x559f69f43520 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559f69e46830 con 0x559f69f3a940 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f915c26b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f69e9a4f0 con 0x559f69f3a020 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 0x559f69f42fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 0x559f69f3a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f45910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60778/0 (socket says 172.21.15.38:60778) 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91537fe640 1 -- 172.21.15.38:0/157850405 learned_addr learned my addr 172.21.15.38:0/157850405 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 -- 172.21.15.38:0/157850405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 msgr2=0x559f69f3a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 0x559f69f3a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 -- 172.21.15.38:0/157850405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 msgr2=0x559f69f45910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f45910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91537fe640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 0x7f913c0a5200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f914c00eb80 tx=0x7f914c00ef80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f91527fc640 1 -- 172.21.15.38:0/157850405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f69e6a2c0 con 0x559f69f3a940 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.191+0000 7f9152ffd640 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 0x559f69f3a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.192+0000 7f91527fc640 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 0x559f69f42fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9148036040 tx=0x7f914802c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=455c57adcd906e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.192+0000 7f91397fa640 1 -- 172.21.15.38:0/157850405 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148045070 con 0x559f69f3a940 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.192+0000 7f91397fa640 1 -- 172.21.15.38:0/157850405 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f914802c880 con 0x559f69f3a940 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.192+0000 7f91397fa640 1 -- 172.21.15.38:0/157850405 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148003c60 con 0x559f69f3a940 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- 172.21.15.38:0/157850405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 msgr2=0x559f69f42fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 0x559f69f42fe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9148036040 tx=0x7f914802c680 comp rx=0 tx=0).stop 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- 172.21.15.38:0/157850405 shutdown_connections 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f45910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a940 0x559f69f42fe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- 172.21.15.38:0/157850405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a020 0x559f69f3a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- 172.21.15.38:0/157850405 >> 172.21.15.38:0/157850405 conn(0x559f69f34750 msgr2=0x559f69f34b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- 172.21.15.38:0/157850405 shutdown_connections 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- 172.21.15.38:0/157850405 wait complete. 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- start start 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 Processor -- start 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f40260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f40800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f69f462b0 con 0x559f69f3a940 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559f69e59f10 con 0x559f69f3a020 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.193+0000 7f915c26b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f69f46430 con 0x559f69f43520 2024-04-22T00:06:45.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f915c26b740 10 client.0 _ll_register_callbacks cb 0x559f69bb1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f91527fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f40260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55268/0 (socket says 172.21.15.38:55268) 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 -- 172.21.15.38:0/1181474593 learned_addr learned my addr 172.21.15.38:0/1181474593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f91537fe640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f40800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 -- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 msgr2=0x559f69f40800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f40800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 -- 172.21.15.38:0/1181474593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 msgr2=0x559f69f40260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f40260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 -- 172.21.15.38:0/1181474593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f69e2bc80 con 0x559f69f3a020 2024-04-22T00:06:45.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f91527fc640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f40260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9152ffd640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91440ccb90 tx=0x7f914409b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccbcaef03c12c3df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9136ff5640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144021070 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f914802c880 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f914409a860 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f914409aa60 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f915c26b740 10 client.26603 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.194+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f69bb23f0 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f914802c880 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f91440ccce0 con 0x559f69f3a020 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 10 client.26603 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 10 client.26603 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 20 client.26603 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69f53550 0x559f69f55940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 4 client.26603 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f69eaba70 con 0x559f69f53550 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f91527fc640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69f53550 0x559f69f55940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f9136ff5640 10 client.26603 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.195+0000 7f91527fc640 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69f53550 0x559f69f55940 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=221cde2b2208dfb8 server_cookie=629c87df9ac85280 in_seq=0 out_seq=0 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f9136ff5640 1 -- 172.21.15.38:0/1181474593 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f69eaba70 con 0x559f69f53550 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f9136ff5640 10 client.26603 handle_command_reply: tid=0 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 client.26603 shutdown 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 msgr2=0x559f69f42ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91440ccb90 tx=0x7f914409b410 comp rx=0 tx=0).stop 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 shutdown_connections 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69f53550 0x559f69f55940 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f40800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f40260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 --2- 172.21.15.38:0/1181474593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f42ed0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f9136ff5640 0 client.26603 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 >> 172.21.15.38:0/1181474593 conn(0x559f69f34750 msgr2=0x559f69f364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 shutdown_connections 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 1 -- 172.21.15.38:0/1181474593 wait complete. 2024-04-22T00:06:45.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.196+0000 7f915c26b740 20 client.26603 trim_cache size 0 max 16384 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 Processor -- start 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 -- start start 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 0x559f69f362b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91541ae960 con 0x559f69f43520 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91541b04a0 con 0x559f69f3a020 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f915c26b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91541af030 con 0x559f69f3a940 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f91537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f91537fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60800/0 (socket says 172.21.15.38:60800) 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f91537fe640 1 -- 172.21.15.38:0/3055285350 learned_addr learned my addr 172.21.15.38:0/3055285350 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f91527fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 0x559f69f362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f9152ffd640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f9152ffd640 1 -- 172.21.15.38:0/3055285350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 msgr2=0x559f69f362b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f9152ffd640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 0x559f69f362b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f9152ffd640 1 -- 172.21.15.38:0/3055285350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 msgr2=0x559f69f42990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.211+0000 7f9152ffd640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f9152ffd640 1 -- 172.21.15.38:0/3055285350 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f69f3a770 con 0x559f69f3a020 2024-04-22T00:06:45.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f91537fe640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f91527fc640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 0x559f69f362b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f9152ffd640 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4f170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9144097de0 tx=0x7f9144005b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f78e73e97647b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f9138ff9640 1 -- 172.21.15.38:0/3055285350 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144021070 con 0x559f69f3a020 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f9138ff9640 1 -- 172.21.15.38:0/3055285350 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9144097a80 con 0x559f69f3a020 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.212+0000 7f9138ff9640 1 -- 172.21.15.38:0/3055285350 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914409a2d0 con 0x559f69f3a020 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- 172.21.15.38:0/3055285350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 msgr2=0x559f69f4f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4f170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9144097de0 tx=0x7f9144005b90 comp rx=0 tx=0).stop 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- 172.21.15.38:0/3055285350 shutdown_connections 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f43520 0x559f69f42990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f3a940 0x559f69f362b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- 172.21.15.38:0/3055285350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4f170 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- 172.21.15.38:0/3055285350 >> 172.21.15.38:0/3055285350 conn(0x559f69f34750 msgr2=0x559f69f35ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- 172.21.15.38:0/3055285350 shutdown_connections 2024-04-22T00:06:45.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- 172.21.15.38:0/3055285350 wait complete. 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- start start 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 Processor -- start 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f47180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91541afdd0 con 0x559f69f3a940 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91541af700 con 0x559f69f3a020 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91541ac940 con 0x559f69f43520 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f915c26b740 10 client.0 _ll_register_callbacks cb 0x7f914409b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.213+0000 7f91527fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f91527fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60802/0 (socket says 172.21.15.38:60802) 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f91527fc640 1 -- 172.21.15.38:0/4284826047 learned_addr learned my addr 172.21.15.38:0/4284826047 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f91537fe640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f47180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 -- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 msgr2=0x559f69f47180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f47180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 -- 172.21.15.38:0/4284826047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 msgr2=0x559f69f46c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 -- 172.21.15.38:0/4284826047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f69c7c7d0 con 0x559f69f3a020 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f91527fc640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9152ffd640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4b7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9144097870 tx=0x7f9144097410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46ab84d7b28ffbe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9136ff5640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144021070 con 0x559f69f3a020 2024-04-22T00:06:45.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9144097a80 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f914409a4e0 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f91440bb350 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f915c26b740 10 client.26609 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f69fb96d0 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.214+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9144097a80 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f91440ccce0 con 0x559f69f3a020 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 10 client.26609 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 10 client.26609 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 20 client.26609 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69fb9b00 0x559f69fbbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 4 client.26609 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f69ea5dc0 con 0x559f69fb9b00 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f91527fc640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69fb9b00 0x559f69fbbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f9136ff5640 10 client.26609 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f91527fc640 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69fb9b00 0x559f69fbbef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b5eb1cc865b3b340 server_cookie=6f0bed6cb3f2e103 in_seq=0 out_seq=0 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f9136ff5640 1 -- 172.21.15.38:0/4284826047 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f69ea5dc0 con 0x559f69fb9b00 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f9136ff5640 10 client.26609 handle_command_reply: tid=0 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.215+0000 7f915c26b740 1 client.26609 shutdown 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 msgr2=0x559f69f4b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4b7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9144097870 tx=0x7f9144097410 comp rx=0 tx=0).stop 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 shutdown_connections 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x559f69fb9b00 0x559f69fbbef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f69f43520 0x559f69f47180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9136ff5640 0 client.26609 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f69f3a940 0x559f69f46c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 --2- 172.21.15.38:0/4284826047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f69f3a020 0x559f69f4b7d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 >> 172.21.15.38:0/4284826047 conn(0x559f69f34750 msgr2=0x559f69f36410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 shutdown_connections 2024-04-22T00:06:45.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 1 -- 172.21.15.38:0/4284826047 wait complete. 2024-04-22T00:06:45.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f915c26b740 20 client.26609 trim_cache size 0 max 16384 2024-04-22T00:06:45.218 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 msgr2=0x7f913c0a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 0x7f913c0a5200 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f914c00eb80 tx=0x7f914c00ef80 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 msgr2=0x7f9154198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f914400d950 tx=0x7f914400de10 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 shutdown_connections 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f913c0a2d40 0x7f913c0a5200 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91541066b0 0x7f915419c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9154100fd0 0x7f91541985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 --2- 172.21.15.38:0/2946483755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9154100610 0x7f9154198070 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 >> 172.21.15.38:0/2946483755 conn(0x7f9154077740 msgr2=0x7f91540ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.216+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 shutdown_connections 2024-04-22T00:06:45.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.217+0000 7f9159684640 1 -- 172.21.15.38:0/2946483755 wait complete. 2024-04-22T00:06:45.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:06:45.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.335+0000 7f525f4fa640 1 -- 172.21.15.38:0/3868418237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5258100fd0 msgr2=0x7f525810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.335+0000 7f525f4fa640 1 --2- 172.21.15.38:0/3868418237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5258100fd0 0x7f525810d4e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5250009cd0 tx=0x7f525002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.336+0000 7f525f4fa640 1 -- 172.21.15.38:0/3868418237 shutdown_connections 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.336+0000 7f525f4fa640 1 --2- 172.21.15.38:0/3868418237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5258100fd0 0x7f525810d4e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.336+0000 7f525f4fa640 1 --2- 172.21.15.38:0/3868418237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5258100610 0x7f5258100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.336+0000 7f525f4fa640 1 --2- 172.21.15.38:0/3868418237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52581066b0 0x7f5258106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.336+0000 7f525f4fa640 1 -- 172.21.15.38:0/3868418237 >> 172.21.15.38:0/3868418237 conn(0x7f5258077740 msgr2=0x7f5258077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.337+0000 7f525f4fa640 1 -- 172.21.15.38:0/3868418237 shutdown_connections 2024-04-22T00:06:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.337+0000 7f525f4fa640 1 -- 172.21.15.38:0/3868418237 wait complete. 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.338+0000 7f525f4fa640 1 Processor -- start 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.338+0000 7f525f4fa640 1 -- start start 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.338+0000 7f525f4fa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5258100610 0x7f5258073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.338+0000 7f525f4fa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5258100fd0 0x7f525806f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.338+0000 7f525f4fa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525f4fa640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f525806bb70 con 0x7f52581066b0 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525da97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525da97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60812/0 (socket says 172.21.15.38:60812) 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525da97640 1 -- 172.21.15.38:0/138062062 learned_addr learned my addr 172.21.15.38:0/138062062 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f525806b9f0 con 0x7f5258100610 2024-04-22T00:06:45.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.339+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f525806bcf0 con 0x7f5258100fd0 2024-04-22T00:06:45.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.341+0000 7f525da97640 1 -- 172.21.15.38:0/138062062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5258100fd0 msgr2=0x7f525806f300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:06:45.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.341+0000 7f525da97640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5258100fd0 0x7f525806f300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.341+0000 7f525da97640 1 -- 172.21.15.38:0/138062062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5258100610 msgr2=0x7f5258073220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.341+0000 7f525da97640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5258100610 0x7f5258073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.341+0000 7f525da97640 1 -- 172.21.15.38:0/138062062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52580700b0 con 0x7f52581066b0 2024-04-22T00:06:45.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.342+0000 7f525da97640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5250005e30 tx=0x7f5250032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.342+0000 7f524e7fc640 1 -- 172.21.15.38:0/138062062 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250041020 con 0x7f52581066b0 2024-04-22T00:06:45.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.343+0000 7f524e7fc640 1 -- 172.21.15.38:0/138062062 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5250034cd0 con 0x7f52581066b0 2024-04-22T00:06:45.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.343+0000 7f524e7fc640 1 -- 172.21.15.38:0/138062062 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250039040 con 0x7f52581066b0 2024-04-22T00:06:45.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.343+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52581aee90 con 0x7f52581066b0 2024-04-22T00:06:45.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.343+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52581af300 con 0x7f52581066b0 2024-04-22T00:06:45.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.346+0000 7f524e7fc640 1 -- 172.21.15.38:0/138062062 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5250044020 con 0x7f52581066b0 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.346+0000 7f52621b6740 1 Processor -- start 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.346+0000 7f52621b6740 1 -- start start 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf60cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf568530 con 0x55bdaf604940 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf510830 con 0x55bdaf604020 2024-04-22T00:06:45.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f52621b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf5644f0 con 0x55bdaf60d520 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f525ca95640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f525ca95640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55308/0 (socket says 172.21.15.38:55308) 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f525ca95640 1 -- 172.21.15.38:0/4143732565 learned_addr learned my addr 172.21.15.38:0/4143732565 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f525ca95640 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.347+0000 7f525d296640 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf60cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.348+0000 7f525ca95640 1 -- 172.21.15.38:0/4143732565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 msgr2=0x55bdaf60f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.348+0000 7f525ca95640 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.348+0000 7f525ca95640 1 -- 172.21.15.38:0/4143732565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 msgr2=0x55bdaf60cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.348+0000 7f525ca95640 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf60cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.348+0000 7f525ca95640 1 -- 172.21.15.38:0/4143732565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaf5342c0 con 0x55bdaf604020 2024-04-22T00:06:45.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.349+0000 7f525ca95640 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5248009cf0 tx=0x7f5248016c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d4cd9574b700018 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.349+0000 7f52437fe640 1 -- 172.21.15.38:0/4143732565 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524800cb60 con 0x55bdaf604020 2024-04-22T00:06:45.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.349+0000 7f52437fe640 1 -- 172.21.15.38:0/4143732565 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5248005500 con 0x55bdaf604020 2024-04-22T00:06:45.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.349+0000 7f52437fe640 1 -- 172.21.15.38:0/4143732565 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248010460 con 0x55bdaf604020 2024-04-22T00:06:45.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.350+0000 7f52621b6740 1 -- 172.21.15.38:0/4143732565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 msgr2=0x55bdaf604400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.350+0000 7f52621b6740 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5248009cf0 tx=0x7f5248016c40 comp rx=0 tx=0).stop 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.350+0000 7f52621b6740 1 -- 172.21.15.38:0/4143732565 shutdown_connections 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f52621b6740 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f52621b6740 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf60cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f52621b6740 1 --2- 172.21.15.38:0/4143732565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604020 0x55bdaf604400 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f52621b6740 1 -- 172.21.15.38:0/4143732565 >> 172.21.15.38:0/4143732565 conn(0x55bdaf5fe750 msgr2=0x55bdaf5feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f524e7fc640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 0x7f52440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.351+0000 7f524e7fc640 1 -- 172.21.15.38:0/138062062 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52500f1370 con 0x7f52581066b0 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.352+0000 7f525d296640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 0x7f52440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.352+0000 7f52621b6740 1 -- 172.21.15.38:0/4143732565 shutdown_connections 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.352+0000 7f52621b6740 1 -- 172.21.15.38:0/4143732565 wait complete. 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.352+0000 7f525d296640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 0x7f52440a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55bdaf603af0 tx=0x7f5254009640 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.352+0000 7f52621b6740 1 -- start start 2024-04-22T00:06:45.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 Processor -- start 2024-04-22T00:06:45.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf60cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 0x55bdaf60a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf6102b0 con 0x55bdaf60d520 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf523f10 con 0x55bdaf604940 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f52621b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf610430 con 0x55bdaf604020 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f525d296640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf60cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f525da97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 0x55bdaf60a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f525d296640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf60cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42412/0 (socket says 172.21.15.38:42412) 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.353+0000 7f525d296640 1 -- 172.21.15.38:0/3291655709 learned_addr learned my addr 172.21.15.38:0/3291655709 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525ca95640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f52621b6740 10 client.0 _ll_register_callbacks cb 0x55bdaf27bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525da97640 1 -- 172.21.15.38:0/3291655709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 msgr2=0x55bdaf60cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525da97640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf60cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525da97640 1 -- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 msgr2=0x55bdaf60a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525da97640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.354+0000 7f525da97640 1 -- 172.21.15.38:0/3291655709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaf4f5c80 con 0x55bdaf60d520 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.355+0000 7f525da97640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 0x55bdaf60a6a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f52500ee890 tx=0x7f52500b99c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b43ee8c04d8079 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.355+0000 7f5240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.355+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52500bcfc0 con 0x55bdaf60d520 2024-04-22T00:06:45.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.355+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52500b9cd0 con 0x55bdaf60d520 2024-04-22T00:06:45.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.356+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5248005500 con 0x55bdaf60d520 2024-04-22T00:06:45.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.356+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5250034920 con 0x55bdaf60d520 2024-04-22T00:06:45.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.356+0000 7f52621b6740 10 client.17226 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.356+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bdaf27c3f0 con 0x55bdaf60d520 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.356+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5248005500 con 0x55bdaf60d520 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f525003c030 con 0x55bdaf60d520 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 10 client.17226 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 10 client.17226 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 20 client.17226 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf61f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 4 client.17226 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.357+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdaf575a70 con 0x55bdaf61d450 2024-04-22T00:06:45.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.358+0000 7f525d296640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf61f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.358+0000 7f525d296640 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf61f8e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9da7ec6880863cf7 server_cookie=b75b55efbd6370a5 in_seq=0 out_seq=0 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.358+0000 7f5240ff9640 10 client.17226 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.359+0000 7f5240ff9640 1 -- 172.21.15.38:0/3291655709 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdaf575a70 con 0x55bdaf61d450 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.359+0000 7f5240ff9640 10 client.17226 handle_command_reply: tid=0 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.359+0000 7f52621b6740 1 client.17226 shutdown 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.359+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 msgr2=0x55bdaf60a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.359+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 0x55bdaf60a6a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f52500ee890 tx=0x7f52500b99c0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 shutdown_connections 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf61f8e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf60d520 0x55bdaf60a6a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f5240ff9640 0 client.17226 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:45.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 --2- 172.21.15.38:0/3291655709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf60cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.360+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 >> 172.21.15.38:0/3291655709 conn(0x55bdaf5fe750 msgr2=0x55bdaf6004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.361+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 shutdown_connections 2024-04-22T00:06:45.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.361+0000 7f52621b6740 1 -- 172.21.15.38:0/3291655709 wait complete. 2024-04-22T00:06:45.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.361+0000 7f52621b6740 20 client.17226 trim_cache size 0 max 16384 2024-04-22T00:06:45.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.387+0000 7f52621b6740 1 Processor -- start 2024-04-22T00:06:45.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.387+0000 7f52621b6740 1 -- start start 2024-04-22T00:06:45.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b4fe0 con 0x55bdaf604020 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b6b20 con 0x55bdaf604940 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f52621b6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b56b0 con 0x55bdaf60d520 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f525d296640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f525d296640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60850/0 (socket says 172.21.15.38:60850) 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f525d296640 1 -- 172.21.15.38:0/1093547985 learned_addr learned my addr 172.21.15.38:0/1093547985 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f525ca95640 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.388+0000 7f525da97640 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 -- 172.21.15.38:0/1093547985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 msgr2=0x55bdaf60cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 -- 172.21.15.38:0/1093547985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 msgr2=0x55bdaf60c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 -- 172.21.15.38:0/1093547985 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaf3467d0 con 0x55bdaf604020 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f525d296640 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5254005be0 tx=0x7f5254028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a791d96a126ccfb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f5242ffd640 1 -- 172.21.15.38:0/1093547985 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254037540 con 0x55bdaf604020 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f5242ffd640 1 -- 172.21.15.38:0/1093547985 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525403b030 con 0x55bdaf604020 2024-04-22T00:06:45.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.389+0000 7f5242ffd640 1 -- 172.21.15.38:0/1093547985 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525403f630 con 0x55bdaf604020 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 -- 172.21.15.38:0/1093547985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 msgr2=0x55bdaf6191d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5254005be0 tx=0x7f5254028000 comp rx=0 tx=0).stop 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 -- 172.21.15.38:0/1093547985 shutdown_connections 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf60d520 0x55bdaf60cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf604940 0x55bdaf60c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 --2- 172.21.15.38:0/1093547985 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604020 0x55bdaf6191d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.390+0000 7f52621b6740 1 -- 172.21.15.38:0/1093547985 >> 172.21.15.38:0/1093547985 conn(0x55bdaf5fe750 msgr2=0x55bdaf600410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.391+0000 7f52621b6740 1 -- 172.21.15.38:0/1093547985 shutdown_connections 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.391+0000 7f52621b6740 1 -- 172.21.15.38:0/1093547985 wait complete. 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.391+0000 7f52621b6740 1 -- start start 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.391+0000 7f52621b6740 1 Processor -- start 2024-04-22T00:06:45.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf6112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf6117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 0x55bdaf611d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f525d296640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf6112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f525ca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf6117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f525ca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf6117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60858/0 (socket says 172.21.15.38:60858) 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f525ca95640 1 -- 172.21.15.38:0/2068412986 learned_addr learned my addr 172.21.15.38:0/2068412986 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b6450 con 0x55bdaf604940 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f525da97640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 0x55bdaf611d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b5d80 con 0x55bdaf60d520 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.392+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52581b3040 con 0x55bdaf604020 2024-04-22T00:06:45.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f52621b6740 10 client.0 _ll_register_callbacks cb 0x7f5250009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 -- 172.21.15.38:0/2068412986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 msgr2=0x55bdaf6112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf6112a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 -- 172.21.15.38:0/2068412986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 msgr2=0x55bdaf6117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf6117e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaf682bf0 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f525da97640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 0x55bdaf611d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f52500bd5f0 tx=0x7f525002c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9169e2293792f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f5240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5250034920 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.393+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5250004b40 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525403b030 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f52500a3050 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 10 client.26624 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bdaf683e90 con 0x55bdaf60d520 2024-04-22T00:06:45.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52500da070 con 0x55bdaf60d520 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f52500bcab0 con 0x55bdaf60d520 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 10 client.26624 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 10 client.26624 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 20 client.26624 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf6864c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 4 client.26624 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.394+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f525403b030 con 0x55bdaf61d450 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f525d296640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf6864c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f5240ff9640 10 client.26624 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f525d296640 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf6864c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34f70e1cbba3a503 server_cookie=623687a64b2e936 in_seq=0 out_seq=0 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f5240ff9640 1 -- 172.21.15.38:0/2068412986 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f525403b030 con 0x55bdaf61d450 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f5240ff9640 10 client.26624 handle_command_reply: tid=0 2024-04-22T00:06:45.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.395+0000 7f52621b6740 1 client.26624 shutdown 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.396+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 msgr2=0x55bdaf611d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.396+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 0x55bdaf611d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f52500bd5f0 tx=0x7f525002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 shutdown_connections 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bdaf60d520 0x55bdaf611d20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55bdaf61d450 0x55bdaf6864c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdaf604940 0x55bdaf6117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 --2- 172.21.15.38:0/2068412986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bdaf604020 0x55bdaf6112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.398+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 >> 172.21.15.38:0/2068412986 conn(0x55bdaf5fe750 msgr2=0x55bdaf600410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.399+0000 7f5240ff9640 0 client.26624 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:06:45.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.400+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 shutdown_connections 2024-04-22T00:06:45.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.400+0000 7f52621b6740 1 -- 172.21.15.38:0/2068412986 wait complete. 2024-04-22T00:06:45.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.401+0000 7f52621b6740 20 client.26624 trim_cache size 0 max 16384 2024-04-22T00:06:45.396 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.402+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 msgr2=0x7f52440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.402+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 0x7f52440a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55bdaf603af0 tx=0x7f5254009640 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.402+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 msgr2=0x7f525806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.402+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5250005e30 tx=0x7f5250032ea0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 shutdown_connections 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52440a2d80 0x7f52440a5240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52581066b0 0x7f525806f840 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5258100fd0 0x7f525806f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 --2- 172.21.15.38:0/138062062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5258100610 0x7f5258073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 >> 172.21.15.38:0/138062062 conn(0x7f5258077740 msgr2=0x7f5258105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 shutdown_connections 2024-04-22T00:06:45.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.403+0000 7f525f4fa640 1 -- 172.21.15.38:0/138062062 wait complete. 2024-04-22T00:06:45.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:06:45.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.609+0000 7f8a7af93640 1 -- 172.21.15.38:0/121383480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 msgr2=0x7f8a741122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.609+0000 7f8a7af93640 1 --2- 172.21.15.38:0/121383480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741122b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009930 tx=0x7f8a6402bd10 comp rx=0 tx=0).stop 2024-04-22T00:06:45.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.611+0000 7f8a7af93640 1 -- 172.21.15.38:0/121383480 shutdown_connections 2024-04-22T00:06:45.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.612+0000 7f8a7af93640 1 --2- 172.21.15.38:0/121383480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a741127f0 0x7f8a74114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.612+0000 7f8a7af93640 1 --2- 172.21.15.38:0/121383480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741122b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.612+0000 7f8a7af93640 1 --2- 172.21.15.38:0/121383480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a74073a10 0x7f8a74073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.612+0000 7f8a7af93640 1 -- 172.21.15.38:0/121383480 >> 172.21.15.38:0/121383480 conn(0x7f8a74078b90 msgr2=0x7f8a74078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.613+0000 7f8a7af93640 1 -- 172.21.15.38:0/121383480 shutdown_connections 2024-04-22T00:06:45.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.613+0000 7f8a7af93640 1 -- 172.21.15.38:0/121383480 wait complete. 2024-04-22T00:06:45.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.614+0000 7f8a7af93640 1 Processor -- start 2024-04-22T00:06:45.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.614+0000 7f8a7af93640 1 -- start start 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a7af93640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 0x7f8a741a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a7af93640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741a4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a7af93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 0x7f8a7419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a78d2f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 0x7f8a741a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a78d2f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 0x7f8a741a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55346/0 (socket says 172.21.15.38:55346) 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a78d2f640 1 -- 172.21.15.38:0/237236540 learned_addr learned my addr 172.21.15.38:0/237236540 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a7af93640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8a74118260 con 0x7f8a74074330 2024-04-22T00:06:45.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.615+0000 7f8a73fff640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 0x7f8a7419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741180e0 con 0x7f8a74073a10 2024-04-22T00:06:45.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741183e0 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 -- 172.21.15.38:0/237236540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 msgr2=0x7f8a741a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 0x7f8a741a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 -- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 msgr2=0x7f8a741a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741a4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.616+0000 7f8a79530640 1 -- 172.21.15.38:0/237236540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7419edc0 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a79530640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 0x7f8a7419e640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00bdd0 tx=0x7f8a6c009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a71ffb640 1 -- 172.21.15.38:0/237236540 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c019070 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7419f0b0 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a71ffb640 1 -- 172.21.15.38:0/237236540 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c00cdf0 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a71ffb640 1 -- 172.21.15.38:0/237236540 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0128e0 con 0x7f8a741127f0 2024-04-22T00:06:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.617+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a741ab300 con 0x7f8a741127f0 2024-04-22T00:06:45.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.619+0000 7f8a71ffb640 1 -- 172.21.15.38:0/237236540 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8a6c01c070 con 0x7f8a741127f0 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 Processor -- start 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 -- start start 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a71ffb640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 0x7f8a5c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969cc3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969ccbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a71ffb640 1 -- 172.21.15.38:0/237236540 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8a6c0cee30 con 0x7f8a741127f0 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557969c27530 con 0x557969ccc520 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557969bcf830 con 0x557969cc3020 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a7db77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557969c234f0 con 0x557969cc3940 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.620+0000 7f8a73fff640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 0x7f8a5c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60888/0 (socket says 172.21.15.38:60888) 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 -- 172.21.15.38:0/2938001083 learned_addr learned my addr 172.21.15.38:0/2938001083 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969cc3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a73fff640 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969ccbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 -- 172.21.15.38:0/2938001083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 msgr2=0x557969ccbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969ccbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 -- 172.21.15.38:0/2938001083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 msgr2=0x557969cc3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969cc3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 -- 172.21.15.38:0/2938001083 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557969bf32c0 con 0x557969ccc520 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a79530640 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c097c30 tx=0x7f8a6c09b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3963117a4af4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a73fff640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 0x7f8a5c0a5280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000c00 tx=0x7f8a64004310 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a5affd640 1 -- 172.21.15.38:0/2938001083 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0a3d90 con 0x557969ccc520 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.621+0000 7f8a5affd640 1 -- 172.21.15.38:0/2938001083 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c099560 con 0x557969ccc520 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a5affd640 1 -- 172.21.15.38:0/2938001083 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0a3b60 con 0x557969ccc520 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 -- 172.21.15.38:0/2938001083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 msgr2=0x557969cce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c097c30 tx=0x7f8a6c09b260 comp rx=0 tx=0).stop 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 -- 172.21.15.38:0/2938001083 shutdown_connections 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cce910 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969ccbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 --2- 172.21.15.38:0/2938001083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969cc3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 -- 172.21.15.38:0/2938001083 >> 172.21.15.38:0/2938001083 conn(0x557969cbd750 msgr2=0x557969cbdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 -- 172.21.15.38:0/2938001083 shutdown_connections 2024-04-22T00:06:45.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.622+0000 7f8a7db77740 1 -- 172.21.15.38:0/2938001083 wait complete. 2024-04-22T00:06:45.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 -- start start 2024-04-22T00:06:45.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 Processor -- start 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969cc9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cc9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557969ccf300 con 0x557969ccc520 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557969be2f10 con 0x557969cc3020 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557969ccf480 con 0x557969cc3940 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a7db77740 10 client.0 _ll_register_callbacks cb 0x55796993abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a78d2f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969cc9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a79530640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cc9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a78d2f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55374/0 (socket says 172.21.15.38:55374) 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.623+0000 7f8a78d2f640 1 -- 172.21.15.38:0/3426292392 learned_addr learned my addr 172.21.15.38:0/3426292392 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 -- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 msgr2=0x557969cc9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969cc9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 -- 172.21.15.38:0/3426292392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 msgr2=0x557969cc9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cc9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 -- 172.21.15.38:0/3426292392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557969bb4c80 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a73fff640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969cc9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a79530640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cc9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a68011640 tx=0x7f8a68011b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ceba6f8d2dc6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a53fff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800f780 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8a6800f920 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a6c099560 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8a68026050 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a7db77740 10 client.26633 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.624+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55796993b3f0 con 0x557969cc3020 2024-04-22T00:06:45.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c099560 con 0x557969cc3020 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8a6801d400 con 0x557969cc3020 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 10 client.26633 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 10 client.26633 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 20 client.26633 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969cdc760 0x557969cdeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 4 client.26633 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557969c34a70 con 0x557969cdc760 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a73fff640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969cdc760 0x557969cdeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a73fff640 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969cdc760 0x557969cdeb50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=71d02f81ee7efe91 server_cookie=8626c4fb1e03b6b4 in_seq=0 out_seq=0 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.625+0000 7f8a53fff640 10 client.26633 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a53fff640 1 -- 172.21.15.38:0/3426292392 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557969c34a70 con 0x557969cdc760 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a53fff640 10 client.26633 handle_command_reply: tid=0 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 client.26633 shutdown 2024-04-22T00:06:45.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 msgr2=0x557969ccbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a68011640 tx=0x7f8a68011b00 comp rx=0 tx=0).stop 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 shutdown_connections 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969cdc760 0x557969cdeb50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969cc9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3940 0x557969cc9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 --2- 172.21.15.38:0/3426292392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3020 0x557969ccbef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a53fff640 0 client.26633 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 >> 172.21.15.38:0/3426292392 conn(0x557969cbd750 msgr2=0x557969cbf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 shutdown_connections 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.626+0000 7f8a7db77740 1 -- 172.21.15.38:0/3426292392 wait complete. 2024-04-22T00:06:45.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.627+0000 7f8a7db77740 20 client.26633 trim_cache size 0 max 16384 2024-04-22T00:06:45.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 Processor -- start 2024-04-22T00:06:45.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 -- start start 2024-04-22T00:06:45.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969ccb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3940 0x557969ccbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b5130 con 0x557969ccc520 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b6c70 con 0x557969cc3940 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a7db77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b5800 con 0x557969cc3020 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a79530640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a79530640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60904/0 (socket says 172.21.15.38:60904) 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a79530640 1 -- 172.21.15.38:0/602026520 learned_addr learned my addr 172.21.15.38:0/602026520 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969ccb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.640+0000 7f8a73fff640 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3940 0x557969ccbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 -- 172.21.15.38:0/602026520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 msgr2=0x557969ccb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969ccb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 -- 172.21.15.38:0/602026520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3940 msgr2=0x557969ccbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3940 0x557969ccbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 -- 172.21.15.38:0/602026520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557969cc9c00 con 0x557969ccc520 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a79530640 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c09af60 tx=0x7f8a6c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc18d7fe55f3d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a5a7fc640 1 -- 172.21.15.38:0/602026520 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0b3070 con 0x557969ccc520 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a5a7fc640 1 -- 172.21.15.38:0/602026520 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c0a3d90 con 0x557969ccc520 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.641+0000 7f8a5a7fc640 1 -- 172.21.15.38:0/602026520 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0a3900 con 0x557969ccc520 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- 172.21.15.38:0/602026520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 msgr2=0x557969ce4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c09af60 tx=0x7f8a6c004a80 comp rx=0 tx=0).stop 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- 172.21.15.38:0/602026520 shutdown_connections 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969ccc520 0x557969ce4380 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969cc3940 0x557969ccbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- 172.21.15.38:0/602026520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969ccb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- 172.21.15.38:0/602026520 >> 172.21.15.38:0/602026520 conn(0x557969cbd750 msgr2=0x557969cbe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- 172.21.15.38:0/602026520 shutdown_connections 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- 172.21.15.38:0/602026520 wait complete. 2024-04-22T00:06:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- start start 2024-04-22T00:06:45.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 Processor -- start 2024-04-22T00:06:45.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969cd01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 0x557969cd0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b65a0 con 0x557969cc3940 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b5ed0 con 0x557969ccc520 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.642+0000 7f8a7db77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a741b3110 con 0x557969cc3020 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a7db77740 10 client.0 _ll_register_callbacks cb 0x7f8a68009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a79530640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 0x557969cd0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60914/0 (socket says 172.21.15.38:60914) 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 -- 172.21.15.38:0/1796430216 learned_addr learned my addr 172.21.15.38:0/1796430216 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969cd01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 -- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 msgr2=0x557969cd01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969cd01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 -- 172.21.15.38:0/1796430216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 msgr2=0x557969cd0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 0x557969cd0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 -- 172.21.15.38:0/1796430216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557969a057d0 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a79530640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 0x557969cd0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a73fff640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8a6402cfd0 tx=0x7f8a6402bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4b5e58291205ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a53fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64059070 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8a6403a040 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a6c0a3d90 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8a64065050 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a7db77740 10 client.17250 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.643+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557969d42e50 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c0a3d90 con 0x557969cc3940 2024-04-22T00:06:45.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8a6404d040 con 0x557969cc3940 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 10 client.17250 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 10 client.17250 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 20 client.17250 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969d43580 0x557969d459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 4 client.17250 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557969c2edc0 con 0x557969d43580 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969d43580 0x557969d459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a78d2f640 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969d43580 0x557969d459a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e4af57b7d4a7f1b server_cookie=2cb12a8e93fde9d5 in_seq=0 out_seq=0 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a53fff640 10 client.17250 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a53fff640 1 -- 172.21.15.38:0/1796430216 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557969c2edc0 con 0x557969d43580 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a53fff640 10 client.17250 handle_command_reply: tid=0 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.644+0000 7f8a7db77740 1 client.17250 shutdown 2024-04-22T00:06:45.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 msgr2=0x557969cd0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8a6402cfd0 tx=0x7f8a6402bd10 comp rx=0 tx=0).stop 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 shutdown_connections 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x557969d43580 0x557969d459a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557969ccc520 0x557969cd0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a53fff640 0 client.17250 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557969cc3940 0x557969cd0710 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 --2- 172.21.15.38:0/1796430216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557969cc3020 0x557969cd01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 >> 172.21.15.38:0/1796430216 conn(0x557969cbd750 msgr2=0x557969cc3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 shutdown_connections 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 1 -- 172.21.15.38:0/1796430216 wait complete. 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7db77740 20 client.17250 trim_cache size 0 max 16384 2024-04-22T00:06:45.647 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:45.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 msgr2=0x7f8a5c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 0x7f8a5c0a5280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a64000c00 tx=0x7f8a64004310 comp rx=0 tx=0).stop 2024-04-22T00:06:45.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 msgr2=0x7f8a7419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 0x7f8a7419e640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00bdd0 tx=0x7f8a6c009ed0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 shutdown_connections 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8a5c0a2dc0 0x7f8a5c0a5280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a741127f0 0x7f8a7419e640 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a74074330 0x7f8a741a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 --2- 172.21.15.38:0/237236540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a74073a10 0x7f8a741a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.645+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 >> 172.21.15.38:0/237236540 conn(0x7f8a74078b90 msgr2=0x7f8a740fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.646+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 shutdown_connections 2024-04-22T00:06:45.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.646+0000 7f8a7af93640 1 -- 172.21.15.38:0/237236540 wait complete. 2024-04-22T00:06:45.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:06:45.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:45 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2794448833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:45.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- 172.21.15.38:0/4126846083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f50100610 msgr2=0x7f3f50100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4126846083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f50100610 0x7f3f50100a90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009930 tx=0x7f3f4002bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- 172.21.15.38:0/4126846083 shutdown_connections 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4126846083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f5010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4126846083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f50100610 0x7f3f50100a90 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4126846083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f501066b0 0x7f3f50106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- 172.21.15.38:0/4126846083 >> 172.21.15.38:0/4126846083 conn(0x7f3f500769c0 msgr2=0x7f3f50076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- 172.21.15.38:0/4126846083 shutdown_connections 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- 172.21.15.38:0/4126846083 wait complete. 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 Processor -- start 2024-04-22T00:06:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.773+0000 7f3f57a40640 1 -- start start 2024-04-22T00:06:45.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f50100610 0x7f3f50198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f50198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5006ac40 con 0x7f3f501066b0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5006aac0 con 0x7f3f50100610 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f57a40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50111d40 con 0x7f3f50100fd0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60924/0 (socket says 172.21.15.38:60924) 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 -- 172.21.15.38:0/4091576072 learned_addr learned my addr 172.21.15.38:0/4091576072 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f50198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f50100610 0x7f3f50198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 -- 172.21.15.38:0/4091576072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 msgr2=0x7f3f50198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f50198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 -- 172.21.15.38:0/4091576072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f50100610 msgr2=0x7f3f50198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f50100610 0x7f3f50198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 -- 172.21.15.38:0/4091576072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5019d000 con 0x7f3f501066b0 2024-04-22T00:06:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f50198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f55fdd640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800eb10 tx=0x7f3f4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.774+0000 7f3f467fc640 1 -- 172.21.15.38:0/4091576072 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48004b20 con 0x7f3f501066b0 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.775+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5019d2f0 con 0x7f3f501066b0 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.775+0000 7f3f467fc640 1 -- 172.21.15.38:0/4091576072 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48004cc0 con 0x7f3f501066b0 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.775+0000 7f3f467fc640 1 -- 172.21.15.38:0/4091576072 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48010760 con 0x7f3f501066b0 2024-04-22T00:06:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.775+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f501a4b80 con 0x7f3f501066b0 2024-04-22T00:06:45.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f467fc640 1 -- 172.21.15.38:0/4091576072 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3f48005020 con 0x7f3f501066b0 2024-04-22T00:06:45.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 Processor -- start 2024-04-22T00:06:45.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 -- start start 2024-04-22T00:06:45.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4940 0x55efb7afcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7afd520 0x55efb7aff910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55efb7a58530 con 0x55efb7af4020 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55efb7a00830 con 0x55efb7af4940 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55efb7a544f0 con 0x55efb7afd520 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60934/0 (socket says 172.21.15.38:60934) 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f54fdb640 1 -- 172.21.15.38:0/1896402683 learned_addr learned my addr 172.21.15.38:0/1896402683 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7afd520 0x55efb7aff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f467fc640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.778+0000 7f3f557dc640 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4940 0x55efb7afcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f467fc640 1 -- 172.21.15.38:0/4091576072 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3f480cff20 con 0x7f3f501066b0 2024-04-22T00:06:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f55fdd640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 -- 172.21.15.38:0/1896402683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7afd520 msgr2=0x55efb7aff910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7afd520 0x55efb7aff910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 -- 172.21.15.38:0/1896402683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4940 msgr2=0x55efb7afcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4940 0x55efb7afcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 -- 172.21.15.38:0/1896402683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb7a242c0 con 0x55efb7af4020 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.779+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005c40 tx=0x7f3f400058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ac69b887f2c166 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.780+0000 7f3f3b7fe640 1 -- 172.21.15.38:0/1896402683 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40004020 con 0x55efb7af4020 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.780+0000 7f3f3b7fe640 1 -- 172.21.15.38:0/1896402683 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f40005530 con 0x55efb7af4020 2024-04-22T00:06:45.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.780+0000 7f3f3b7fe640 1 -- 172.21.15.38:0/1896402683 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40004ae0 con 0x55efb7af4020 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f55fdd640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3f480cd3e0 tx=0x7f3f480058d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 -- 172.21.15.38:0/1896402683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 msgr2=0x55efb7af4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f3f40005c40 tx=0x7f3f400058d0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 -- 172.21.15.38:0/1896402683 shutdown_connections 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7afd520 0x55efb7aff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4940 0x55efb7afcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/1896402683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4020 0x55efb7af4400 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 -- 172.21.15.38:0/1896402683 >> 172.21.15.38:0/1896402683 conn(0x55efb7aee750 msgr2=0x55efb7aeeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 -- 172.21.15.38:0/1896402683 shutdown_connections 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.781+0000 7f3f5a62e740 1 -- 172.21.15.38:0/1896402683 wait complete. 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 -- start start 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 Processor -- start 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7afa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55efb7b002b0 con 0x55efb7af4940 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42494/0 (socket says 172.21.15.38:42494) 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f54fdb640 1 -- 172.21.15.38:0/4261024958 learned_addr learned my addr 172.21.15.38:0/4261024958 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.782+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55efb7a13f10 con 0x55efb7afd520 2024-04-22T00:06:45.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55efb7b00430 con 0x55efb7af4020 2024-04-22T00:06:45.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f5a62e740 10 client.0 _ll_register_callbacks cb 0x55efb776bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 -- 172.21.15.38:0/4261024958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 msgr2=0x55efb7afa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 -- 172.21.15.38:0/4261024958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 msgr2=0x55efb7afa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7afa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb79e5c80 con 0x55efb7af4020 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f400047c0 tx=0x7f3f400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e7f3c866cf2b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.783+0000 7f3f38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40049040 con 0x55efb7af4020 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3f40003820 con 0x55efb7af4020 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f40005530 con 0x55efb7af4020 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3f40003a20 con 0x55efb7af4020 2024-04-22T00:06:45.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f5a62e740 10 client.26521 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.784+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55efb776c3f0 con 0x55efb7af4020 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f40005530 con 0x55efb7af4020 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3f4004f050 con 0x55efb7af4020 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 10 client.26521 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 10 client.26521 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 20 client.26521 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b1e410 0x55efb7b20800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 4 client.26521 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:45.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.786+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efb7a65a70 con 0x55efb7b1e410 2024-04-22T00:06:45.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:45 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2794448833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.787+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b1e410 0x55efb7b20800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.788+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b1e410 0x55efb7b20800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea873de6ffcaa81 server_cookie=ec25a24084ebac82 in_seq=0 out_seq=0 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.788+0000 7f3f38ff9640 10 client.26521 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f38ff9640 1 -- 172.21.15.38:0/4261024958 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efb7a65a70 con 0x55efb7b1e410 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f38ff9640 10 client.26521 handle_command_reply: tid=0 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 client.26521 shutdown 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 msgr2=0x55efb7afceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f400047c0 tx=0x7f3f400046c0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 shutdown_connections 2024-04-22T00:06:45.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b1e410 0x55efb7b20800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7afa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4261024958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7afceb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.789+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 >> 172.21.15.38:0/4261024958 conn(0x55efb7aee750 msgr2=0x55efb7af04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.790+0000 7f3f38ff9640 0 client.26521 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.790+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 shutdown_connections 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.791+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4261024958 wait complete. 2024-04-22T00:06:45.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.791+0000 7f3f5a62e740 20 client.26521 trim_cache size 0 max 16384 2024-04-22T00:06:45.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.817+0000 7f3f5a62e740 1 Processor -- start 2024-04-22T00:06:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 -- start start 2024-04-22T00:06:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7af93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a62b0 con 0x55efb7af4940 2024-04-22T00:06:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a7df0 con 0x55efb7afd520 2024-04-22T00:06:45.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.818+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a6980 con 0x55efb7af4020 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.819+0000 7f3f557dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.819+0000 7f3f557dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60962/0 (socket says 172.21.15.38:60962) 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.819+0000 7f3f557dc640 1 -- 172.21.15.38:0/4206940621 learned_addr learned my addr 172.21.15.38:0/4206940621 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.819+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7af93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.819+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 -- 172.21.15.38:0/4206940621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 msgr2=0x55efb7af93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7af93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 -- 172.21.15.38:0/4206940621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 msgr2=0x55efb7afc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 -- 172.21.15.38:0/4206940621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb78367d0 con 0x55efb7af4940 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f557dc640 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x55efb7af3af0 tx=0x7f3f4c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b60e81860dc303 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f3affd640 1 -- 172.21.15.38:0/4206940621 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c021070 con 0x55efb7af4940 2024-04-22T00:06:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f3affd640 1 -- 172.21.15.38:0/4206940621 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c01c8a0 con 0x55efb7af4940 2024-04-22T00:06:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.820+0000 7f3f3affd640 1 -- 172.21.15.38:0/4206940621 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c003b20 con 0x55efb7af4940 2024-04-22T00:06:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.821+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4206940621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 msgr2=0x55efb7b19e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.821+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x55efb7af3af0 tx=0x7f3f4c010800 comp rx=0 tx=0).stop 2024-04-22T00:06:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4206940621 shutdown_connections 2024-04-22T00:06:45.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7afd520 0x55efb7afc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7af4940 0x55efb7b19e80 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/4206940621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4020 0x55efb7af93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4206940621 >> 172.21.15.38:0/4206940621 conn(0x55efb7aee750 msgr2=0x55efb7b19770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4206940621 shutdown_connections 2024-04-22T00:06:45.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.822+0000 7f3f5a62e740 1 -- 172.21.15.38:0/4206940621 wait complete. 2024-04-22T00:06:45.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.823+0000 7f3f5a62e740 1 -- start start 2024-04-22T00:06:45.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.823+0000 7f3f5a62e740 1 Processor -- start 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.823+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4940 0x55efb7b20ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7afd520 0x55efb7b869f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a7720 con 0x55efb7afd520 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3f501a7050 con 0x55efb7af4020 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50072df0 con 0x55efb7af4940 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f5a62e740 10 client.0 _ll_register_callbacks cb 0x7f3f4c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55458/0 (socket says 172.21.15.38:55458) 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 -- 172.21.15.38:0/2832132149 learned_addr learned my addr 172.21.15.38:0/2832132149 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7afd520 0x55efb7b869f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f557dc640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4940 0x55efb7b20ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 -- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4940 msgr2=0x55efb7b20ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4940 0x55efb7b20ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 -- 172.21.15.38:0/2832132149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7afd520 msgr2=0x55efb7b869f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7afd520 0x55efb7b869f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.824+0000 7f3f54fdb640 1 -- 172.21.15.38:0/2832132149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb7b86f30 con 0x55efb7af4020 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.825+0000 7f3f54fdb640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f400096c0 tx=0x7f3f400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6fe34d3fa868a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.825+0000 7f3f38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.825+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40060030 con 0x55efb7af4020 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.825+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3f40003bc0 con 0x55efb7af4020 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.825+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4c01c8a0 con 0x55efb7af4020 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3f40042440 con 0x55efb7af4020 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 10 client.26657 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:45.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55efb7b88200 con 0x55efb7af4020 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c01c8a0 con 0x55efb7af4020 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3f4004f040 con 0x55efb7af4020 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 10 client.26657 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 10 client.26657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 20 client.26657 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.826+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b887b0 0x55efb7b8abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.827+0000 7f3f5a62e740 4 client.26657 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.827+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55efb7a5fdc0 con 0x55efb7b887b0 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.827+0000 7f3f557dc640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b887b0 0x55efb7b8abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.827+0000 7f3f38ff9640 10 client.26657 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:45.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.827+0000 7f3f557dc640 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b887b0 0x55efb7b8abd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73fe4b69281a4693 server_cookie=375228b68895e44c in_seq=0 out_seq=0 2024-04-22T00:06:45.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f38ff9640 1 -- 172.21.15.38:0/2832132149 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55efb7a5fdc0 con 0x55efb7b887b0 2024-04-22T00:06:45.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f38ff9640 10 client.26657 handle_command_reply: tid=0 2024-04-22T00:06:45.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 client.26657 shutdown 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 msgr2=0x55efb7b20590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f400096c0 tx=0x7f3f400046c0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 shutdown_connections 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55efb7b887b0 0x55efb7b8abd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55efb7afd520 0x55efb7b869f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55efb7af4940 0x55efb7b20ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f38ff9640 0 client.26657 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 --2- 172.21.15.38:0/2832132149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb7af4020 0x55efb7b20590 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 >> 172.21.15.38:0/2832132149 conn(0x55efb7aee750 msgr2=0x55efb7aecfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 shutdown_connections 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 1 -- 172.21.15.38:0/2832132149 wait complete. 2024-04-22T00:06:45.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.828+0000 7f3f5a62e740 20 client.26657 trim_cache size 0 max 16384 2024-04-22T00:06:45.826 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:45.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 msgr2=0x7f3f3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3f480cd3e0 tx=0x7f3f480058d0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 msgr2=0x7f3f5019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800eb10 tx=0x7f3f4800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 shutdown_connections 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f501066b0 0x7f3f5019c8d0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100fd0 0x7f3f50198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 --2- 172.21.15.38:0/4091576072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f50100610 0x7f3f50198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 >> 172.21.15.38:0/4091576072 conn(0x7f3f500769c0 msgr2=0x7f3f500feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 shutdown_connections 2024-04-22T00:06:45.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:45.830+0000 7f3f57a40640 1 -- 172.21.15.38:0/4091576072 wait complete. 2024-04-22T00:06:45.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:45 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2794448833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:06:45.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:06:46.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.065+0000 7fc53d4fa640 1 -- 172.21.15.38:0/1393254472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5381010b0 msgr2=0x7fc538105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.065+0000 7fc5367fc640 1 -- 172.21.15.38:0/1393254472 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520039070 con 0x7fc5381010b0 2024-04-22T00:06:46.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.065+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/1393254472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5381010b0 0x7fc538105570 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc520009860 tx=0x7fc52002bc10 comp rx=0 tx=0).stop 2024-04-22T00:06:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.068+0000 7fc53d4fa640 1 -- 172.21.15.38:0/1393254472 shutdown_connections 2024-04-22T00:06:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.068+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/1393254472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc538105b90 0x7fc538106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.068+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/1393254472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5381010b0 0x7fc538105570 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.068+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/1393254472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc538100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.068+0000 7fc53d4fa640 1 -- 172.21.15.38:0/1393254472 >> 172.21.15.38:0/1393254472 conn(0x7fc5380769c0 msgr2=0x7fc538076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.071+0000 7fc53d4fa640 1 -- 172.21.15.38:0/1393254472 shutdown_connections 2024-04-22T00:06:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.071+0000 7fc53d4fa640 1 -- 172.21.15.38:0/1393254472 wait complete. 2024-04-22T00:06:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.072+0000 7fc53d4fa640 1 Processor -- start 2024-04-22T00:06:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.072+0000 7fc53d4fa640 1 -- start start 2024-04-22T00:06:46.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 0x7fc53810b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538105b90 0x7fc538072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc538107150 con 0x7fc538100790 2024-04-22T00:06:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc538106fd0 con 0x7fc538105b90 2024-04-22T00:06:46.069 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc53d4fa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381072d0 con 0x7fc5381010b0 2024-04-22T00:06:46.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60990/0 (socket says 172.21.15.38:60990) 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.073+0000 7fc536ffd640 1 -- 172.21.15.38:0/3086372710 learned_addr learned my addr 172.21.15.38:0/3086372710 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc52ffff640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 0x7fc53810b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 -- 172.21.15.38:0/3086372710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 msgr2=0x7fc53810b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 0x7fc53810b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 -- 172.21.15.38:0/3086372710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538105b90 msgr2=0x7fc538072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538105b90 0x7fc538072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 -- 172.21.15.38:0/3086372710 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc538072910 con 0x7fc538100790 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc52ffff640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 0x7fc53810b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.074+0000 7fc536ffd640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc52800d950 tx=0x7fc52800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.075+0000 7fc534ff9640 1 -- 172.21.15.38:0/3086372710 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528017070 con 0x7fc538100790 2024-04-22T00:06:46.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.075+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc538072c00 con 0x7fc538100790 2024-04-22T00:06:46.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.075+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc538112e70 con 0x7fc538100790 2024-04-22T00:06:46.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.075+0000 7fc534ff9640 1 -- 172.21.15.38:0/3086372710 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc528009c30 con 0x7fc538100790 2024-04-22T00:06:46.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.075+0000 7fc534ff9640 1 -- 172.21.15.38:0/3086372710 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52800f690 con 0x7fc538100790 2024-04-22T00:06:46.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.082+0000 7fc534ff9640 1 -- 172.21.15.38:0/3086372710 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc52800f830 con 0x7fc538100790 2024-04-22T00:06:46.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.083+0000 7fc5401b7740 1 Processor -- start 2024-04-22T00:06:46.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.083+0000 7fc534ff9640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 0x7fc5240a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.084+0000 7fc534ff9640 1 -- 172.21.15.38:0/3086372710 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc5280cf2d0 con 0x7fc538100790 2024-04-22T00:06:46.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.085+0000 7fc5377fe640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 0x7fc5240a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.086+0000 7fc5401b7740 1 -- start start 2024-04-22T00:06:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5c8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f75a52c530 con 0x55f75a5c8020 2024-04-22T00:06:46.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f75a4d4830 con 0x55f75a5c8940 2024-04-22T00:06:46.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.087+0000 7fc5401b7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f75a5284f0 con 0x55f75a5d1520 2024-04-22T00:06:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.088+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42550/0 (socket says 172.21.15.38:42550) 2024-04-22T00:06:46.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc5377fe640 1 -- 172.21.15.38:0/4008346521 learned_addr learned my addr 172.21.15.38:0/4008346521 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc52ffff640 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 -- 172.21.15.38:0/4008346521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 msgr2=0x55f75a5d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 -- 172.21.15.38:0/4008346521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 msgr2=0x55f75a5d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc5377fe640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 0x7fc5240a53c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc530005fd0 tx=0x7fc530012810 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.089+0000 7fc536ffd640 1 -- 172.21.15.38:0/4008346521 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f75a4f82c0 con 0x55f75a5c8020 2024-04-22T00:06:46.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.090+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5c8400 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc5280cc680 tx=0x7fc52809b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34031cdca0abe5e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.091+0000 7fc52d7fa640 1 -- 172.21.15.38:0/4008346521 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528020040 con 0x55f75a5c8020 2024-04-22T00:06:46.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.092+0000 7fc52d7fa640 1 -- 172.21.15.38:0/4008346521 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc52809fc30 con 0x55f75a5c8020 2024-04-22T00:06:46.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.093+0000 7fc5401b7740 1 -- 172.21.15.38:0/4008346521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 msgr2=0x55f75a5c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.093+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5c8400 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc5280cc680 tx=0x7fc52809b000 comp rx=0 tx=0).stop 2024-04-22T00:06:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.095+0000 7fc5401b7740 1 -- 172.21.15.38:0/4008346521 shutdown_connections 2024-04-22T00:06:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.095+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.095+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.095+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4008346521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5c8400 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.095+0000 7fc5401b7740 1 -- 172.21.15.38:0/4008346521 >> 172.21.15.38:0/4008346521 conn(0x55f75a5c2750 msgr2=0x55f75a5c2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.097+0000 7fc5401b7740 1 -- 172.21.15.38:0/4008346521 shutdown_connections 2024-04-22T00:06:46.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.100+0000 7fc5401b7740 1 -- 172.21.15.38:0/4008346521 wait complete. 2024-04-22T00:06:46.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 -- start start 2024-04-22T00:06:46.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 Processor -- start 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 0x55f75a5d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 0x55f75a5ce650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f75a5d42a0 con 0x55f75a5c8940 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f75a4e7f10 con 0x55f75a5d1520 2024-04-22T00:06:46.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f75a5d4420 con 0x55f75a5c8020 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.101+0000 7fc5401b7740 10 client.0 _ll_register_callbacks cb 0x55f75a23fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 0x55f75a5d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 0x55f75a5d0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42552/0 (socket says 172.21.15.38:42552) 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32778/0 (socket says 172.21.15.38:32778) 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc536ffd640 1 -- 172.21.15.38:0/2116447843 learned_addr learned my addr 172.21.15.38:0/2116447843 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc5377fe640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 0x55f75a5ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 -- 172.21.15.38:0/2116447843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 msgr2=0x55f75a5d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 0x55f75a5d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 -- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 msgr2=0x55f75a5ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 0x55f75a5ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.102+0000 7fc52ffff640 1 -- 172.21.15.38:0/2116447843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f75a4b9c80 con 0x55f75a5c8940 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.103+0000 7fc52ffff640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc538077180 tx=0x7fc520004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560073511725273a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.103+0000 7fc51e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.103+0000 7fc5377fe640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 0x55f75a5ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.103+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520039030 con 0x55f75a5c8940 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.104+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc52809fc30 con 0x55f75a5c8940 2024-04-22T00:06:46.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.104+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc520034640 con 0x55f75a5c8940 2024-04-22T00:06:46.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.106+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc52004e050 con 0x55f75a5c8940 2024-04-22T00:06:46.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.107+0000 7fc5401b7740 10 client.17277 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:46.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.107+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f75a2403f0 con 0x55f75a5c8940 2024-04-22T00:06:46.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc52809fc30 con 0x55f75a5c8940 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc520004c30 con 0x55f75a5c8940 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 10 client.17277 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 10 client.17277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 20 client.17277 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a603910 0x55f75a605d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 4 client.17277 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:46.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.111+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f75a539a70 con 0x55f75a603910 2024-04-22T00:06:46.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.113+0000 7fc536ffd640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a603910 0x55f75a605d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.117+0000 7fc536ffd640 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a603910 0x55f75a605d30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb97a362d273b13b server_cookie=d62bf15fe53b51ed in_seq=0 out_seq=0 2024-04-22T00:06:46.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.118+0000 7fc51e7fc640 10 client.17277 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:46.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.119+0000 7fc51e7fc640 1 -- 172.21.15.38:0/2116447843 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f75a539a70 con 0x55f75a603910 2024-04-22T00:06:46.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.119+0000 7fc51e7fc640 10 client.17277 handle_command_reply: tid=0 2024-04-22T00:06:46.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.119+0000 7fc5401b7740 1 client.17277 shutdown 2024-04-22T00:06:46.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.125+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 msgr2=0x55f75a5ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.125+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc538077180 tx=0x7fc520004270 comp rx=0 tx=0).stop 2024-04-22T00:06:46.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 shutdown_connections 2024-04-22T00:06:46.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5d1520 0x55f75a5ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a603910 0x55f75a605d30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5ce110 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 --2- 172.21.15.38:0/2116447843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5c8020 0x55f75a5d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.128+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 >> 172.21.15.38:0/2116447843 conn(0x55f75a5c2750 msgr2=0x55f75a5c44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.129+0000 7fc51e7fc640 0 client.17277 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:46.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.129+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 shutdown_connections 2024-04-22T00:06:46.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.130+0000 7fc5401b7740 1 -- 172.21.15.38:0/2116447843 wait complete. 2024-04-22T00:06:46.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.130+0000 7fc5401b7740 20 client.17277 trim_cache size 0 max 16384 2024-04-22T00:06:46.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.157+0000 7fc5401b7740 1 Processor -- start 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.157+0000 7fc5401b7740 1 -- start start 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8020 0x55f75a5c9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5c16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc51800fe20 con 0x55f75a5c8940 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc518011960 con 0x55f75a5c8020 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5401b7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc5180104f0 con 0x55f75a5d1520 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42560/0 (socket says 172.21.15.38:42560) 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 -- 172.21.15.38:0/112060286 learned_addr learned my addr 172.21.15.38:0/112060286 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc536ffd640 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8020 0x55f75a5c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc52ffff640 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5c16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 -- 172.21.15.38:0/112060286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8020 msgr2=0x55f75a5c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8020 0x55f75a5c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 -- 172.21.15.38:0/112060286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 msgr2=0x55f75a5c16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5c16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.158+0000 7fc5377fe640 1 -- 172.21.15.38:0/112060286 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f75a5c8770 con 0x55f75a5d1520 2024-04-22T00:06:46.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.159+0000 7fc5377fe640 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc530012df0 tx=0x7fc530019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fadd7b425a1c7c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.159+0000 7fc52cff9640 1 -- 172.21.15.38:0/112060286 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53002b070 con 0x55f75a5d1520 2024-04-22T00:06:46.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.159+0000 7fc52cff9640 1 -- 172.21.15.38:0/112060286 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc530026920 con 0x55f75a5d1520 2024-04-22T00:06:46.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.159+0000 7fc52cff9640 1 -- 172.21.15.38:0/112060286 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc530010a60 con 0x55f75a5d1520 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 -- 172.21.15.38:0/112060286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 msgr2=0x55f75a5d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc530012df0 tx=0x7fc530019000 comp rx=0 tx=0).stop 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 -- 172.21.15.38:0/112060286 shutdown_connections 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d0840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8940 0x55f75a5c16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 --2- 172.21.15.38:0/112060286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8020 0x55f75a5c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 -- 172.21.15.38:0/112060286 >> 172.21.15.38:0/112060286 conn(0x55f75a5c2750 msgr2=0x55f75a5c0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 -- 172.21.15.38:0/112060286 shutdown_connections 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.160+0000 7fc5401b7740 1 -- 172.21.15.38:0/112060286 wait complete. 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.161+0000 7fc5401b7740 1 -- start start 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.161+0000 7fc5401b7740 1 Processor -- start 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.161+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5d9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.161+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.161+0000 7fc5401b7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5401b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc518011290 con 0x55f75a5c8020 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42566/0 (socket says 172.21.15.38:42566) 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 -- 172.21.15.38:0/4192412822 learned_addr learned my addr 172.21.15.38:0/4192412822 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:46.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5d9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc518010bc0 con 0x55f75a5c8940 2024-04-22T00:06:46.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc51800de00 con 0x55f75a5d1520 2024-04-22T00:06:46.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc52ffff640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5401b7740 10 client.0 _ll_register_callbacks cb 0x7fc5300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 -- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 msgr2=0x55f75a5d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 -- 172.21.15.38:0/4192412822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 msgr2=0x55f75a5d9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5d9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f75a30a7d0 con 0x55f75a5d1520 2024-04-22T00:06:46.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc52ffff640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:46.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc530026920 con 0x55f75a5d1520 2024-04-22T00:06:46.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc5377fe640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5300266f0 tx=0x7fc530026720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2cb8c1317879f34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:46.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.162+0000 7fc51e7fc640 10 client.26539 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:46.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc51e7fc640 1 -- 172.21.15.38:0/4192412822 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53002b070 con 0x55f75a5d1520 2024-04-22T00:06:46.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc51e7fc640 1 -- 172.21.15.38:0/4192412822 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc5300106e0 con 0x55f75a5d1520 2024-04-22T00:06:46.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc51e7fc640 1 -- 172.21.15.38:0/4192412822 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc530010940 con 0x55f75a5d1520 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 10 client.26539 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55f75a669c70 con 0x55f75a5d1520 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc51e7fc640 1 -- 172.21.15.38:0/4192412822 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc530010ae0 con 0x55f75a5d1520 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 10 client.26539 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 10 client.26539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:46.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 20 client.26539 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a66a200 0x55f75a66c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 4 client.26539 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:06:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f75a533dc0 con 0x55f75a66a200 2024-04-22T00:06:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.163+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a66a200 0x55f75a66c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.165+0000 7fc536ffd640 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a66a200 0x55f75a66c620 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d957677e8df3ed server_cookie=2276c3b23c794e1a in_seq=0 out_seq=0 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.165+0000 7fc51e7fc640 10 client.26539 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.165+0000 7fc51e7fc640 1 -- 172.21.15.38:0/4192412822 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f75a533dc0 con 0x55f75a66a200 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.165+0000 7fc51e7fc640 10 client.26539 handle_command_reply: tid=0 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.165+0000 7fc5401b7740 1 client.26539 shutdown 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 msgr2=0x55f75a5d5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5300266f0 tx=0x7fc530026720 comp rx=0 tx=0).stop 2024-04-22T00:06:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 shutdown_connections 2024-04-22T00:06:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f75a5d1520 0x55f75a5d5540 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f75a5c8940 0x55f75a5d4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55f75a66a200 0x55f75a66c620 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 --2- 172.21.15.38:0/4192412822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f75a5c8020 0x55f75a5d9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 >> 172.21.15.38:0/4192412822 conn(0x55f75a5c2750 msgr2=0x55f75a5c9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.166+0000 7fc51e7fc640 0 client.26539 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.167+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 shutdown_connections 2024-04-22T00:06:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.167+0000 7fc5401b7740 1 -- 172.21.15.38:0/4192412822 wait complete. 2024-04-22T00:06:46.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.169+0000 7fc5401b7740 20 client.26539 trim_cache size 0 max 16384 2024-04-22T00:06:46.173 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:06:46.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 msgr2=0x7fc5240a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 0x7fc5240a53c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc530005fd0 tx=0x7fc530012810 comp rx=0 tx=0).stop 2024-04-22T00:06:46.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 msgr2=0x7fc53810ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc52800d950 tx=0x7fc52800de10 comp rx=0 tx=0).stop 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 shutdown_connections 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc5240a2f00 0x7fc5240a53c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc538105b90 0x7fc538072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5381010b0 0x7fc53810b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 --2- 172.21.15.38:0/3086372710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc538100790 0x7fc53810ad70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 >> 172.21.15.38:0/3086372710 conn(0x7fc5380769c0 msgr2=0x7fc538103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 shutdown_connections 2024-04-22T00:06:46.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:46.170+0000 7fc53d4fa640 1 -- 172.21.15.38:0/3086372710 wait complete. 2024-04-22T00:06:46.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:46 smithi203 ceph-mon[31431]: pgmap v143: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-04-22T00:06:46.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:46 smithi111 ceph-mon[31543]: pgmap v143: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-04-22T00:06:46.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:46 smithi038 ceph-mon[27972]: pgmap v143: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 2.92k op/s 2024-04-22T00:06:47.182 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:06:47.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:06:47.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 -- 172.21.15.38:0/1453089892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100107730 msgr2=0x7fd100107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 --2- 172.21.15.38:0/1453089892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100107730 0x7fd100107b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009900 tx=0x7fd0f402bc60 comp rx=0 tx=0).stop 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 -- 172.21.15.38:0/1453089892 shutdown_connections 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 --2- 172.21.15.38:0/1453089892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10006a780 0x7fd10010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 --2- 172.21.15.38:0/1453089892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd100069dc0 0x7fd10006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 --2- 172.21.15.38:0/1453089892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100107730 0x7fd100107b10 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009900 tx=0x7fd0f402bc60 comp rx=0 tx=0).stop 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 -- 172.21.15.38:0/1453089892 >> 172.21.15.38:0/1453089892 conn(0x7fd1000776a0 msgr2=0x7fd100077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 -- 172.21.15.38:0/1453089892 shutdown_connections 2024-04-22T00:06:47.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.368+0000 7fd1057e5640 1 -- 172.21.15.38:0/1453089892 wait complete. 2024-04-22T00:06:47.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.369+0000 7fd1057e5640 1 Processor -- start 2024-04-22T00:06:47.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.369+0000 7fd1057e5640 1 -- start start 2024-04-22T00:06:47.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100069dc0 0x7fd1001160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd10006a780 0x7fd100116600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001022e0 con 0x7fd10006a780 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd100102160 con 0x7fd100069dc0 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd1057e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd100102460 con 0x7fd10011bdd0 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42582/0 (socket says 172.21.15.38:42582) 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/429790611 learned_addr learned my addr 172.21.15.38:0/429790611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.370+0000 7fd0feffd640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100069dc0 0x7fd1001160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/429790611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100069dc0 msgr2=0x7fd1001160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100069dc0 0x7fd1001160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd10006a780 msgr2=0x7fd100116600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd10006a780 0x7fd100116600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/429790611 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1001111d0 con 0x7fd10011bdd0 2024-04-22T00:06:47.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.371+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800b9a0 tx=0x7fd0f800be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.372+0000 7fd0e7fff640 1 -- 172.21.15.38:0/429790611 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8019070 con 0x7fd10011bdd0 2024-04-22T00:06:47.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.372+0000 7fd0e7fff640 1 -- 172.21.15.38:0/429790611 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0f800cba0 con 0x7fd10011bdd0 2024-04-22T00:06:47.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.372+0000 7fd0e7fff640 1 -- 172.21.15.38:0/429790611 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8007660 con 0x7fd10011bdd0 2024-04-22T00:06:47.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.374+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd100111460 con 0x7fd10011bdd0 2024-04-22T00:06:47.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.374+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd100111960 con 0x7fd10011bdd0 2024-04-22T00:06:47.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 Processor -- start 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 -- start start 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99861a40 0x562c99861ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c9986d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c997c5170 con 0x562c99862720 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c9976d7c0 con 0x562c99861a40 2024-04-22T00:06:47.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.378+0000 7fd1083d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c997c1130 con 0x562c9986adb0 2024-04-22T00:06:47.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.379+0000 7fd0fe7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c9986d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.379+0000 7fd0fe7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.379+0000 7fd0fe7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32818/0 (socket says 172.21.15.38:32818) 2024-04-22T00:06:47.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.379+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/705422913 learned_addr learned my addr 172.21.15.38:0/705422913 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.380+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/705422913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 msgr2=0x562c9986d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.380+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c9986d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.380+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/705422913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99861a40 msgr2=0x562c99861ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:06:47.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.380+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99861a40 0x562c99861ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.380+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/705422913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c9977ee60 con 0x562c99862720 2024-04-22T00:06:47.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.381+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0012910 tx=0x7fd0f0006960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b4e24e2e1abc24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.381+0000 7fd0e57fa640 1 -- 172.21.15.38:0/705422913 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0004230 con 0x562c99862720 2024-04-22T00:06:47.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.382+0000 7fd0e57fa640 1 -- 172.21.15.38:0/705422913 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0f001a070 con 0x562c99862720 2024-04-22T00:06:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.382+0000 7fd0e57fa640 1 -- 172.21.15.38:0/705422913 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0004c10 con 0x562c99862720 2024-04-22T00:06:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.382+0000 7fd0e7fff640 1 -- 172.21.15.38:0/429790611 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd0f8014440 con 0x7fd10011bdd0 2024-04-22T00:06:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.383+0000 7fd0e7fff640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 0x7fd0e80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.383+0000 7fd0e7fff640 1 -- 172.21.15.38:0/429790611 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd0f80ceb10 con 0x7fd10011bdd0 2024-04-22T00:06:47.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 -- 172.21.15.38:0/705422913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 msgr2=0x562c99861470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0012910 tx=0x7fd0f0006960 comp rx=0 tx=0).stop 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 -- 172.21.15.38:0/705422913 shutdown_connections 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c9986d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99861a40 0x562c99861ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 --2- 172.21.15.38:0/705422913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99861470 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.385+0000 7fd1083d5740 1 -- 172.21.15.38:0/705422913 >> 172.21.15.38:0/705422913 conn(0x562c998284f0 msgr2=0x562c9985c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.386+0000 7fd0feffd640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 0x7fd0e80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.387+0000 7fd0feffd640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 0x7fd0e80a53c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f4000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.387+0000 7fd1083d5740 1 -- 172.21.15.38:0/705422913 shutdown_connections 2024-04-22T00:06:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.387+0000 7fd1083d5740 1 -- 172.21.15.38:0/705422913 wait complete. 2024-04-22T00:06:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.388+0000 7fd1083d5740 1 -- start start 2024-04-22T00:06:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.388+0000 7fd1083d5740 1 Processor -- start 2024-04-22T00:06:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c99867dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c99868850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c9986dbc0 con 0x562c99862720 2024-04-22T00:06:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562c99780d40 con 0x562c9986adb0 2024-04-22T00:06:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c9986dd40 con 0x562c99861a40 2024-04-22T00:06:47.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd1083d5740 10 client.0 _ll_register_callbacks cb 0x562c997bc820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.389+0000 7fd0fe7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.390+0000 7fd0fe7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32834/0 (socket says 172.21.15.38:32834) 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.390+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/1656461205 learned_addr learned my addr 172.21.15.38:0/1656461205 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.390+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c99868850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.391+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/1656461205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 msgr2=0x562c99867dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.391+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c99867dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.391+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 msgr2=0x562c99868850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.391+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c99868850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.391+0000 7fd0fe7fc640 1 -- 172.21.15.38:0/1656461205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c995a17e0 con 0x562c99862720 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.393+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00063a0 tx=0x7fd0f00063d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de84b41a3a9fe6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.393+0000 7fd0e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00049d0 con 0x562c99862720 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd0f0004b70 con 0x562c99862720 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0f001a070 con 0x562c99862720 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd0f0012e90 con 0x562c99862720 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd1083d5740 10 client.17289 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.394+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562c99601f70 con 0x562c99862720 2024-04-22T00:06:47.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.395+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0f000a9a0 con 0x562c99862720 2024-04-22T00:06:47.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f0026400 con 0x562c99862720 2024-04-22T00:06:47.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 10 client.17289 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:47.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:47.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 20 client.17289 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:47.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c9987abc0 0x562c9987cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 4 client.17289 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:47.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.396+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd0f001a070 con 0x562c9987abc0 2024-04-22T00:06:47.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.402+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c9987abc0 0x562c9987cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.416+0000 7fd0e2ff5640 10 client.17289 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.416+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c9987abc0 0x562c9987cfb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9703864a77a704 server_cookie=a127add2636d6732 in_seq=0 out_seq=0 2024-04-22T00:06:47.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.418+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1656461205 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fd0f001a070 con 0x562c9987abc0 2024-04-22T00:06:47.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.419+0000 7fd0e2ff5640 10 client.17289 handle_command_reply: tid=0 2024-04-22T00:06:47.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.419+0000 7fd1083d5740 1 client.17289 shutdown 2024-04-22T00:06:47.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.419+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 msgr2=0x562c99868310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.420+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00063a0 tx=0x7fd0f00063d0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 shutdown_connections 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c99868850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c9987abc0 0x562c9987cfb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99868310 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1656461205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c99867dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 >> 172.21.15.38:0/1656461205 conn(0x562c998284f0 msgr2=0x562c9985e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.421+0000 7fd0e2ff5640 0 client.17289 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.425+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 shutdown_connections 2024-04-22T00:06:47.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.425+0000 7fd1083d5740 1 -- 172.21.15.38:0/1656461205 wait complete. 2024-04-22T00:06:47.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.426+0000 7fd1083d5740 20 client.17289 trim_cache size 0 max 16384 2024-04-22T00:06:47.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 Processor -- start 2024-04-22T00:06:47.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 -- start start 2024-04-22T00:06:47.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c9986a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99866d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001c65f0 con 0x562c99862720 2024-04-22T00:06:47.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001c8130 con 0x562c9986adb0 2024-04-22T00:06:47.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.468+0000 7fd1083d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001c6cc0 con 0x562c99861a40 2024-04-22T00:06:47.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.469+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.469+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55556/0 (socket says 172.21.15.38:55556) 2024-04-22T00:06:47.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.469+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/3370091182 learned_addr learned my addr 172.21.15.38:0/3370091182 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.473+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99866d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.473+0000 7fd0feffd640 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c9986a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.474+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/3370091182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 msgr2=0x562c9986a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.475+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c9986a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.475+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/3370091182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 msgr2=0x562c99866d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.475+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99866d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.475+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/3370091182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c99499170 con 0x562c9986adb0 2024-04-22T00:06:47.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.480+0000 7fd0ff7fe640 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0f809ab50 tx=0x7fd0f8004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d670b657ad0ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.483+0000 7fd0e4ff9640 1 -- 172.21.15.38:0/3370091182 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8099150 con 0x562c9986adb0 2024-04-22T00:06:47.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.483+0000 7fd0e4ff9640 1 -- 172.21.15.38:0/3370091182 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0f800cd50 con 0x562c9986adb0 2024-04-22T00:06:47.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.483+0000 7fd0e4ff9640 1 -- 172.21.15.38:0/3370091182 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80a3580 con 0x562c9986adb0 2024-04-22T00:06:47.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.485+0000 7fd1083d5740 1 -- 172.21.15.38:0/3370091182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 msgr2=0x562c9986c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.485+0000 7fd1083d5740 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0f809ab50 tx=0x7fd0f8004b50 comp rx=0 tx=0).stop 2024-04-22T00:06:47.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.486+0000 7fd1083d5740 1 -- 172.21.15.38:0/3370091182 shutdown_connections 2024-04-22T00:06:47.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.486+0000 7fd1083d5740 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c9986adb0 0x562c9986c1a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.486+0000 7fd1083d5740 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99862720 0x562c99866d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.486+0000 7fd1083d5740 1 --2- 172.21.15.38:0/3370091182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c99861a40 0x562c9986a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.486+0000 7fd1083d5740 1 -- 172.21.15.38:0/3370091182 >> 172.21.15.38:0/3370091182 conn(0x562c998284f0 msgr2=0x562c99876860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.490+0000 7fd1083d5740 1 -- 172.21.15.38:0/3370091182 shutdown_connections 2024-04-22T00:06:47.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.493+0000 7fd1083d5740 1 -- 172.21.15.38:0/3370091182 wait complete. 2024-04-22T00:06:47.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.493+0000 7fd1083d5740 1 -- start start 2024-04-22T00:06:47.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.494+0000 7fd1083d5740 1 Processor -- start 2024-04-22T00:06:47.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.494+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 0x562c99865180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.494+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99862720 0x562c998656c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd1083d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c998dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd1083d5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001c7a60 con 0x562c99861a40 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd1083d5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001c7390 con 0x562c99862720 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd1083d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd100102e60 con 0x562c9986adb0 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c998dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c998dfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42606/0 (socket says 172.21.15.38:42606) 2024-04-22T00:06:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd0ff7fe640 1 -- 172.21.15.38:0/1816120792 learned_addr learned my addr 172.21.15.38:0/1816120792 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.495+0000 7fd1083d5740 10 client.0 _ll_register_callbacks cb 0x7fd0f000aea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99862720 0x562c998656c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 0x562c99865180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 -- 172.21.15.38:0/1816120792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 msgr2=0x562c998dfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c998dfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 -- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99862720 msgr2=0x562c998656c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99862720 0x562c998656c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.496+0000 7fd0feffd640 1 -- 172.21.15.38:0/1816120792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c998e02d0 con 0x562c99861a40 2024-04-22T00:06:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.499+0000 7fd0feffd640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 0x562c99865180 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4008000 tx=0x7fd0f402bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f2a40a20189c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.499+0000 7fd0e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.500+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4051d50 con 0x562c99861a40 2024-04-22T00:06:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.500+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd0f402c8e0 con 0x562c99861a40 2024-04-22T00:06:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.500+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0f800cd50 con 0x562c99861a40 2024-04-22T00:06:47.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.500+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd0f402cae0 con 0x562c99861a40 2024-04-22T00:06:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.501+0000 7fd1083d5740 10 client.17295 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.501+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562c998e1550 con 0x562c99861a40 2024-04-22T00:06:47.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.504+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0f800cd50 con 0x562c99861a40 2024-04-22T00:06:47.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.504+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f40047d0 con 0x562c99861a40 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 10 client.17295 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 20 client.17295 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c998e1bd0 0x562c998e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 4 client.17295 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:06:47.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.507+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562c99799450 con 0x562c998e1bd0 2024-04-22T00:06:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.508+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c998e1bd0 0x562c998e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.508+0000 7fd0fe7fc640 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c998e1bd0 0x562c998e3ff0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d216b9b05bad1ee server_cookie=6caa3c1785f71d00 in_seq=0 out_seq=0 2024-04-22T00:06:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.508+0000 7fd0e2ff5640 10 client.17295 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd0e2ff5640 1 -- 172.21.15.38:0/1816120792 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562c99799450 con 0x562c998e1bd0 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd0e2ff5640 10 client.17295 handle_command_reply: tid=0 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 client.17295 shutdown 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 msgr2=0x562c99865180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 0x562c99865180 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4008000 tx=0x7fd0f402bc60 comp rx=0 tx=0).stop 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 shutdown_connections 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c9986adb0 0x562c998dfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562c99862720 0x562c998656c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562c998e1bd0 0x562c998e3ff0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 --2- 172.21.15.38:0/1816120792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c99861a40 0x562c99865180 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 >> 172.21.15.38:0/1816120792 conn(0x562c998284f0 msgr2=0x562c99867d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.510+0000 7fd0e2ff5640 0 client.17295 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.511+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 shutdown_connections 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.511+0000 7fd1083d5740 1 -- 172.21.15.38:0/1816120792 wait complete. 2024-04-22T00:06:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.511+0000 7fd1083d5740 20 client.17295 trim_cache size 0 max 16384 2024-04-22T00:06:47.507 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:06:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "aa8cf65f-5be7-4432-9b58-46fb65ce28fc", 2024-04-22T00:06:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:06:47.507 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 msgr2=0x7fd0e80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 0x7fd0e80a53c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009850 tx=0x7fd0f4000bc0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 msgr2=0x7fd10011c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800b9a0 tx=0x7fd0f800be60 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 shutdown_connections 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0e80a2f00 0x7fd0e80a53c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd10011bdd0 0x7fd10011c280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd10006a780 0x7fd100116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 --2- 172.21.15.38:0/429790611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd100069dc0 0x7fd1001160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 >> 172.21.15.38:0/429790611 conn(0x7fd1000776a0 msgr2=0x7fd1000fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 shutdown_connections 2024-04-22T00:06:47.509 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.512+0000 7fd1057e5640 1 -- 172.21.15.38:0/429790611 wait complete. 2024-04-22T00:06:47.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:06:47.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 -- 172.21.15.38:0/2834979679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d20107720 msgr2=0x7f2d20107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/2834979679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d20107720 0x7f2d20107b00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009930 tx=0x7f2d1002bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 -- 172.21.15.38:0/2834979679 shutdown_connections 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/2834979679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d201020d0 0x7f2d2010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/2834979679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20101710 0x7f2d20101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/2834979679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d20107720 0x7f2d20107b00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 -- 172.21.15.38:0/2834979679 >> 172.21.15.38:0/2834979679 conn(0x7f2d200fd180 msgr2=0x7f2d200ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 -- 172.21.15.38:0/2834979679 shutdown_connections 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.694+0000 7f2d25fbf640 1 -- 172.21.15.38:0/2834979679 wait complete. 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 Processor -- start 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 -- start start 2024-04-22T00:06:47.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 0x7f2d20073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 0x7f2d200744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d20111b30 con 0x7f2d201020d0 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201119b0 con 0x7f2d20107720 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d25fbf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d20111cb0 con 0x7f2d20101710 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32888/0 (socket says 172.21.15.38:32888) 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 -- 172.21.15.38:0/1890500385 learned_addr learned my addr 172.21.15.38:0/1890500385 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 0x7f2d20073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 0x7f2d200744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 -- 172.21.15.38:0/1890500385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 msgr2=0x7f2d20073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 0x7f2d20073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 -- 172.21.15.38:0/1890500385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 msgr2=0x7f2d200744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 0x7f2d200744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1effd640 1 -- 172.21.15.38:0/1890500385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d201a6510 con 0x7f2d201020d0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 0x7f2d20073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.695+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 0x7f2d200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d1effd640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400d950 tx=0x7f2d1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d1cff9640 1 -- 172.21.15.38:0/1890500385 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14017030 con 0x7f2d201020d0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d1cff9640 1 -- 172.21.15.38:0/1890500385 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d14004020 con 0x7f2d201020d0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d201a6800 con 0x7f2d201020d0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d1cff9640 1 -- 172.21.15.38:0/1890500385 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d140047a0 con 0x7f2d201020d0 2024-04-22T00:06:47.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.696+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d201a6d40 con 0x7f2d201020d0 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.698+0000 7f2d1cff9640 1 -- 172.21.15.38:0/1890500385 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2d140041d0 con 0x7f2d201020d0 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.698+0000 7f2d28ba2740 1 Processor -- start 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.698+0000 7f2d28ba2740 1 -- start start 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd831e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555dd8277560 con 0x555dd8312f30 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555dd821f730 con 0x555dd831c430 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555dd8273520 con 0x555dd8313850 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1cff9640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 0x7f2d080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32890/0 (socket says 172.21.15.38:32890) 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd831e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 -- 172.21.15.38:0/2051091414 learned_addr learned my addr 172.21.15.38:0/2051091414 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 -- 172.21.15.38:0/2051091414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 msgr2=0x555dd831bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 -- 172.21.15.38:0/2051091414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 msgr2=0x555dd831e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd831e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 -- 172.21.15.38:0/2051091414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd80557d0 con 0x555dd8312f30 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 0x7f2d080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1cff9640 1 -- 172.21.15.38:0/1890500385 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2d140cf1d0 con 0x7f2d201020d0 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.699+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2d18012440 tx=0x7f2d18012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f6dff72bfadce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d05ffb640 1 -- 172.21.15.38:0/2051091414 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1801d910 con 0x555dd8312f30 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d05ffb640 1 -- 172.21.15.38:0/2051091414 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1801c440 con 0x555dd8312f30 2024-04-22T00:06:47.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d05ffb640 1 -- 172.21.15.38:0/2051091414 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1801b630 con 0x555dd8312f30 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 0x7f2d080a5240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d18021000 tx=0x7f2d1801e000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2051091414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 msgr2=0x555dd8313310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f2d18012440 tx=0x7f2d18012900 comp rx=0 tx=0).stop 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2051091414 shutdown_connections 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd831e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2051091414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd8313310 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2051091414 >> 172.21.15.38:0/2051091414 conn(0x555dd830d660 msgr2=0x555dd830da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2051091414 shutdown_connections 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.700+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2051091414 wait complete. 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 -- start start 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 Processor -- start 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8319f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555dd831f220 con 0x555dd8313850 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555dd8232e00 con 0x555dd831c430 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555dd831f3a0 con 0x555dd8312f30 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d28ba2740 10 client.0 _ll_register_callbacks cb 0x555dd826deb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32892/0 (socket says 172.21.15.38:32892) 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d1effd640 1 -- 172.21.15.38:0/2610750937 learned_addr learned my addr 172.21.15.38:0/2610750937 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.701+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8319f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 -- 172.21.15.38:0/2610750937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 msgr2=0x555dd83194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 -- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 msgr2=0x555dd8319f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8319f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 -- 172.21.15.38:0/2610750937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd7f8b3f0 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d1effd640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2d14000c00 tx=0x7f2d140051f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c018b5592bf44c42 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1409b3b0 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2d1409a800 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1801c440 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d1409f760 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 10 client.17307 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555dd7f4b0f0 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1801c440 con 0x555dd8313850 2024-04-22T00:06:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2d140b41d0 con 0x555dd8313850 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 10 client.17307 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 20 client.17307 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.702+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd832c830 0x555dd832ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.703+0000 7f2d28ba2740 4 client.17307 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.703+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555dd8284aa0 con 0x555dd832c830 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.703+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd832c830 0x555dd832ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.703+0000 7f2cfeffd640 10 client.17307 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.703+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd832c830 0x555dd832ec20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87dcc1bd3cf60e89 server_cookie=560053cb06fc027 in_seq=0 out_seq=0 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2cfeffd640 1 -- 172.21.15.38:0/2610750937 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555dd8284aa0 con 0x555dd832c830 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2cfeffd640 10 client.17307 handle_command_reply: tid=0 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 client.17307 shutdown 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 msgr2=0x555dd8319a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2d14000c00 tx=0x7f2d140051f0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 shutdown_connections 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8319f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd832c830 0x555dd832ec20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8313850 0x555dd8319a30 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/2610750937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 >> 172.21.15.38:0/2610750937 conn(0x555dd830d660 msgr2=0x555dd830f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2cfeffd640 0 client.17307 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 shutdown_connections 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 1 -- 172.21.15.38:0/2610750937 wait complete. 2024-04-22T00:06:47.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.704+0000 7f2d28ba2740 20 client.17307 trim_cache size 0 max 16384 2024-04-22T00:06:47.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 Processor -- start 2024-04-22T00:06:47.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 -- start start 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8318450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201b0d60 con 0x555dd8312f30 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201b28a0 con 0x555dd831c430 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201b1430 con 0x555dd8313850 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32900/0 (socket says 172.21.15.38:32900) 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 -- 172.21.15.38:0/1408865092 learned_addr learned my addr 172.21.15.38:0/1408865092 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8318450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1effd640 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 -- 172.21.15.38:0/1408865092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 msgr2=0x555dd831bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 -- 172.21.15.38:0/1408865092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 msgr2=0x555dd8318450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.722+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8318450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d1f7fe640 1 -- 172.21.15.38:0/1408865092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd832f280 con 0x555dd8312f30 2024-04-22T00:06:47.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2d10005c00 tx=0x7f2d10009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3c9c70399336ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d057fa640 1 -- 172.21.15.38:0/1408865092 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1002cd60 con 0x555dd8312f30 2024-04-22T00:06:47.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d057fa640 1 -- 172.21.15.38:0/1408865092 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1004a070 con 0x555dd8312f30 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d057fa640 1 -- 172.21.15.38:0/1408865092 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d100458f0 con 0x555dd8312f30 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 -- 172.21.15.38:0/1408865092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 msgr2=0x555dd83280e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2d10005c00 tx=0x7f2d10009680 comp rx=0 tx=0).stop 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 -- 172.21.15.38:0/1408865092 shutdown_connections 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd831c430 0x555dd8318450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8313850 0x555dd831bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/1408865092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd8312f30 0x555dd83280e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 -- 172.21.15.38:0/1408865092 >> 172.21.15.38:0/1408865092 conn(0x555dd830d660 msgr2=0x555dd830c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 -- 172.21.15.38:0/1408865092 shutdown_connections 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.723+0000 7f2d28ba2740 1 -- 172.21.15.38:0/1408865092 wait complete. 2024-04-22T00:06:47.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 -- start start 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 Processor -- start 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83245f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 0x555dd8324b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201b21d0 con 0x555dd831c430 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201b1b00 con 0x555dd8313850 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d201aed40 con 0x555dd8312f30 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d28ba2740 10 client.0 _ll_register_callbacks cb 0x7f2d14004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32908/0 (socket says 172.21.15.38:32908) 2024-04-22T00:06:47.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d1effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 0x555dd8324b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d1ffff640 1 -- 172.21.15.38:0/459436753 learned_addr learned my addr 172.21.15.38:0/459436753 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.724+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83245f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 -- 172.21.15.38:0/459436753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 msgr2=0x555dd83245f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83245f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 -- 172.21.15.38:0/459436753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 msgr2=0x555dd83215c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 -- 172.21.15.38:0/459436753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd8321c80 con 0x555dd8313850 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1ffff640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d1effd640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 0x555dd8324b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2d14097810 tx=0x7f2d1409aab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10036e9ae2979be2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1409b3b0 con 0x555dd8313850 2024-04-22T00:06:47.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1004a070 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f2d140a08f0 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d1409a270 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d28ba2740 10 client.26699 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555dd8393700 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.725+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1004a070 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2d140cc1c0 con 0x555dd8313850 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 10 client.26699 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 20 client.26699 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:47.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd8393ea0 0x555dd83962c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 4 client.26699 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555dd827edf0 con 0x555dd8393ea0 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd8393ea0 0x555dd83962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2cfeffd640 10 client.26699 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d1f7fe640 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd8393ea0 0x555dd83962c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c77f7863e58d63d server_cookie=3494272b02df0ca9 in_seq=0 out_seq=0 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2cfeffd640 1 -- 172.21.15.38:0/459436753 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x555dd827edf0 con 0x555dd8393ea0 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2cfeffd640 10 client.26699 handle_command_reply: tid=0 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.726+0000 7f2d28ba2740 1 client.26699 shutdown 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 msgr2=0x555dd8324b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 0x555dd8324b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2d14097810 tx=0x7f2d1409aab0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 shutdown_connections 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x555dd8393ea0 0x555dd83962c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555dd831c430 0x555dd83215c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555dd8313850 0x555dd8324b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 --2- 172.21.15.38:0/459436753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555dd8312f30 0x555dd83245f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2cfeffd640 0 client.26699 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:47.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 >> 172.21.15.38:0/459436753 conn(0x555dd830d660 msgr2=0x555dd830e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 shutdown_connections 2024-04-22T00:06:47.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 1 -- 172.21.15.38:0/459436753 wait complete. 2024-04-22T00:06:47.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d28ba2740 20 client.26699 trim_cache size 0 max 16384 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (227 inodes in the stack)", 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: "aa8cf65f-5be7-4432-9b58-46fb65ce28fc": { 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:06:47.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "aa8cf65f-5be7-4432-9b58-46fb65ce28fc", 2024-04-22T00:06:47.734 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:06:47.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:06:47.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:06:47.734 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:06:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 msgr2=0x7f2d080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 0x7f2d080a5240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d18021000 tx=0x7f2d1801e000 comp rx=0 tx=0).stop 2024-04-22T00:06:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 msgr2=0x7f2d20073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400d950 tx=0x7f2d1400de10 comp rx=0 tx=0).stop 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 shutdown_connections 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2d080a2d80 0x7f2d080a5240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d20107720 0x7f2d200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2d201020d0 0x7f2d20073f80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 --2- 172.21.15.38:0/1890500385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d20101710 0x7f2d20073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:47.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 >> 172.21.15.38:0/1890500385 conn(0x7f2d200fd180 msgr2=0x7f2d200fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:47.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 shutdown_connections 2024-04-22T00:06:47.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:06:47.727+0000 7f2d25fbf640 1 -- 172.21.15.38:0/1890500385 wait complete. 2024-04-22T00:06:47.739 INFO:tasks.cephfs.filesystem:scrub status for tag:aa8cf65f-5be7-4432-9b58-46fb65ce28fc - {'path': '/', 'tag': 'aa8cf65f-5be7-4432-9b58-46fb65ce28fc', 'options': 'recursive,force'} 2024-04-22T00:06:47.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:47 smithi038 ceph-mon[27972]: pgmap v144: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:48.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:47 smithi203 ceph-mon[31431]: pgmap v144: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:48.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:47 smithi111 ceph-mon[31543]: pgmap v144: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 14 MiB/s wr, 3.38k op/s 2024-04-22T00:06:48.387 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 -- 172.21.15.38:0/3888720306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620107750 msgr2=0x7f9620107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 --2- 172.21.15.38:0/3888720306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620107750 0x7f9620107b30 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f961c009930 tx=0x7f961c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 -- 172.21.15.38:0/3888720306 shutdown_connections 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 --2- 172.21.15.38:0/3888720306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620102100 0x7f962010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 --2- 172.21.15.38:0/3888720306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 0x7f9620101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 --2- 172.21.15.38:0/3888720306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620107750 0x7f9620107b30 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 -- 172.21.15.38:0/3888720306 >> 172.21.15.38:0/3888720306 conn(0x7f96200fd210 msgr2=0x7f96200ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 -- 172.21.15.38:0/3888720306 shutdown_connections 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.390+0000 7f96273b8640 1 -- 172.21.15.38:0/3888720306 wait complete. 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 Processor -- start 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 -- start start 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 0x7f9620197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.388 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620107750 0x7f962019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9620111b20 con 0x7f9620102100 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96201119a0 con 0x7f9620101740 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f96273b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9620111ca0 con 0x7f9620107750 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32932/0 (socket says 172.21.15.38:32932) 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 -- 172.21.15.38:0/3643540224 learned_addr learned my addr 172.21.15.38:0/3643540224 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9625154640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 0x7f9620197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9625955640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620107750 0x7f962019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 -- 172.21.15.38:0/3643540224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620107750 msgr2=0x7f962019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620107750 0x7f962019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 -- 172.21.15.38:0/3643540224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 msgr2=0x7f9620197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.389 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 0x7f9620197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.391+0000 7f9624953640 1 -- 172.21.15.38:0/3643540224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962019cf30 con 0x7f9620102100 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f9624953640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f961000c920 tx=0x7f961000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f96167fc640 1 -- 172.21.15.38:0/3643540224 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610007c70 con 0x7f9620102100 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f96167fc640 1 -- 172.21.15.38:0/3643540224 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9610007e10 con 0x7f9620102100 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962019d220 con 0x7f9620102100 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f96167fc640 1 -- 172.21.15.38:0/3643540224 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610018890 con 0x7f9620102100 2024-04-22T00:06:48.390 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.392+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96201a4ab0 con 0x7f9620102100 2024-04-22T00:06:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.395+0000 7f9629f95740 1 Processor -- start 2024-04-22T00:06:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.395+0000 7f96167fc640 1 -- 172.21.15.38:0/3643540224 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9610018a70 con 0x7f9620102100 2024-04-22T00:06:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.396+0000 7f9629f95740 1 -- start start 2024-04-22T00:06:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f96167fc640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 0x7f960c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.391 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c35649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c35720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5629c34c8b70 con 0x5629c3572620 2024-04-22T00:06:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5629c3471470 con 0x5629c3564ee0 2024-04-22T00:06:48.392 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9629f95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5629c34c4b30 con 0x5629c35645c0 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f96167fc640 1 -- 172.21.15.38:0/3643540224 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96100cf890 con 0x7f9620102100 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32934/0 (socket says 172.21.15.38:32934) 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 -- 172.21.15.38:0/3380741819 learned_addr learned my addr 172.21.15.38:0/3380741819 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9624953640 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c35720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 0x7f960c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 -- 172.21.15.38:0/3380741819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 msgr2=0x5629c35649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.397+0000 7f9625955640 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c35649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f9625955640 1 -- 172.21.15.38:0/3380741819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 msgr2=0x5629c35720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f9625955640 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c35720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.393 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f9625955640 1 -- 172.21.15.38:0/3380741819 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c31dd210 con 0x5629c3572620 2024-04-22T00:06:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f9625955640 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9618012a00 tx=0x7f9618012ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dab2c910b638cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f9625955640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 0x7f960c0a52e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f961801c000 tx=0x7f961801e000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f960b7fe640 1 -- 172.21.15.38:0/3380741819 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618004d20 con 0x5629c3572620 2024-04-22T00:06:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f960b7fe640 1 -- 172.21.15.38:0/3380741819 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96180123f0 con 0x5629c3572620 2024-04-22T00:06:48.394 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.398+0000 7f960b7fe640 1 -- 172.21.15.38:0/3380741819 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961801ec30 con 0x5629c3572620 2024-04-22T00:06:48.399 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 -- 172.21.15.38:0/3380741819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 msgr2=0x5629c3574a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9618012a00 tx=0x7f9618012ec0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 -- 172.21.15.38:0/3380741819 shutdown_connections 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c3574a10 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c35720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 --2- 172.21.15.38:0/3380741819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c35649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 -- 172.21.15.38:0/3380741819 >> 172.21.15.38:0/3380741819 conn(0x5629c355f440 msgr2=0x5629c355f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 -- 172.21.15.38:0/3380741819 shutdown_connections 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.399+0000 7f9629f95740 1 -- 172.21.15.38:0/3380741819 wait complete. 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 -- start start 2024-04-22T00:06:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 Processor -- start 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c356f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 0x5629c35700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5629c3575190 con 0x5629c35645c0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5629c3484a10 con 0x5629c3564ee0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5629c3575310 con 0x5629c3572620 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.400+0000 7f9629f95740 10 client.0 _ll_register_callbacks cb 0x5629c34bfb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:55684/0 (socket says 172.21.15.38:55684) 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 -- 172.21.15.38:0/2345179196 learned_addr learned my addr 172.21.15.38:0/2345179196 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9625955640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 0x5629c35700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9625154640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c356f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 -- 172.21.15.38:0/2345179196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 msgr2=0x5629c35700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.401 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 0x5629c35700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 -- 172.21.15.38:0/2345179196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 msgr2=0x5629c356f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c356f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 -- 172.21.15.38:0/2345179196 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c32a9620 con 0x5629c3564ee0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9625955640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 0x5629c35700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9624953640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9610007590 tx=0x7f961009b370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=599154d7427184c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9608ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.401+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96100be020 con 0x5629c3564ee0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5629c34d60b0 con 0x5629c3564ee0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f961000fb30 con 0x5629c3564ee0 2024-04-22T00:06:48.402 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f961009fdc0 con 0x5629c3564ee0 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 10 client.26708 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5629c3581520 con 0x5629c3564ee0 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5629c34d60b0 con 0x5629c3564ee0 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f96100a4280 con 0x5629c3564ee0 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 10 client.26708 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 10 client.26708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 20 client.26708 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.402+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c3582990 0x5629c3584d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.403+0000 7f9629f95740 4 client.26708 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:06:48.403 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.403+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629c34d0400 con 0x5629c3582990 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.403+0000 7f9625154640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c3582990 0x5629c3584d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.403+0000 7f9608ff9640 10 client.26708 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.403+0000 7f9625154640 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c3582990 0x5629c3584d80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=171a3bd61a342735 server_cookie=4415547b1866a8fd in_seq=0 out_seq=0 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9608ff9640 1 -- 172.21.15.38:0/2345179196 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629c34d0400 con 0x5629c3582990 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9608ff9640 10 client.26708 handle_command_reply: tid=0 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 client.26708 shutdown 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 msgr2=0x5629c356fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9610007590 tx=0x7f961009b370 comp rx=0 tx=0).stop 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 shutdown_connections 2024-04-22T00:06:48.404 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3572620 0x5629c35700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356fba0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c3582990 0x5629c3584d80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 --2- 172.21.15.38:0/2345179196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c356f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9608ff9640 0 client.26708 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 >> 172.21.15.38:0/2345179196 conn(0x5629c355f440 msgr2=0x5629c3561190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 shutdown_connections 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 1 -- 172.21.15.38:0/2345179196 wait complete. 2024-04-22T00:06:48.405 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.404+0000 7f9629f95740 20 client.26708 trim_cache size 0 max 16384 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.421+0000 7f9629f95740 1 Processor -- start 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 -- start start 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3564ee0 0x5629c357e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3572620 0x5629c358a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96201ac810 con 0x5629c35645c0 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96201ae350 con 0x5629c3572620 2024-04-22T00:06:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9629f95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96201acee0 con 0x5629c3564ee0 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32952/0 (socket says 172.21.15.38:32952) 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 -- 172.21.15.38:0/2776285575 learned_addr learned my addr 172.21.15.38:0/2776285575 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9624953640 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3564ee0 0x5629c357e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625955640 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3572620 0x5629c358a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 -- 172.21.15.38:0/2776285575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3564ee0 msgr2=0x5629c357e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3564ee0 0x5629c357e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 -- 172.21.15.38:0/2776285575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3572620 msgr2=0x5629c358a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3572620 0x5629c358a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 -- 172.21.15.38:0/2776285575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c356dbf0 con 0x5629c35645c0 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.422+0000 7f9625154640 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f961c005ab0 tx=0x7f961c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5557879c76a680 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f960affd640 1 -- 172.21.15.38:0/2776285575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c02c900 con 0x5629c35645c0 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f960affd640 1 -- 172.21.15.38:0/2776285575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f961c02caa0 con 0x5629c35645c0 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f960affd640 1 -- 172.21.15.38:0/2776285575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c0462d0 con 0x5629c35645c0 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 -- 172.21.15.38:0/2776285575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 msgr2=0x5629c3571d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f961c005ab0 tx=0x7f961c02e040 comp rx=0 tx=0).stop 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 -- 172.21.15.38:0/2776285575 shutdown_connections 2024-04-22T00:06:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3572620 0x5629c358a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c3564ee0 0x5629c357e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 --2- 172.21.15.38:0/2776285575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c35645c0 0x5629c3571d90 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 -- 172.21.15.38:0/2776285575 >> 172.21.15.38:0/2776285575 conn(0x5629c355f440 msgr2=0x5629c356df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.423+0000 7f9629f95740 1 -- 172.21.15.38:0/2776285575 shutdown_connections 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 -- 172.21.15.38:0/2776285575 wait complete. 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 -- start start 2024-04-22T00:06:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 Processor -- start 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c356c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96201adc80 con 0x5629c3572620 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96201ad5b0 con 0x5629c3564ee0 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96201aa7f0 con 0x5629c35645c0 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9629f95740 10 client.0 _ll_register_callbacks cb 0x7f961000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:06:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9625955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c356c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9625154640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9625154640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:42716/0 (socket says 172.21.15.38:42716) 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9625955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c356c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32962/0 (socket says 172.21.15.38:32962) 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.424+0000 7f9625154640 1 -- 172.21.15.38:0/1771295145 learned_addr learned my addr 172.21.15.38:0/1771295145 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9624953640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 -- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 msgr2=0x5629c356c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 -- 172.21.15.38:0/1771295145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 msgr2=0x5629c356c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c356c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 -- 172.21.15.38:0/1771295145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c356cd50 con 0x5629c35645c0 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9624953640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9625154640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a2f346c39b2390 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9608ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c053060 con 0x5629c35645c0 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f961c02caa0 con 0x5629c35645c0 2024-04-22T00:06:48.428 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f961c04b070 con 0x5629c35645c0 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f961c05a050 con 0x5629c35645c0 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9629f95740 10 client.26584 fetch_fsmap learned FSMap version 35 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.425+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5629c35e8ec0 con 0x5629c35645c0 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f961c02caa0 con 0x5629c35645c0 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f961c0466a0 con 0x5629c35645c0 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 10 client.26584 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 10 client.26584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:06:48.429 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 20 client.26584 populate_metadata read hostname 'smithi038' 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c35e9580 0x5629c35eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 4 client.26584 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5629c344e7a0 con 0x5629c35e9580 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9624953640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c35e9580 0x5629c35eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9608ff9640 10 client.26584 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:48.430 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.426+0000 7f9624953640 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c35e9580 0x5629c35eb970 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44cada55305de8bb server_cookie=c3a54bbba38c2fdf in_seq=0 out_seq=0 2024-04-22T00:06:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9608ff9640 1 -- 172.21.15.38:0/1771295145 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5629c344e7a0 con 0x5629c35e9580 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9608ff9640 10 client.26584 handle_command_reply: tid=0 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 client.26584 shutdown 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 msgr2=0x5629c357a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f961c000c00 tx=0x7f961c005d50 comp rx=0 tx=0).stop 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 shutdown_connections 2024-04-22T00:06:48.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5629c35e9580 0x5629c35eb970 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5629c3572620 0x5629c356c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5629c3564ee0 0x5629c356c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 --2- 172.21.15.38:0/1771295145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5629c35645c0 0x5629c357a950 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 >> 172.21.15.38:0/1771295145 conn(0x5629c355f440 msgr2=0x5629c356f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.751+0000 7f9608ff9640 0 client.26584 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:06:48.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.752+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 shutdown_connections 2024-04-22T00:06:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.752+0000 7f9629f95740 1 -- 172.21.15.38:0/1771295145 wait complete. 2024-04-22T00:06:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.752+0000 7f9629f95740 20 client.26584 trim_cache size 0 max 16384 2024-04-22T00:06:48.750 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:06:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:06:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:06:48.751 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:06:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 msgr2=0x7f960c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 0x7f960c0a52e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f961801c000 tx=0x7f961801e000 comp rx=0 tx=0).stop 2024-04-22T00:06:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 msgr2=0x7f9620198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f961000c920 tx=0x7f961000cde0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 shutdown_connections 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f960c0a2e20 0x7f960c0a52e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9620107750 0x7f962019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9620102100 0x7f9620198480 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 --2- 172.21.15.38:0/3643540224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9620101740 0x7f9620197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.753+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 >> 172.21.15.38:0/3643540224 conn(0x7f96200fd210 msgr2=0x7f96200fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.754+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 shutdown_connections 2024-04-22T00:06:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:06:48.754+0000 7f96273b8640 1 -- 172.21.15.38:0/3643540224 wait complete. 2024-04-22T00:06:48.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:48 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:06:48.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:48 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:48.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:48 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:49.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:48 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:06:49.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:48 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:49.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:48 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:49.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:48 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:06:49.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:48 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:49.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:48 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:49.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:49 smithi038 ceph-mon[27972]: pgmap v145: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:06:50.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:49 smithi203 ceph-mon[31431]: pgmap v145: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:06:50.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:49 smithi111 ceph-mon[31543]: pgmap v145: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:06:52.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:51 smithi203 ceph-mon[31431]: pgmap v146: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:52.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:51 smithi111 ceph-mon[31543]: pgmap v146: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:52.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:51 smithi038 ceph-mon[27972]: pgmap v146: 129 pgs: 129 active+clean; 14 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-04-22T00:06:54.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:53 smithi203 ceph-mon[31431]: pgmap v147: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 15 MiB/s wr, 3.66k op/s 2024-04-22T00:06:54.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:53 smithi111 ceph-mon[31543]: pgmap v147: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 15 MiB/s wr, 3.66k op/s 2024-04-22T00:06:54.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:53 smithi038 ceph-mon[27972]: pgmap v147: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 15 MiB/s wr, 3.66k op/s 2024-04-22T00:06:55.829 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:06:55.536+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v409>,<0x100000001fb/client.0 v386>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v387>,<0x10000000001/sv_1 v407>,<0x10000000000/qa v390>,<0x1/volumes v375>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:06:56.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: pgmap v148: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-04-22T00:06:56.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:56.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:56.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:56.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:56.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:55 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:56.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: pgmap v148: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-04-22T00:06:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:55 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:56.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: pgmap v148: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 2.76k op/s 2024-04-22T00:06:56.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:56.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:06:56.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:56.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:56.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:55 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:57.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:56 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:57.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:56 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:57.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:56 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:58.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:58 smithi203 ceph-mon[31431]: pgmap v149: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-04-22T00:06:58.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:58 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:06:58.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:06:58 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:58.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:58 smithi111 ceph-mon[31543]: pgmap v149: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-04-22T00:06:58.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:58 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:06:58.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:06:58 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:06:58.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:58 smithi038 ceph-mon[27972]: pgmap v149: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.62k op/s 2024-04-22T00:06:58.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:58 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:06:58.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:06:58 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:00.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:00 smithi038 ceph-mon[27972]: pgmap v150: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 3.16k op/s 2024-04-22T00:07:00.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:00 smithi203 ceph-mon[31431]: pgmap v150: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 3.16k op/s 2024-04-22T00:07:00.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:00 smithi111 ceph-mon[31543]: pgmap v150: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 3.16k op/s 2024-04-22T00:07:01.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:01 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:01.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:01 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:01.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:01 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.882+0000 7f587089d640 1 -- 172.21.15.38:0/1976160125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 msgr2=0x7f586c113150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.882+0000 7f587089d640 1 --2- 172.21.15.38:0/1976160125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 0x7f586c113150 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f5854009900 tx=0x7f585402bc80 comp rx=0 tx=0).stop 2024-04-22T00:07:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.884+0000 7f587089d640 1 -- 172.21.15.38:0/1976160125 shutdown_connections 2024-04-22T00:07:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.884+0000 7f587089d640 1 --2- 172.21.15.38:0/1976160125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c113690 0x7f586c115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.884+0000 7f587089d640 1 --2- 172.21.15.38:0/1976160125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 0x7f586c113150 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.884+0000 7f587089d640 1 --2- 172.21.15.38:0/1976160125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c10a0e0 0x7f586c10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.884+0000 7f587089d640 1 -- 172.21.15.38:0/1976160125 >> 172.21.15.38:0/1976160125 conn(0x7f586c06c810 msgr2=0x7f586c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:01.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.886+0000 7f587089d640 1 -- 172.21.15.38:0/1976160125 shutdown_connections 2024-04-22T00:07:01.884 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.888+0000 7f587089d640 1 -- 172.21.15.38:0/1976160125 wait complete. 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 Processor -- start 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 -- start start 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c10a0e0 0x7f586c11e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 0x7f586c11ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f586c118270 con 0x7f586c10aa00 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1180f0 con 0x7f586c113690 2024-04-22T00:07:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.889+0000 7f587089d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f586c1183f0 con 0x7f586c10a0e0 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.890+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.890+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46796/0 (socket says 172.21.15.38:46796) 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.890+0000 7f586ad9d640 1 -- 172.21.15.38:0/1343298924 learned_addr learned my addr 172.21.15.38:0/1343298924 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.891+0000 7f586a59c640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c10a0e0 0x7f586c11e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.892+0000 7f586ad9d640 1 -- 172.21.15.38:0/1343298924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c10a0e0 msgr2=0x7f586c11e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.892+0000 7f586ad9d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c10a0e0 0x7f586c11e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.892+0000 7f586ad9d640 1 -- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 msgr2=0x7f586c11ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.892+0000 7f586ad9d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 0x7f586c11ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.892+0000 7f586ad9d640 1 -- 172.21.15.38:0/1343298924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c11a140 con 0x7f586c113690 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.895+0000 7f586ad9d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f586400b9e0 tx=0x7f586400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.896+0000 7f58637fe640 1 -- 172.21.15.38:0/1343298924 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864019070 con 0x7f586c113690 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.896+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c11a430 con 0x7f586c113690 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.896+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c124610 con 0x7f586c113690 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.896+0000 7f58637fe640 1 -- 172.21.15.38:0/1343298924 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5864014440 con 0x7f586c113690 2024-04-22T00:07:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.896+0000 7f58637fe640 1 -- 172.21.15.38:0/1343298924 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58640125e0 con 0x7f586c113690 2024-04-22T00:07:01.897 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.899+0000 7f5873567740 1 Processor -- start 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.902+0000 7f5873567740 1 -- start start 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ffa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560151f52b70 con 0x560151fef440 2024-04-22T00:07:01.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560151efb470 con 0x560151ff8210 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5873567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560151f4eb30 con 0x560151feeb20 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5863fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5863fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:41578/0 (socket says 172.21.15.38:41578) 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f5863fff640 1 -- 172.21.15.38:0/2881753772 learned_addr learned my addr 172.21.15.38:0/2881753772 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.903+0000 7f58637fe640 1 -- 172.21.15.38:0/1343298924 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f586401c070 con 0x7f586c113690 2024-04-22T00:07:01.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f586a59c640 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 -- 172.21.15.38:0/2881753772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 msgr2=0x560151ffa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ffa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 -- 172.21.15.38:0/2881753772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 msgr2=0x560151ff7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 -- 172.21.15.38:0/2881753772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560151c67210 con 0x560151feeb20 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.904+0000 7f5863fff640 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5854005ab0 tx=0x7f585402c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d28fe148fde2e293 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.905+0000 7f58637fe640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 0x7f58580a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.905+0000 7f58637fe640 1 -- 172.21.15.38:0/1343298924 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f58640cebe0 con 0x7f586c113690 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.907+0000 7f5860ff9640 1 -- 172.21.15.38:0/2881753772 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854004050 con 0x560151feeb20 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.907+0000 7f5860ff9640 1 -- 172.21.15.38:0/2881753772 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58540041f0 con 0x560151feeb20 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.908+0000 7f5873567740 1 -- 172.21.15.38:0/2881753772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 msgr2=0x560151feef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.908+0000 7f5873567740 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5854005ab0 tx=0x7f585402c700 comp rx=0 tx=0).stop 2024-04-22T00:07:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.909+0000 7f5873567740 1 -- 172.21.15.38:0/2881753772 shutdown_connections 2024-04-22T00:07:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.909+0000 7f5873567740 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ffa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.909+0000 7f5873567740 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.909+0000 7f5873567740 1 --2- 172.21.15.38:0/2881753772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151feef00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.909+0000 7f5873567740 1 -- 172.21.15.38:0/2881753772 >> 172.21.15.38:0/2881753772 conn(0x560151fe9440 msgr2=0x560151fe9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.910+0000 7f586a59c640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 0x7f58580a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.911+0000 7f5873567740 1 -- 172.21.15.38:0/2881753772 shutdown_connections 2024-04-22T00:07:01.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.914+0000 7f5873567740 1 -- 172.21.15.38:0/2881753772 wait complete. 2024-04-22T00:07:01.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.915+0000 7f5873567740 1 -- start start 2024-04-22T00:07:01.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.915+0000 7f5873567740 1 Processor -- start 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151ff51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ff5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560151ffade0 con 0x560151fef440 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560151f0ea10 con 0x560151ff8210 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560151ffaf60 con 0x560151feeb20 2024-04-22T00:07:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5873567740 10 client.0 _ll_register_callbacks cb 0x560151f49b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5863fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38934/0 (socket says 172.21.15.38:38934) 2024-04-22T00:07:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.916+0000 7f5863fff640 1 -- 172.21.15.38:0/3895395213 learned_addr learned my addr 172.21.15.38:0/3895395213 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 -- 172.21.15.38:0/3895395213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 msgr2=0x560151ff51c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:07:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151ff51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 -- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 msgr2=0x560151ff5c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ff5c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 -- 172.21.15.38:0/3895395213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560151d33620 con 0x560151fef440 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5863fff640 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f58540045d0 tx=0x7f58540053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443330dbe8cdbe80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.917+0000 7f5851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.918+0000 7f586a59c640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 0x7f58580a53b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x560151fee5d0 tx=0x7f585c008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.924+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854004050 con 0x560151fef440 2024-04-22T00:07:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.924+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5854003a00 con 0x560151fef440 2024-04-22T00:07:01.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.926+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58540041f0 con 0x560151fef440 2024-04-22T00:07:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.928+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5854003c00 con 0x560151fef440 2024-04-22T00:07:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.928+0000 7f5873567740 10 client.17340 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.928+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560152007070 con 0x560151fef440 2024-04-22T00:07:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58540041f0 con 0x560151fef440 2024-04-22T00:07:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f585404a610 con 0x560151fef440 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 10 client.17340 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 10 client.17340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 20 client.17340 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5601520083c0 0x56015200a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 4 client.17340 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.929+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560151f600b0 con 0x5601520083c0 2024-04-22T00:07:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.930+0000 7f586a59c640 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5601520083c0 0x56015200a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.931+0000 7f5851ffb640 10 client.17340 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.931+0000 7f586a59c640 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5601520083c0 0x56015200a7b0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cf6d27ead113bab server_cookie=5072c37f7b381a79 in_seq=0 out_seq=0 2024-04-22T00:07:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.932+0000 7f5851ffb640 1 -- 172.21.15.38:0/3895395213 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560151f600b0 con 0x5601520083c0 2024-04-22T00:07:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.932+0000 7f5851ffb640 10 client.17340 handle_command_reply: tid=0 2024-04-22T00:07:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.932+0000 7f5873567740 1 client.17340 shutdown 2024-04-22T00:07:01.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.933+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 msgr2=0x560151ff5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.929 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.933+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f58540045d0 tx=0x7f58540053f0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 shutdown_connections 2024-04-22T00:07:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151ff8210 0x560151ff5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5601520083c0 0x56015200a7b0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151fef440 0x560151ff5700 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 --2- 172.21.15.38:0/3895395213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560151ff51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.936+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 >> 172.21.15.38:0/3895395213 conn(0x560151fe9440 msgr2=0x560151feb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.937+0000 7f5851ffb640 0 client.17340 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.938+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 shutdown_connections 2024-04-22T00:07:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.938+0000 7f5873567740 1 -- 172.21.15.38:0/3895395213 wait complete. 2024-04-22T00:07:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.938+0000 7f5873567740 20 client.17340 trim_cache size 0 max 16384 2024-04-22T00:07:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.967+0000 7f5873567740 1 Processor -- start 2024-04-22T00:07:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.967+0000 7f5873567740 1 -- start start 2024-04-22T00:07:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.967+0000 7f5873567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.967+0000 7f5873567740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5873567740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x56015200fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5863fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5863fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46822/0 (socket says 172.21.15.38:46822) 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5863fff640 1 -- 172.21.15.38:0/3091950969 learned_addr learned my addr 172.21.15.38:0/3091950969 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f584c00fed0 con 0x560151ff8210 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f584c011a10 con 0x560151fef440 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0105a0 con 0x560151feeb20 2024-04-22T00:07:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.968+0000 7f586ad9d640 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x56015200fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 -- 172.21.15.38:0/3091950969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 msgr2=0x560152003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 -- 172.21.15.38:0/3091950969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 msgr2=0x56015200fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x56015200fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 -- 172.21.15.38:0/3091950969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560151fe9270 con 0x560151fef440 2024-04-22T00:07:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.969+0000 7f5863fff640 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f585404ab80 tx=0x7f585402ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbaa5b1fabf51c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.970+0000 7f5853fff640 1 -- 172.21.15.38:0/3091950969 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58540058d0 con 0x560151fef440 2024-04-22T00:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.971+0000 7f5853fff640 1 -- 172.21.15.38:0/3091950969 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f585404f070 con 0x560151fef440 2024-04-22T00:07:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.971+0000 7f5853fff640 1 -- 172.21.15.38:0/3091950969 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854003a60 con 0x560151fef440 2024-04-22T00:07:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.972+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 msgr2=0x560151fe4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.972+0000 7f5873567740 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f585404ab80 tx=0x7f585402ccc0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 shutdown_connections 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x56015200fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151fe4220 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 --2- 172.21.15.38:0/3091950969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 >> 172.21.15.38:0/3091950969 conn(0x560151fe9440 msgr2=0x560151ff3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 shutdown_connections 2024-04-22T00:07:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.982+0000 7f5873567740 1 -- 172.21.15.38:0/3091950969 wait complete. 2024-04-22T00:07:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 -- start start 2024-04-22T00:07:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 Processor -- start 2024-04-22T00:07:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152000390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151ffb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f584c011340 con 0x560151ff8210 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f584c010c70 con 0x560151fef440 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f584c00deb0 con 0x560151feeb20 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.983+0000 7f5873567740 10 client.0 _ll_register_callbacks cb 0x7f5854009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38946/0 (socket says 172.21.15.38:38946) 2024-04-22T00:07:01.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 -- 172.21.15.38:0/2309030966 learned_addr learned my addr 172.21.15.38:0/2309030966 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 -- 172.21.15.38:0/2309030966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 msgr2=0x560152000390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:07:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152000390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 -- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 msgr2=0x560151ffb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.984+0000 7f586ad9d640 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151ffb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.985+0000 7f586ad9d640 1 -- 172.21.15.38:0/2309030966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560151ffa600 con 0x560151ff8210 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.985+0000 7f586ad9d640 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5864014eb0 tx=0x7f586409b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b91d97c3aacefd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.985+0000 7f5851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58640cc420 con 0x560151ff8210 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f58640a3850 con 0x560151ff8210 2024-04-22T00:07:01.982 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f585404f070 con 0x560151ff8210 2024-04-22T00:07:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f58640a3ae0 con 0x560151ff8210 2024-04-22T00:07:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5873567740 10 client.17346 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.986+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56015206e200 con 0x560151ff8210 2024-04-22T00:07:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f585404f070 con 0x560151ff8210 2024-04-22T00:07:01.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f58640b9ab0 con 0x560151ff8210 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 10 client.17346 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 10 client.17346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 20 client.17346 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56015206e9a0 0x560152070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 4 client.17346 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.987+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560151fe4900 con 0x56015206e9a0 2024-04-22T00:07:01.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.988+0000 7f586a59c640 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56015206e9a0 0x560152070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:01.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.992+0000 7f586a59c640 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56015206e9a0 0x560152070e60 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31c9fa40bf75640 server_cookie=5afebe1a28ebe719 in_seq=0 out_seq=0 2024-04-22T00:07:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:01.994+0000 7f5851ffb640 10 client.17346 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:02.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:02 smithi203 ceph-mon[31431]: pgmap v151: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.314+0000 7f5851ffb640 1 -- 172.21.15.38:0/2309030966 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560151fe4900 con 0x56015206e9a0 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.314+0000 7f5851ffb640 10 client.17346 handle_command_reply: tid=0 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 client.17346 shutdown 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 msgr2=0x560151ffbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5864014eb0 tx=0x7f586409b570 comp rx=0 tx=0).stop 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 shutdown_connections 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56015206e9a0 0x560152070e60 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5851ffb640 0 client.17346 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560151ff8210 0x560151ffbd40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560151fef440 0x560151ffb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 --2- 172.21.15.38:0/2309030966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560151feeb20 0x560152000390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 >> 172.21.15.38:0/2309030966 conn(0x560151fe9440 msgr2=0x560151fe7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 shutdown_connections 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 1 -- 172.21.15.38:0/2309030966 wait complete. 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.315+0000 7f5873567740 20 client.17346 trim_cache size 0 max 16384 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:07:02.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:07:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 msgr2=0x7f58580a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 0x7f58580a53b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x560151fee5d0 tx=0x7f585c008040 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 msgr2=0x7f586c119880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f586400b9e0 tx=0x7f586400bea0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 shutdown_connections 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f58580a2ef0 0x7f58580a53b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f586c113690 0x7f586c119880 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f586c10aa00 0x7f586c11ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 --2- 172.21.15.38:0/1343298924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f586c10a0e0 0x7f586c11e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 >> 172.21.15.38:0/1343298924 conn(0x7f586c06c810 msgr2=0x7f586c070460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 shutdown_connections 2024-04-22T00:07:02.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:02.316+0000 7f587089d640 1 -- 172.21.15.38:0/1343298924 wait complete. 2024-04-22T00:07:02.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:02 smithi111 ceph-mon[31543]: pgmap v151: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-04-22T00:07:02.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:02 smithi038 ceph-mon[27972]: pgmap v151: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 12 MiB/s wr, 3.02k op/s 2024-04-22T00:07:04.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:04 smithi203 ceph-mon[31431]: pgmap v152: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-04-22T00:07:04.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:04 smithi111 ceph-mon[31543]: pgmap v152: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-04-22T00:07:04.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:04 smithi038 ceph-mon[27972]: pgmap v152: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-04-22T00:07:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:06 smithi203 ceph-mon[31431]: pgmap v153: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-04-22T00:07:06.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:06 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:06.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:06 smithi111 ceph-mon[31543]: pgmap v153: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-04-22T00:07:06.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:06 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:06.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:06 smithi038 ceph-mon[27972]: pgmap v153: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-04-22T00:07:06.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:06 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:07.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:07.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:07.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:07.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:07 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:07.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:07.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:07.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:07.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:07 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:07.459 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:07.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:07.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:07.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:07 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:08.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:08 smithi203 ceph-mon[31431]: pgmap v154: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 14 MiB/s wr, 3.45k op/s 2024-04-22T00:07:08.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:08 smithi111 ceph-mon[31543]: pgmap v154: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 14 MiB/s wr, 3.45k op/s 2024-04-22T00:07:08.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:08 smithi038 ceph-mon[27972]: pgmap v154: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 14 MiB/s wr, 3.45k op/s 2024-04-22T00:07:10.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:10 smithi111 ceph-mon[31543]: pgmap v155: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 2.66k op/s 2024-04-22T00:07:10.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:10 smithi038 ceph-mon[27972]: pgmap v155: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 2.66k op/s 2024-04-22T00:07:10.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:10 smithi203 ceph-mon[31431]: pgmap v155: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 2.66k op/s 2024-04-22T00:07:12.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:12 smithi111 ceph-mon[31543]: pgmap v156: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-04-22T00:07:12.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:12 smithi038 ceph-mon[27972]: pgmap v156: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-04-22T00:07:12.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:12 smithi203 ceph-mon[31431]: pgmap v156: 129 pgs: 129 active+clean; 14 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 9.4 MiB/s wr, 2.36k op/s 2024-04-22T00:07:14.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:14 smithi203 ceph-mon[31431]: pgmap v157: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:14.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:14 smithi111 ceph-mon[31543]: pgmap v157: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:14.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:14 smithi038 ceph-mon[27972]: pgmap v157: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c0bfff640 1 -- 172.21.15.38:0/2052008731 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1002cd40 con 0x7f8c141029d0 2024-04-22T00:07:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/2052008731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141029d0 msgr2=0x7f8c14102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.459 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/2052008731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141029d0 0x7f8c14102db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009900 tx=0x7f8c1002bcf0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/2052008731 shutdown_connections 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/2052008731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1410bda0 0x7f8c1410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/2052008731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c141032f0 0x7f8c1410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/2052008731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141029d0 0x7f8c14102db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/2052008731 >> 172.21.15.38:0/2052008731 conn(0x7f8c14077760 msgr2=0x7f8c14077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.463+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/2052008731 shutdown_connections 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.464+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/2052008731 wait complete. 2024-04-22T00:07:15.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.464+0000 7f8c1b5d7640 1 Processor -- start 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 -- start start 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141032f0 0x7f8c14115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c1410bda0 0x7f8c14116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141020f0 con 0x7f8c1410bda0 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c14101f70 con 0x7f8c1411bcd0 2024-04-22T00:07:15.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c1b5d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c14102270 con 0x7f8c141032f0 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c19373640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141032f0 0x7f8c14115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c19b74640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c19b74640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:57178/0 (socket says 172.21.15.38:57178) 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c19b74640 1 -- 172.21.15.38:0/3915003623 learned_addr learned my addr 172.21.15.38:0/3915003623 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.465+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c1410bda0 0x7f8c14116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 -- 172.21.15.38:0/3915003623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141032f0 msgr2=0x7f8c14115fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141032f0 0x7f8c14115fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 -- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c1410bda0 msgr2=0x7f8c14116500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c1410bda0 0x7f8c14116500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 -- 172.21.15.38:0/3915003623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1411c990 con 0x7f8c1411bcd0 2024-04-22T00:07:15.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c19b74640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00eb40 tx=0x7f8c0c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.466+0000 7f8c0a7fc640 1 -- 172.21.15.38:0/3915003623 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c004b20 con 0x7f8c1411bcd0 2024-04-22T00:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.467+0000 7f8c0a7fc640 1 -- 172.21.15.38:0/3915003623 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c004cc0 con 0x7f8c1411bcd0 2024-04-22T00:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.467+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c141111e0 con 0x7f8c1411bcd0 2024-04-22T00:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.467+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c14111730 con 0x7f8c1411bcd0 2024-04-22T00:07:15.463 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.467+0000 7f8c0a7fc640 1 -- 172.21.15.38:0/3915003623 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c010790 con 0x7f8c1411bcd0 2024-04-22T00:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.470+0000 7f8c1e1c2740 1 Processor -- start 2024-04-22T00:07:15.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.471+0000 7f8c0a7fc640 1 -- 172.21.15.38:0/3915003623 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8c0c005000 con 0x7f8c1411bcd0 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.473+0000 7f8c0a7fc640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 0x7f8c000a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.473+0000 7f8c0a7fc640 1 -- 172.21.15.38:0/3915003623 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8c0c0cfea0 con 0x7f8c1411bcd0 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.473+0000 7f8c1e1c2740 1 -- start start 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.474+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 0x7f8c000a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a0b20 0x5572122a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557212204b70 con 0x5572122a1440 2024-04-22T00:07:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5572121ad470 con 0x5572122a0b20 2024-04-22T00:07:15.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.475+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557212200b30 con 0x5572122aa210 2024-04-22T00:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57498/0 (socket says 172.21.15.38:57498) 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 -- 172.21.15.38:0/1586588543 learned_addr learned my addr 172.21.15.38:0/1586588543 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c19373640 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a0b20 0x5572122a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 0x7f8c000a5360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c14112890 tx=0x7f8c04009830 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 -- 172.21.15.38:0/1586588543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 msgr2=0x5572122ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 -- 172.21.15.38:0/1586588543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a0b20 msgr2=0x5572122a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a0b20 0x5572122a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.476+0000 7f8c18b72640 1 -- 172.21.15.38:0/1586588543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557211f19210 con 0x5572122a1440 2024-04-22T00:07:15.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.477+0000 7f8c18b72640 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009f20 tx=0x7f8c0401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810b81c630c17aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.478+0000 7f8bff7fe640 1 -- 172.21.15.38:0/1586588543 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04004030 con 0x5572122a1440 2024-04-22T00:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.478+0000 7f8bff7fe640 1 -- 172.21.15.38:0/1586588543 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c04005500 con 0x5572122a1440 2024-04-22T00:07:15.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.479+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/1586588543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 msgr2=0x5572122a9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.479+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009f20 tx=0x7f8c0401a000 comp rx=0 tx=0).stop 2024-04-22T00:07:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.480+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/1586588543 shutdown_connections 2024-04-22T00:07:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.480+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.480+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a9cd0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.480+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/1586588543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a0b20 0x5572122a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.480+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/1586588543 >> 172.21.15.38:0/1586588543 conn(0x55721229b440 msgr2=0x55721229b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.482+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/1586588543 shutdown_connections 2024-04-22T00:07:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.483+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/1586588543 wait complete. 2024-04-22T00:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.483+0000 7f8c1e1c2740 1 -- start start 2024-04-22T00:07:15.478 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.483+0000 7f8c1e1c2740 1 Processor -- start 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122a7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x5572122a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5572122ace40 con 0x5572122a1440 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5572121c0a10 con 0x5572122aa210 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5572122acfc0 con 0x5572122a0b20 2024-04-22T00:07:15.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.484+0000 7f8c1e1c2740 10 client.0 _ll_register_callbacks cb 0x5572121fbb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.485+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.485+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57502/0 (socket says 172.21.15.38:57502) 2024-04-22T00:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.485+0000 7f8c18b72640 1 -- 172.21.15.38:0/3631032786 learned_addr learned my addr 172.21.15.38:0/3631032786 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:15.480 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.485+0000 7f8c19b74640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x5572122a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.486+0000 7f8c18b72640 1 -- 172.21.15.38:0/3631032786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 msgr2=0x5572122a7280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.486+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122a7280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.486+0000 7f8c18b72640 1 -- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 msgr2=0x5572122a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.486+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x5572122a7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.486+0000 7f8c18b72640 1 -- 172.21.15.38:0/3631032786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557211fe5620 con 0x5572122a1440 2024-04-22T00:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.487+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8c04004880 tx=0x7f8c04005360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a9f7f9530a383 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.487+0000 7f8bfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.490+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04004660 con 0x5572122a1440 2024-04-22T00:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.490+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8c04025430 con 0x5572122a1440 2024-04-22T00:07:15.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.490+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c04005500 con 0x5572122a1440 2024-04-22T00:07:15.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.492+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c04025630 con 0x5572122a1440 2024-04-22T00:07:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.492+0000 7f8c1e1c2740 10 client.17358 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:15.487 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.492+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572122b9180 con 0x5572122a1440 2024-04-22T00:07:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.494+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c04005500 con 0x5572122a1440 2024-04-22T00:07:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.494+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8c040108f0 con 0x5572122a1440 2024-04-22T00:07:15.490 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 10 client.17358 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 10 client.17358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 20 client.17358 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572122ba4d0 0x5572122bc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 4 client.17358 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:15.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.495+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572122120b0 con 0x5572122ba4d0 2024-04-22T00:07:15.492 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.497+0000 7f8c19373640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572122ba4d0 0x5572122bc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.501+0000 7f8c19373640 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572122ba4d0 0x5572122bc8c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db2f176ef49e0467 server_cookie=a4cb6dcdbc009c21 in_seq=0 out_seq=0 2024-04-22T00:07:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.502+0000 7f8bfcff9640 10 client.17358 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.504+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3631032786 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572122120b0 con 0x5572122ba4d0 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.504+0000 7f8bfcff9640 10 client.17358 handle_command_reply: tid=0 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.504+0000 7f8c1e1c2740 1 client.17358 shutdown 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 msgr2=0x5572122a77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8c04004880 tx=0x7f8c04005360 comp rx=0 tx=0).stop 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 shutdown_connections 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x5572122a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572122ba4d0 0x5572122bc8c0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122a77c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3631032786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 >> 172.21.15.38:0/3631032786 conn(0x55721229b440 msgr2=0x55721229d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8bfcff9640 0 client.17358 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 shutdown_connections 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3631032786 wait complete. 2024-04-22T00:07:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.505+0000 7f8c1e1c2740 20 client.17358 trim_cache size 0 max 16384 2024-04-22T00:07:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.536+0000 7f8c1e1c2740 1 Processor -- start 2024-04-22T00:07:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.536+0000 7f8c1e1c2740 1 -- start start 2024-04-22T00:07:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 0x5572122a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 0x557212296450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141c6810 con 0x5572122a0b20 2024-04-22T00:07:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141c8350 con 0x5572122a1440 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141c6ee0 con 0x5572122aa210 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c19373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 0x5572122a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c19373640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 0x5572122a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57504/0 (socket says 172.21.15.38:57504) 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c19373640 1 -- 172.21.15.38:0/2360074369 learned_addr learned my addr 172.21.15.38:0/2360074369 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 0x557212296450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c19b74640 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 -- 172.21.15.38:0/2360074369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 msgr2=0x5572122b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 -- 172.21.15.38:0/2360074369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 msgr2=0x5572122a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 0x5572122a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 -- 172.21.15.38:0/2360074369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572122a57d0 con 0x5572122a1440 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.537+0000 7f8c18b72640 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 0x557212296450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c04013fa0 tx=0x7f8c04005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f0608aeee55d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.538+0000 7f8bfeffd640 1 -- 172.21.15.38:0/2360074369 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c040054d0 con 0x5572122a1440 2024-04-22T00:07:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.538+0000 7f8bfeffd640 1 -- 172.21.15.38:0/2360074369 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c04025430 con 0x5572122a1440 2024-04-22T00:07:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.538+0000 7f8bfeffd640 1 -- 172.21.15.38:0/2360074369 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04004280 con 0x5572122a1440 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/2360074369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 msgr2=0x557212296450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 0x557212296450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c04013fa0 tx=0x7f8c04005fd0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/2360074369 shutdown_connections 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122aa210 0x5572122b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122a1440 0x557212296450 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/2360074369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a0b20 0x5572122a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/2360074369 >> 172.21.15.38:0/2360074369 conn(0x55721229b440 msgr2=0x5572122a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/2360074369 shutdown_connections 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/2360074369 wait complete. 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 -- start start 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.539+0000 7f8c1e1c2740 1 Processor -- start 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x55721231e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141c7c80 con 0x5572122a1440 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8c141c75b0 con 0x5572122aa210 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1404f500 con 0x5572122a0b20 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c1e1c2740 10 client.0 _ll_register_callbacks cb 0x7f8c0c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c19373640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57508/0 (socket says 172.21.15.38:57508) 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 -- 172.21.15.38:0/3705845476 learned_addr learned my addr 172.21.15.38:0/3705845476 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c19373640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45730/0 (socket says 172.21.15.38:45730) 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c19b74640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x55721231e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 -- 172.21.15.38:0/3705845476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 msgr2=0x5572122ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 -- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 msgr2=0x55721231e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x55721231e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 -- 172.21.15.38:0/3705845476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55721231ec30 con 0x5572122a1440 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.540+0000 7f8c18b72640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8c04010de0 tx=0x7f8c04025a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4777bea6ef62b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04025e00 con 0x5572122a1440 2024-04-22T00:07:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8c04009e40 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c04025430 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c04004040 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 10 client.17364 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55721231fee0 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c04025430 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8c04010730 con 0x5572122a1440 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 10 client.17364 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 10 client.17364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 20 client.17364 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572123203e0 0x557212322800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 4 client.17364 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:07:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.541+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55721220c400 con 0x5572123203e0 2024-04-22T00:07:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.542+0000 7f8c19373640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572123203e0 0x557212322800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.542+0000 7f8c19373640 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572123203e0 0x557212322800 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e98537f4afe828 server_cookie=b8670ebe8a5888e8 in_seq=0 out_seq=0 2024-04-22T00:07:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:15.542+0000 7f8bfcff9640 10 client.17364 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:16.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:16 smithi111 ceph-mon[31543]: pgmap v158: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.5 MiB/s wr, 2.35k op/s 2024-04-22T00:07:16.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:16 smithi038 ceph-mon[27972]: pgmap v158: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.5 MiB/s wr, 2.35k op/s 2024-04-22T00:07:16.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:16 smithi203 ceph-mon[31431]: pgmap v158: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 9.5 MiB/s wr, 2.35k op/s 2024-04-22T00:07:17.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:07:17.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 -- 172.21.15.38:0/2213027114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 msgr2=0x7f744410e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 --2- 172.21.15.38:0/2213027114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744410e1b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f743c009be0 tx=0x7f743c02c010 comp rx=0 tx=0).stop 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 -- 172.21.15.38:0/2213027114 shutdown_connections 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 --2- 172.21.15.38:0/2213027114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744410e1b0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 --2- 172.21.15.38:0/2213027114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441032f0 0x7f744410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 --2- 172.21.15.38:0/2213027114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441029d0 0x7f7444102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 -- 172.21.15.38:0/2213027114 >> 172.21.15.38:0/2213027114 conn(0x7f7444077740 msgr2=0x7f7444077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 -- 172.21.15.38:0/2213027114 shutdown_connections 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.920+0000 7f744bd12640 1 -- 172.21.15.38:0/2213027114 wait complete. 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.921+0000 7f744bd12640 1 Processor -- start 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.921+0000 7f744bd12640 1 -- start start 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 0x7f744419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7444111b10 con 0x7f744410bd10 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7444111990 con 0x7f74441032f0 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744bd12640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7444111c90 con 0x7f74441029d0 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45742/0 (socket says 172.21.15.38:45742) 2024-04-22T00:07:17.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 -- 172.21.15.38:0/2648918004 learned_addr learned my addr 172.21.15.38:0/2648918004 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:17.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f74492ad640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 0x7f744419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f744a2af640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 -- 172.21.15.38:0/2648918004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 msgr2=0x7f744419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 0x7f744419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 -- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 msgr2=0x7f744419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f7449aae640 1 -- 172.21.15.38:0/2648918004 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744419f1c0 con 0x7f74441029d0 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.922+0000 7f74492ad640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 0x7f744419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.923+0000 7f7449aae640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f744000e9f0 tx=0x7f744000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.923+0000 7f743affd640 1 -- 172.21.15.38:0/2648918004 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744000ccc0 con 0x7f74441029d0 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.923+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744419f4b0 con 0x7f74441029d0 2024-04-22T00:07:17.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.923+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74441a6d40 con 0x7f74441029d0 2024-04-22T00:07:17.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.923+0000 7f743affd640 1 -- 172.21.15.38:0/2648918004 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7440010910 con 0x7f74441029d0 2024-04-22T00:07:17.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.924+0000 7f743affd640 1 -- 172.21.15.38:0/2648918004 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440018800 con 0x7f74441029d0 2024-04-22T00:07:17.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.925+0000 7f743affd640 1 -- 172.21.15.38:0/2648918004 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f74400189a0 con 0x7f74441029d0 2024-04-22T00:07:17.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.926+0000 7f744e9d0740 1 Processor -- start 2024-04-22T00:07:17.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.926+0000 7f744e9d0740 1 -- start start 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f743affd640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 0x7f74300a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fcf30 0x55db851fd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 0x55db85205ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85208820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55db85161560 con 0x55db85206430 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55db85109730 con 0x55db851fcf30 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744e9d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55db8515d520 con 0x55db851fd850 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744a2af640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 0x7f74300a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744a2af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85208820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744a2af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85208820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:57538/0 (socket says 172.21.15.38:57538) 2024-04-22T00:07:17.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f744a2af640 1 -- 172.21.15.38:0/2601098410 learned_addr learned my addr 172.21.15.38:0/2601098410 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:17.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 0x55db85205ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f7449aae640 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fcf30 0x55db851fd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 -- 172.21.15.38:0/2601098410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fcf30 msgr2=0x55db851fd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fcf30 0x55db851fd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 -- 172.21.15.38:0/2601098410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 msgr2=0x55db85208820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85208820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.927+0000 7f74492ad640 1 -- 172.21.15.38:0/2601098410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db84f3f7d0 con 0x55db851fd850 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f743affd640 1 -- 172.21.15.38:0/2648918004 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f74400cfa80 con 0x7f74441029d0 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f744a2af640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 0x7f74300a5280 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f743c005f00 tx=0x7f743c009960 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f74492ad640 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 0x55db85205ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7434005fd0 tx=0x7f74340118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56abf1828a4af6b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f742ffff640 1 -- 172.21.15.38:0/2601098410 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743401e070 con 0x55db851fd850 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f742ffff640 1 -- 172.21.15.38:0/2601098410 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74340035f0 con 0x55db851fd850 2024-04-22T00:07:17.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.928+0000 7f742ffff640 1 -- 172.21.15.38:0/2601098410 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743400f7a0 con 0x55db851fd850 2024-04-22T00:07:17.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 -- 172.21.15.38:0/2601098410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 msgr2=0x55db85205ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:17.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 0x55db85205ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7434005fd0 tx=0x7f74340118d0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 -- 172.21.15.38:0/2601098410 shutdown_connections 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85208820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fd850 0x55db85205ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 --2- 172.21.15.38:0/2601098410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fcf30 0x55db851fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.929+0000 7f744e9d0740 1 -- 172.21.15.38:0/2601098410 >> 172.21.15.38:0/2601098410 conn(0x55db851f7660 msgr2=0x55db851f7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.930+0000 7f744e9d0740 1 -- 172.21.15.38:0/2601098410 shutdown_connections 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.930+0000 7f744e9d0740 1 -- 172.21.15.38:0/2601098410 wait complete. 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 -- start start 2024-04-22T00:07:17.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 Processor -- start 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852034a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db852039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55db85209220 con 0x55db851fcf30 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55db8511ce00 con 0x55db851fd850 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.931+0000 7f744e9d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55db852093a0 con 0x55db85206430 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744e9d0740 10 client.0 _ll_register_callbacks cb 0x55db85157eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45762/0 (socket says 172.21.15.38:45762) 2024-04-22T00:07:17.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 -- 172.21.15.38:0/1846409413 learned_addr learned my addr 172.21.15.38:0/1846409413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:17.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f74492ad640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db852039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f7449aae640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852034a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 -- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 msgr2=0x55db852039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:17.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db852039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 -- 172.21.15.38:0/1846409413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 msgr2=0x55db852034a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:17.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852034a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.932+0000 7f744a2af640 1 -- 172.21.15.38:0/1846409413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db84e753f0 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f744a2af640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f743c02dc60 tx=0x7f743c048000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55e7d7c4f86f8ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f742d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c052070 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74340035f0 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f743c03a040 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f743c034430 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f744e9d0740 10 client.26614 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.933+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55db84e350f0 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74340035f0 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f743c04d070 con 0x55db85206430 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 10 client.26614 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:17.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 10 client.26614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 20 client.26614 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db8521f4e0 0x55db85221980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 4 client.26614 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.934+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db8516eaa0 con 0x55db8521f4e0 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.935+0000 7f7449aae640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db8521f4e0 0x55db85221980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.935+0000 7f742d7fa640 10 client.26614 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:17.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:17.935+0000 7f7449aae640 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db8521f4e0 0x55db85221980 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43b8c98122b3200 server_cookie=7f409bb5585cadbd in_seq=0 out_seq=0 2024-04-22T00:07:18.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:18 smithi111 ceph-mon[31543]: pgmap v159: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 2.97k op/s 2024-04-22T00:07:18.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:18 smithi038 ceph-mon[27972]: pgmap v159: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 2.97k op/s 2024-04-22T00:07:18.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:18 smithi203 ceph-mon[31431]: pgmap v159: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 2.97k op/s 2024-04-22T00:07:20.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:20 smithi111 ceph-mon[31543]: pgmap v160: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-04-22T00:07:20.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:20 smithi038 ceph-mon[27972]: pgmap v160: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-04-22T00:07:20.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:20 smithi203 ceph-mon[31431]: pgmap v160: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-04-22T00:07:22.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:22 smithi111 ceph-mon[31543]: pgmap v161: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:07:22.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:22 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:07:22.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:22 smithi038 ceph-mon[27972]: pgmap v161: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:07:22.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:22 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:07:22.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:22 smithi203 ceph-mon[31431]: pgmap v161: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:07:22.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:22 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:07:24.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:24 smithi111 ceph-mon[31543]: pgmap v162: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:24.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:24 smithi038 ceph-mon[27972]: pgmap v162: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:24.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:24 smithi203 ceph-mon[31431]: pgmap v162: 129 pgs: 129 active+clean; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 3.22k op/s 2024-04-22T00:07:26.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:26 smithi111 ceph-mon[31543]: pgmap v163: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:07:26.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:26 smithi038 ceph-mon[27972]: pgmap v163: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:07:26.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:26 smithi203 ceph-mon[31431]: pgmap v163: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 10 MiB/s wr, 2.49k op/s 2024-04-22T00:07:27.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:27 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:07:27.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:27 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:07:27.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:27 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:07:27.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:27 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:07:27.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:27 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:07:27.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:27 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:07:27.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:27 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:07:27.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:27 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:07:27.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:27 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:07:28.325 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:28 smithi111 ceph-mon[31543]: pgmap v164: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-04-22T00:07:28.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:28 smithi038 ceph-mon[27972]: pgmap v164: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-04-22T00:07:28.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:28 smithi203 ceph-mon[31431]: pgmap v164: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-04-22T00:07:30.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:30 smithi111 ceph-mon[31543]: pgmap v165: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:07:30.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:30 smithi038 ceph-mon[27972]: pgmap v165: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:07:30.531 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:30 smithi203 ceph-mon[31431]: pgmap v165: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 2.54k op/s 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.346+0000 7f8bfcff9640 1 -- 172.21.15.38:0/3705845476 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55721220c400 con 0x5572123203e0 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.346+0000 7f8bfcff9640 10 client.17364 handle_command_reply: tid=0 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.346+0000 7f8c1e1c2740 1 client.17364 shutdown 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 msgr2=0x5572122ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f8c04010de0 tx=0x7f8c04025a00 comp rx=0 tx=0).stop 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 shutdown_connections 2024-04-22T00:07:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572122aa210 0x55721231e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572123203e0 0x557212322800 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572122a1440 0x5572122ad950 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 --2- 172.21.15.38:0/3705845476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572122a0b20 0x5572122ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 >> 172.21.15.38:0/3705845476 conn(0x55721229b440 msgr2=0x557212299c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8bfcff9640 0 client.17364 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 shutdown_connections 2024-04-22T00:07:32.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f742d7fa640 1 -- 172.21.15.38:0/1846409413 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db8516eaa0 con 0x55db8521f4e0 2024-04-22T00:07:32.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f742d7fa640 10 client.26614 handle_command_reply: tid=0 2024-04-22T00:07:32.343 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f744e9d0740 1 client.26614 shutdown 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 1 -- 172.21.15.38:0/3705845476 wait complete. 2024-04-22T00:07:32.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f8c1e1c2740 20 client.17364 trim_cache size 0 max 16384 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 msgr2=0x55db85203f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.347+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f743c02dc60 tx=0x7f743c048000 comp rx=0 tx=0).stop 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 shutdown_connections 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db85203f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db852039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db8521f4e0 0x55db85221980 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1846409413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 >> 172.21.15.38:0/1846409413 conn(0x55db851f7660 msgr2=0x55db851f93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f742d7fa640 0 client.26614 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:32.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 shutdown_connections 2024-04-22T00:07:32.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 1 -- 172.21.15.38:0/1846409413 wait complete. 2024-04-22T00:07:32.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.348+0000 7f744e9d0740 20 client.26614 trim_cache size 0 max 16384 2024-04-22T00:07:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.352+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 msgr2=0x7f8c000a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.352+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 0x7f8c000a5360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c14112890 tx=0x7f8c04009830 comp rx=0 tx=0).stop 2024-04-22T00:07:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.352+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 msgr2=0x7f8c1411c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.352+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00eb40 tx=0x7f8c0c00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 shutdown_connections 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8c000a2ea0 0x7f8c000a5360 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c1411bcd0 0x7f8c1411c180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c1410bda0 0x7f8c14116500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 --2- 172.21.15.38:0/3915003623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c141032f0 0x7f8c14115fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 >> 172.21.15.38:0/3915003623 conn(0x7f8c14077760 msgr2=0x7f8c140fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 shutdown_connections 2024-04-22T00:07:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:32.353+0000 7f8c1b5d7640 1 -- 172.21.15.38:0/3915003623 wait complete. 2024-04-22T00:07:32.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.391+0000 7f744e9d0740 1 Processor -- start 2024-04-22T00:07:32.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.391+0000 7f744e9d0740 1 -- start start 2024-04-22T00:07:32.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.392+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 0x55db85205bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.392+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db851fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.392+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85200700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.392+0000 7f7449aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 0x55db85205bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.392+0000 7f7449aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 0x55db85205bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:56668/0 (socket says 172.21.15.38:56668) 2024-04-22T00:07:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.393+0000 7f7449aae640 1 -- 172.21.15.38:0/4078568782 learned_addr learned my addr 172.21.15.38:0/4078568782 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.393+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f742800fd70 con 0x55db85206430 2024-04-22T00:07:32.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.393+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f74280118b0 con 0x55db851fd850 2024-04-22T00:07:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.393+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7428010440 con 0x55db851fcf30 2024-04-22T00:07:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.393+0000 7f74492ad640 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db851fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f744a2af640 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85200700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f74492ad640 1 -- 172.21.15.38:0/4078568782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 msgr2=0x55db85205bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:32.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f74492ad640 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 0x55db85205bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f74492ad640 1 -- 172.21.15.38:0/4078568782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 msgr2=0x55db85200700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:32.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f74492ad640 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85200700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.394+0000 7f74492ad640 1 -- 172.21.15.38:0/4078568782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db852019f0 con 0x55db851fd850 2024-04-22T00:07:32.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.399+0000 7f74492ad640 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db851fdc30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f743401c040 tx=0x7f7434005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7efd39dc49d059be server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:32.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.400+0000 7f742f7fe640 1 -- 172.21.15.38:0/4078568782 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434027070 con 0x55db851fd850 2024-04-22T00:07:32.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.400+0000 7f742f7fe640 1 -- 172.21.15.38:0/4078568782 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74340045e0 con 0x55db851fd850 2024-04-22T00:07:32.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.400+0000 7f742f7fe640 1 -- 172.21.15.38:0/4078568782 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743401e040 con 0x55db851fd850 2024-04-22T00:07:32.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.401+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 msgr2=0x55db851fdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.401+0000 7f744e9d0740 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db851fdc30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f743401c040 tx=0x7f7434005040 comp rx=0 tx=0).stop 2024-04-22T00:07:32.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.406+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 shutdown_connections 2024-04-22T00:07:32.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.406+0000 7f744e9d0740 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db85206430 0x55db85200700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.406+0000 7f744e9d0740 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db851fdc30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.406+0000 7f744e9d0740 1 --2- 172.21.15.38:0/4078568782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db851fcf30 0x55db85205bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.406+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 >> 172.21.15.38:0/4078568782 conn(0x55db851f7660 msgr2=0x55db85201d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.407+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 shutdown_connections 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.407+0000 7f744e9d0740 1 -- 172.21.15.38:0/4078568782 wait complete. 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.408+0000 7f744e9d0740 1 -- start start 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.408+0000 7f744e9d0740 1 Processor -- start 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db85224830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db852839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f74280111e0 con 0x55db851fcf30 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7428010b10 con 0x55db851fd850 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f742800dd50 con 0x55db85206430 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f744e9d0740 10 client.0 _ll_register_callbacks cb 0x7f743c009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f7449aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f7449aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36882/0 (socket says 172.21.15.38:36882) 2024-04-22T00:07:32.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f7449aae640 1 -- 172.21.15.38:0/1548800079 learned_addr learned my addr 172.21.15.38:0/1548800079 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:32.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.409+0000 7f74492ad640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db85224830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 -- 172.21.15.38:0/1548800079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 msgr2=0x55db852839e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:32.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db852839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 -- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 msgr2=0x55db85224830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db85224830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 -- 172.21.15.38:0/1548800079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db85283f70 con 0x55db851fcf30 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.410+0000 7f7449aae640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f744009bbf0 tx=0x7f74400d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e8dee37560986d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.411+0000 7f742d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.411+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74400b9060 con 0x55db851fcf30 2024-04-22T00:07:32.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.411+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f74400982d0 con 0x55db851fcf30 2024-04-22T00:07:32.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.411+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74340045e0 con 0x55db851fcf30 2024-04-22T00:07:32.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.412+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f74400b4e60 con 0x55db851fcf30 2024-04-22T00:07:32.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.413+0000 7f744e9d0740 10 client.17382 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:32.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.413+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55db852852e0 con 0x55db851fcf30 2024-04-22T00:07:32.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.413+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74340045e0 con 0x55db851fcf30 2024-04-22T00:07:32.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.413+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f744009b0b0 con 0x55db851fcf30 2024-04-22T00:07:32.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.413+0000 7f744e9d0740 10 client.17382 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:32.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f744e9d0740 10 client.17382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f744e9d0740 20 client.17382 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db85285a10 0x55db85287e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f744e9d0740 4 client.17382 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:07:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db85168df0 con 0x55db85285a10 2024-04-22T00:07:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.414+0000 7f74492ad640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db85285a10 0x55db85287e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.417+0000 7f74492ad640 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db85285a10 0x55db85287e30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74254dca299899c5 server_cookie=15a5984afeccbe89 in_seq=0 out_seq=0 2024-04-22T00:07:32.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f742d7fa640 10 client.17382 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:32.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f742d7fa640 1 -- 172.21.15.38:0/1548800079 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55db85168df0 con 0x55db85285a10 2024-04-22T00:07:32.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f742d7fa640 10 client.17382 handle_command_reply: tid=0 2024-04-22T00:07:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f744e9d0740 1 client.17382 shutdown 2024-04-22T00:07:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 msgr2=0x55db852242f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.418+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f744009bbf0 tx=0x7f74400d1040 comp rx=0 tx=0).stop 2024-04-22T00:07:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 shutdown_connections 2024-04-22T00:07:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55db85206430 0x55db852839e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55db851fd850 0x55db85224830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55db85285a10 0x55db85287e30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 --2- 172.21.15.38:0/1548800079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db851fcf30 0x55db852242f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f742d7fa640 0 client.17382 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:32.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.419+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 >> 172.21.15.38:0/1548800079 conn(0x55db851f7660 msgr2=0x55db85211680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.421+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 shutdown_connections 2024-04-22T00:07:32.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.421+0000 7f744e9d0740 1 -- 172.21.15.38:0/1548800079 wait complete. 2024-04-22T00:07:32.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.421+0000 7f744e9d0740 20 client.17382 trim_cache size 0 max 16384 2024-04-22T00:07:32.423 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:07:32.424 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:07:32.424 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:07:32.424 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:07:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 msgr2=0x7f74300a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 0x7f74300a5280 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f743c005f00 tx=0x7f743c009960 comp rx=0 tx=0).stop 2024-04-22T00:07:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 msgr2=0x7f744419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f744000e9f0 tx=0x7f744000edf0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 shutdown_connections 2024-04-22T00:07:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74300a2dc0 0x7f74300a5280 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f744410bd10 0x7f744419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74441032f0 0x7f744419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 --2- 172.21.15.38:0/2648918004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74441029d0 0x7f744419a1d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 >> 172.21.15.38:0/2648918004 conn(0x7f7444077740 msgr2=0x7f74441019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 shutdown_connections 2024-04-22T00:07:32.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.423+0000 7f744bd12640 1 -- 172.21.15.38:0/2648918004 wait complete. 2024-04-22T00:07:32.439 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:07:32.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:07:32.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.629+0000 7f97b223b640 1 -- 172.21.15.38:0/3674311096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac108580 msgr2=0x7f97ac1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.629+0000 7f97b223b640 1 --2- 172.21.15.38:0/3674311096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac108580 0x7f97ac1089e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009890 tx=0x7f97a002bbc0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 -- 172.21.15.38:0/3674311096 shutdown_connections 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 --2- 172.21.15.38:0/3674311096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac108f20 0x7f97ac114bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 --2- 172.21.15.38:0/3674311096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac108580 0x7f97ac1089e0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009890 tx=0x7f97a002bbc0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 --2- 172.21.15.38:0/3674311096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ac10cd50 0x7f97ac10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 -- 172.21.15.38:0/3674311096 >> 172.21.15.38:0/3674311096 conn(0x7f97ac0775b0 msgr2=0x7f97ac0799d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 -- 172.21.15.38:0/3674311096 shutdown_connections 2024-04-22T00:07:32.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.630+0000 7f97b223b640 1 -- 172.21.15.38:0/3674311096 wait complete. 2024-04-22T00:07:32.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.631+0000 7f97b223b640 1 Processor -- start 2024-04-22T00:07:32.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.631+0000 7f97b223b640 1 -- start start 2024-04-22T00:07:32.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.631+0000 7f97b223b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ac108f20 0x7f97ac073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.631+0000 7f97b223b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac10cd50 0x7f97ac06b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.631+0000 7f97b223b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.632+0000 7f97b223b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac1181a0 con 0x7f97ac06b770 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.632+0000 7f97b223b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac118020 con 0x7f97ac10cd50 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.632+0000 7f97b223b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac118320 con 0x7f97ac108f20 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36906/0 (socket says 172.21.15.38:36906) 2024-04-22T00:07:32.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 -- 172.21.15.38:0/2729558444 learned_addr learned my addr 172.21.15.38:0/2729558444 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 -- 172.21.15.38:0/2729558444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ac108f20 msgr2=0x7f97ac073220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ac108f20 0x7f97ac073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 -- 172.21.15.38:0/2729558444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac10cd50 msgr2=0x7f97ac06b230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac10cd50 0x7f97ac06b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 -- 172.21.15.38:0/2729558444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac1b36d0 con 0x7f97ac06b770 2024-04-22T00:07:32.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.633+0000 7f97abfff640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f97a400ea60 tx=0x7f97a400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.634+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a400ce40 con 0x7f97ac06b770 2024-04-22T00:07:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.634+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a4005020 con 0x7f97ac06b770 2024-04-22T00:07:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.634+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ac1b39c0 con 0x7f97ac06b770 2024-04-22T00:07:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.634+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4010690 con 0x7f97ac06b770 2024-04-22T00:07:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.635+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ac1b3ee0 con 0x7f97ac06b770 2024-04-22T00:07:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.638+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9798005290 con 0x7f97ac06b770 2024-04-22T00:07:32.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.639+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f97a4010870 con 0x7f97ac06b770 2024-04-22T00:07:32.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.640+0000 7f97a8ff9640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 0x7f97940a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:32.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.641+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f97a40cfd40 con 0x7f97ac06b770 2024-04-22T00:07:32.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.645+0000 7f97ab7fe640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 0x7f97940a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:32.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.650+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f97a4016330 con 0x7f97ac06b770 2024-04-22T00:07:32.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:32.653+0000 7f97ab7fe640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 0x7f97940a52c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f979c005fd0 tx=0x7f979c009600 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:32.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:32 smithi038 ceph-mon[27972]: pgmap v166: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.2 MiB/s wr, 2.26k op/s 2024-04-22T00:07:32.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:32 smithi203 ceph-mon[31431]: pgmap v166: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.2 MiB/s wr, 2.26k op/s 2024-04-22T00:07:32.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:32 smithi111 ceph-mon[31543]: pgmap v166: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 9.2 MiB/s wr, 2.26k op/s 2024-04-22T00:07:33.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.103+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9798005850 con 0x7f97ac06b770 2024-04-22T00:07:33.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.107+0000 7f97a8ff9640 1 -- 172.21.15.38:0/2729558444 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7f97a409c5c0 con 0x7f97ac06b770 2024-04-22T00:07:33.102 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:07:33.102 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:07:33.102 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:07:33.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.110+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 msgr2=0x7f97940a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.110+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 0x7f97940a52c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f979c005fd0 tx=0x7f979c009600 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.110+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 msgr2=0x7f97ac06bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.110+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f97a400ea60 tx=0x7f97a400ef20 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 shutdown_connections 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97940a2e00 0x7f97940a52c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97ac06b770 0x7f97ac06bc20 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ac10cd50 0x7f97ac06b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 --2- 172.21.15.38:0/2729558444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ac108f20 0x7f97ac073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 >> 172.21.15.38:0/2729558444 conn(0x7f97ac0775b0 msgr2=0x7f97ac10adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 shutdown_connections 2024-04-22T00:07:33.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.111+0000 7f97b223b640 1 -- 172.21.15.38:0/2729558444 wait complete. 2024-04-22T00:07:33.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 -- 172.21.15.38:0/6805464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56881032e0 msgr2=0x7f568810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 --2- 172.21.15.38:0/6805464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56881032e0 0x7f568810b7c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5678009900 tx=0x7f567802bd60 comp rx=0 tx=0).stop 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 -- 172.21.15.38:0/6805464 shutdown_connections 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 --2- 172.21.15.38:0/6805464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f568810bd00 0x7f568810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 --2- 172.21.15.38:0/6805464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f56881032e0 0x7f568810b7c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 --2- 172.21.15.38:0/6805464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 -- 172.21.15.38:0/6805464 >> 172.21.15.38:0/6805464 conn(0x7f5688077740 msgr2=0x7f5688077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 -- 172.21.15.38:0/6805464 shutdown_connections 2024-04-22T00:07:33.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.280+0000 7f568f53a640 1 -- 172.21.15.38:0/6805464 wait complete. 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 Processor -- start 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 -- start start 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56881032e0 0x7f5688198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5688111b50 con 0x7f568810bd00 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f56881119d0 con 0x7f56881032e0 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568f53a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5688111cd0 con 0x7f56881029c0 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568dad7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568dad7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36924/0 (socket says 172.21.15.38:36924) 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568dad7640 1 -- 172.21.15.38:0/2186049882 learned_addr learned my addr 172.21.15.38:0/2186049882 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568d2d6640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.281+0000 7f568cad5640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56881032e0 0x7f5688198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 -- 172.21.15.38:0/2186049882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 msgr2=0x7f5688198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 -- 172.21.15.38:0/2186049882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56881032e0 msgr2=0x7f5688198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56881032e0 0x7f5688198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 -- 172.21.15.38:0/2186049882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568819d040 con 0x7f568810bd00 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568d2d6640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568dad7640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5680009fd0 tx=0x7f568000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f567e7fc640 1 -- 172.21.15.38:0/2186049882 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680017070 con 0x7f568810bd00 2024-04-22T00:07:33.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f567e7fc640 1 -- 172.21.15.38:0/2186049882 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5680003e50 con 0x7f568810bd00 2024-04-22T00:07:33.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568819d330 con 0x7f568810bd00 2024-04-22T00:07:33.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f567e7fc640 1 -- 172.21.15.38:0/2186049882 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800106c0 con 0x7f568810bd00 2024-04-22T00:07:33.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.282+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56881a4bc0 con 0x7f568810bd00 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.284+0000 7f567e7fc640 1 -- 172.21.15.38:0/2186049882 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5680003970 con 0x7f568810bd00 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 Processor -- start 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 -- start start 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 0x5652bffae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5652bff12530 con 0x5652bffae020 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5652bfeba830 con 0x5652bffae940 2024-04-22T00:07:33.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f5692201740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652bff0e4f0 con 0x5652bffb7520 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f567e7fc640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 0x7f56740a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 0x5652bffae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 0x5652bffae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36928/0 (socket says 172.21.15.38:36928) 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568d2d6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568cad5640 1 -- 172.21.15.38:0/4051202433 learned_addr learned my addr 172.21.15.38:0/4051202433 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568cad5640 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.285+0000 7f568dad7640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 0x7f56740a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f567e7fc640 1 -- 172.21.15.38:0/2186049882 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f56800cf430 con 0x7f568810bd00 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 -- 172.21.15.38:0/4051202433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 msgr2=0x5652bffb9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 -- 172.21.15.38:0/4051202433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 msgr2=0x5652bffae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 0x5652bffae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 -- 172.21.15.38:0/4051202433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652bfede2c0 con 0x5652bffae940 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568cad5640 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568d2d6640 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb6fe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5684009c30 tx=0x7f56840117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d5446b1daa46fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f568dad7640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 0x7f56740a5240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f568009b310 tx=0x7f5680004020 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f56737fe640 1 -- 172.21.15.38:0/4051202433 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684005020 con 0x5652bffae940 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f56737fe640 1 -- 172.21.15.38:0/4051202433 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56840047a0 con 0x5652bffae940 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.286+0000 7f56737fe640 1 -- 172.21.15.38:0/4051202433 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684012720 con 0x5652bffae940 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- 172.21.15.38:0/4051202433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 msgr2=0x5652bffb6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb6fe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5684009c30 tx=0x7f56840117b0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- 172.21.15.38:0/4051202433 shutdown_connections 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb6fe0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 --2- 172.21.15.38:0/4051202433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae020 0x5652bffae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- 172.21.15.38:0/4051202433 >> 172.21.15.38:0/4051202433 conn(0x5652bffa8750 msgr2=0x5652bffa8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- 172.21.15.38:0/4051202433 shutdown_connections 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- 172.21.15.38:0/4051202433 wait complete. 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 -- start start 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.287+0000 7f5692201740 1 Processor -- start 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 0x5652bffb6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffb4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5652bffba310 con 0x5652bffb7520 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5652bfecdf10 con 0x5652bffae940 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652bffba490 con 0x5652bffae020 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f5692201740 10 client.0 _ll_register_callbacks cb 0x5652bfc25be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffb4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffb4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36934/0 (socket says 172.21.15.38:36934) 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568cad5640 1 -- 172.21.15.38:0/853520702 learned_addr learned my addr 172.21.15.38:0/853520702 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568cad5640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 0x5652bffb6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 -- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 msgr2=0x5652bffb6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 0x5652bffb6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 -- 172.21.15.38:0/853520702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 msgr2=0x5652bffb4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffb4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568d2d6640 1 -- 172.21.15.38:0/853520702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652bfe9fc80 con 0x5652bffae940 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.288+0000 7f568cad5640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 0x5652bffb6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f568d2d6640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb41c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5684011cb0 tx=0x7f5684009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1521fdb180a58123 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684028020 con 0x5652bffae940 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56840047a0 con 0x5652bffae940 2024-04-22T00:07:33.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5684005210 con 0x5652bffae940 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5684033050 con 0x5652bffae940 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 10 client.26768 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5652bfc263f0 con 0x5652bffae940 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56840047a0 con 0x5652bffae940 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5684024460 con 0x5652bffae940 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 10 client.26768 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 10 client.26768 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 20 client.26768 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652bffc7490 0x5652bffc9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 4 client.26768 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.289+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652bff1fa70 con 0x5652bffc7490 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.290+0000 7f568cad5640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652bffc7490 0x5652bffc9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.290+0000 7f568cad5640 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652bffc7490 0x5652bffc9880 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a964bc3a0372d1b6 server_cookie=768282360a5d076a in_seq=0 out_seq=0 2024-04-22T00:07:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.290+0000 7f5670ff9640 10 client.26768 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5670ff9640 1 -- 172.21.15.38:0/853520702 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652bff1fa70 con 0x5652bffc7490 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5670ff9640 10 client.26768 handle_command_reply: tid=0 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 client.26768 shutdown 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 msgr2=0x5652bffb41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb41c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5684011cb0 tx=0x7f5684009860 comp rx=0 tx=0).stop 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 shutdown_connections 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652bffc7490 0x5652bffc9880 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffb4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae940 0x5652bffb41c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5670ff9640 0 client.26768 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 --2- 172.21.15.38:0/853520702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae020 0x5652bffb6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 >> 172.21.15.38:0/853520702 conn(0x5652bffa8750 msgr2=0x5652bffaa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 shutdown_connections 2024-04-22T00:07:33.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 1 -- 172.21.15.38:0/853520702 wait complete. 2024-04-22T00:07:33.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.291+0000 7f5692201740 20 client.26768 trim_cache size 0 max 16384 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 Processor -- start 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 -- start start 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652bffc3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56881aea10 con 0x5652bffae940 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f56881b0550 con 0x5652bffae020 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.305+0000 7f5692201740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56881af0e0 con 0x5652bffb7520 2024-04-22T00:07:33.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36948/0 (socket says 172.21.15.38:36948) 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 -- 172.21.15.38:0/3700506025 learned_addr learned my addr 172.21.15.38:0/3700506025 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568cad5640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652bffc3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568cad5640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 -- 172.21.15.38:0/3700506025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 msgr2=0x5652bffb68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 -- 172.21.15.38:0/3700506025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 msgr2=0x5652bffc3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652bffc3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 -- 172.21.15.38:0/3700506025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652bfcf07d0 con 0x5652bffae940 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568cad5640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f568d2d6640 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f5684006010 tx=0x7f5684005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e364583e942d38f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f5672ffd640 1 -- 172.21.15.38:0/3700506025 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568403c070 con 0x5652bffae940 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f5672ffd640 1 -- 172.21.15.38:0/3700506025 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5684004780 con 0x5652bffae940 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.306+0000 7f5672ffd640 1 -- 172.21.15.38:0/3700506025 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684028040 con 0x5652bffae940 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- 172.21.15.38:0/3700506025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 msgr2=0x5652bffa76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f5684006010 tx=0x7f5684005a70 comp rx=0 tx=0).stop 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- 172.21.15.38:0/3700506025 shutdown_connections 2024-04-22T00:07:33.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffb7520 0x5652bffb68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffae940 0x5652bffa76d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- 172.21.15.38:0/3700506025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652bffc3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- 172.21.15.38:0/3700506025 >> 172.21.15.38:0/3700506025 conn(0x5652bffa8750 msgr2=0x5652bffa6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- 172.21.15.38:0/3700506025 shutdown_connections 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- 172.21.15.38:0/3700506025 wait complete. 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- start start 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 Processor -- start 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652c0025ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae940 0x5652c0026010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f56881afe80 con 0x5652bffb7520 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f56881af7b0 con 0x5652bffae020 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.307+0000 7f5692201740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56881ac9f0 con 0x5652bffae940 2024-04-22T00:07:33.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5692201740 10 client.0 _ll_register_callbacks cb 0x7f5678009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568d2d6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae940 0x5652c0026010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36960/0 (socket says 172.21.15.38:36960) 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 -- 172.21.15.38:0/2656366307 learned_addr learned my addr 172.21.15.38:0/2656366307 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652c0025ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 -- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae940 msgr2=0x5652c0026010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae940 0x5652c0026010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 -- 172.21.15.38:0/2656366307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 msgr2=0x5652c0025ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652c0025ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 -- 172.21.15.38:0/2656366307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652bffbb280 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f568cad5640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5678009610 tx=0x7f5678045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58af7d99702b8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5670ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567802dd20 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f56780046c0 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5684004780 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5678060050 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5692201740 10 client.17406 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.308+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5652c002d1b0 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5684004780 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5678004ec0 con 0x5652bffb7520 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 10 client.17406 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 10 client.17406 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 20 client.17406 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652c002d8d0 0x5652c002fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 4 client.17406 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5652bff19dc0 con 0x5652c002d8d0 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f568d2d6640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652c002d8d0 0x5652c002fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f5670ff9640 10 client.17406 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.309+0000 7f568d2d6640 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652c002d8d0 0x5652c002fcc0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f2aa46b247699226 server_cookie=440dc82bb96b6b7a in_seq=0 out_seq=0 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5670ff9640 1 -- 172.21.15.38:0/2656366307 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5652bff19dc0 con 0x5652c002d8d0 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5670ff9640 10 client.17406 handle_command_reply: tid=0 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 client.17406 shutdown 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 msgr2=0x5652bffbaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5678009610 tx=0x7f5678045040 comp rx=0 tx=0).stop 2024-04-22T00:07:33.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 shutdown_connections 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5652c002d8d0 0x5652c002fcc0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5652bffb7520 0x5652bffbaa80 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652bffae940 0x5652c0026010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5670ff9640 0 client.17406 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 --2- 172.21.15.38:0/2656366307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5652bffae020 0x5652c0025ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 >> 172.21.15.38:0/2656366307 conn(0x5652bffa8750 msgr2=0x5652bffa7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 shutdown_connections 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 1 -- 172.21.15.38:0/2656366307 wait complete. 2024-04-22T00:07:33.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.310+0000 7f5692201740 20 client.17406 trim_cache size 0 max 16384 2024-04-22T00:07:33.315 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:07:33.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 msgr2=0x7f56740a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 0x7f56740a5240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f568009b310 tx=0x7f5680004020 comp rx=0 tx=0).stop 2024-04-22T00:07:33.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 msgr2=0x7f568819c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5680009fd0 tx=0x7f568000b260 comp rx=0 tx=0).stop 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 shutdown_connections 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f56740a2d80 0x7f56740a5240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f568810bd00 0x7f568819c910 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56881032e0 0x7f5688198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 --2- 172.21.15.38:0/2186049882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56881029c0 0x7f5688198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 >> 172.21.15.38:0/2186049882 conn(0x7f5688077740 msgr2=0x7f56880fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 shutdown_connections 2024-04-22T00:07:33.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.311+0000 7f568f53a640 1 -- 172.21.15.38:0/2186049882 wait complete. 2024-04-22T00:07:33.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:07:33.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 -- 172.21.15.38:0/2830356099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 msgr2=0x7f9fe8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 --2- 172.21.15.38:0/2830356099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8102da0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009900 tx=0x7f9fd802bd60 comp rx=0 tx=0).stop 2024-04-22T00:07:33.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 -- 172.21.15.38:0/2830356099 shutdown_connections 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 --2- 172.21.15.38:0/2830356099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 --2- 172.21.15.38:0/2830356099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 0x7f9fe810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 --2- 172.21.15.38:0/2830356099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8102da0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 -- 172.21.15.38:0/2830356099 >> 172.21.15.38:0/2830356099 conn(0x7f9fe8077740 msgr2=0x7f9fe8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.455+0000 7f9fee1be640 1 -- 172.21.15.38:0/2830356099 shutdown_connections 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 -- 172.21.15.38:0/2830356099 wait complete. 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 Processor -- start 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 -- start start 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 0x7f9fe8100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe8101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8111b50 con 0x7f9fe81029c0 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81119d0 con 0x7f9fe810bd00 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.456+0000 7f9fee1be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe8111cd0 con 0x7f9fe81032e0 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36988/0 (socket says 172.21.15.38:36988) 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 -- 172.21.15.38:0/1814100560 learned_addr learned my addr 172.21.15.38:0/1814100560 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 0x7f9fe8100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 -- 172.21.15.38:0/1814100560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 msgr2=0x7f9fe8100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 0x7f9fe8100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 -- 172.21.15.38:0/1814100560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 msgr2=0x7f9fe8101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe8101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 -- 172.21.15.38:0/1814100560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe81a64d0 con 0x7f9fe81029c0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009a20 tx=0x7f9fd802dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe4ff9640 1 -- 172.21.15.38:0/1814100560 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8043020 con 0x7f9fe81029c0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe4ff9640 1 -- 172.21.15.38:0/1814100560 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd80054e0 con 0x7f9fe81029c0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe81a6760 con 0x7f9fe81029c0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fe4ff9640 1 -- 172.21.15.38:0/1814100560 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8003bd0 con 0x7f9fe81029c0 2024-04-22T00:07:33.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.457+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81a6cb0 con 0x7f9fe81029c0 2024-04-22T00:07:33.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.459+0000 7f9fe4ff9640 1 -- 172.21.15.38:0/1814100560 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9fd8046020 con 0x7f9fe81029c0 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 Processor -- start 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 -- start start 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc82400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 0x55768bc8afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc8d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55768bbe6530 con 0x55768bc82940 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55768bb8e830 con 0x55768bc8b520 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55768bbe24f0 con 0x55768bc82020 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 0x55768bc8afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc82400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc82400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:56748/0 (socket says 172.21.15.38:56748) 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe7fff640 1 -- 172.21.15.38:0/2315697921 learned_addr learned my addr 172.21.15.38:0/2315697921 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc8d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe4ff9640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 0x7f9fd00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.460+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 0x7f9fd00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 -- 172.21.15.38:0/2315697921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 msgr2=0x55768bc82400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc82400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 -- 172.21.15.38:0/2315697921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 msgr2=0x55768bc8afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 0x55768bc8afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 -- 172.21.15.38:0/2315697921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55768bbb22c0 con 0x55768bc8b520 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe4ff9640 1 -- 172.21.15.38:0/1814100560 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9fd80f1770 con 0x7f9fe81029c0 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 0x55768bc8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc8d910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00bbc0 tx=0x7f9fdc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe2f9295d1eb20b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 0x7f9fd00a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005c40 tx=0x7f9fe0004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fcdffb640 1 -- 172.21.15.38:0/2315697921 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc004030 con 0x55768bc8b520 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fcdffb640 1 -- 172.21.15.38:0/2315697921 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc0041d0 con 0x55768bc8b520 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.461+0000 7f9fcdffb640 1 -- 172.21.15.38:0/2315697921 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc011640 con 0x55768bc8b520 2024-04-22T00:07:33.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- 172.21.15.38:0/2315697921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 msgr2=0x55768bc8d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc8d910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00bbc0 tx=0x7f9fdc0107a0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- 172.21.15.38:0/2315697921 shutdown_connections 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc8d910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82940 0x55768bc8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/2315697921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc82400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- 172.21.15.38:0/2315697921 >> 172.21.15.38:0/2315697921 conn(0x55768bc7c750 msgr2=0x55768bc7cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- 172.21.15.38:0/2315697921 shutdown_connections 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- 172.21.15.38:0/2315697921 wait complete. 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- start start 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 Processor -- start 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc88270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc88880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55768bc8e310 con 0x55768bc8b520 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55768bba1f10 con 0x55768bc82020 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55768bc8e490 con 0x55768bc82940 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.462+0000 7f9ff0daf740 10 client.0 _ll_register_callbacks cb 0x55768b8f9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc88880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:35482/0 (socket says 172.21.15.38:35482) 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 -- 172.21.15.38:0/889780529 learned_addr learned my addr 172.21.15.38:0/889780529 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc88270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 -- 172.21.15.38:0/889780529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 msgr2=0x55768bc88270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc88270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 -- 172.21.15.38:0/889780529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 msgr2=0x55768bc88880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc88880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 -- 172.21.15.38:0/889780529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55768bb73c80 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc88270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8000c00 tx=0x7f9fd8005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde5af7e6d35460b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80bd5a0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fdc0041d0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9fd80ba0d0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9fd80bd3c0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9ff0daf740 10 client.26786 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.463+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55768b8fa3f0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc0041d0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9fd80f13b0 con 0x55768bc82020 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 10 client.26786 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 10 client.26786 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:07:33.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 20 client.26786 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bc9b460 0x55768bc9d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 4 client.26786 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:33.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.464+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55768bbf3a70 con 0x55768bc9b460 2024-04-22T00:07:33.467 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:33 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2729558444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:07:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.465+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bc9b460 0x55768bc9d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.465+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bc9b460 0x55768bc9d8d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=beca1c6a12f9872e server_cookie=226abb1a90994bc3 in_seq=0 out_seq=0 2024-04-22T00:07:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.465+0000 7f9fc6ffd640 10 client.26786 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:07:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/889780529 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55768bbf3a70 con 0x55768bc9b460 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9fc6ffd640 10 client.26786 handle_command_reply: tid=0 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 client.26786 shutdown 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 msgr2=0x55768bc8aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8000c00 tx=0x7f9fd8005b50 comp rx=0 tx=0).stop 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 shutdown_connections 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bc9b460 0x55768bc9d8d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.466+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc88880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9fc6ffd640 0 client.26786 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc88270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/889780529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82020 0x55768bc8aee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 >> 172.21.15.38:0/889780529 conn(0x55768bc7c750 msgr2=0x55768bc7e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 shutdown_connections 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 1 -- 172.21.15.38:0/889780529 wait complete. 2024-04-22T00:07:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.467+0000 7f9ff0daf740 20 client.26786 trim_cache size 0 max 16384 2024-04-22T00:07:33.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 Processor -- start 2024-04-22T00:07:33.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 -- start start 2024-04-22T00:07:33.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82940 0x55768bc8a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81b0bf0 con 0x55768bc8b520 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81b2730 con 0x55768bc82940 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81b12c0 con 0x55768bc82020 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37020/0 (socket says 172.21.15.38:37020) 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 -- 172.21.15.38:0/66778457 learned_addr learned my addr 172.21.15.38:0/66778457 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82940 0x55768bc8a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 -- 172.21.15.38:0/66778457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 msgr2=0x55768bc971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 -- 172.21.15.38:0/66778457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82940 msgr2=0x55768bc8a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82940 0x55768bc8a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 -- 172.21.15.38:0/66778457 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55768b9c47d0 con 0x55768bc8b520 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.481+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ffc0 tx=0x7f9fe0015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8078fa010ae9169d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9fcd7fa640 1 -- 172.21.15.38:0/66778457 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0004b20 con 0x55768bc8b520 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9fcd7fa640 1 -- 172.21.15.38:0/66778457 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0004cc0 con 0x55768bc8b520 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9fcd7fa640 1 -- 172.21.15.38:0/66778457 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe002a6e0 con 0x55768bc8b520 2024-04-22T00:07:33.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 -- 172.21.15.38:0/66778457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 msgr2=0x55768bc8aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ffc0 tx=0x7f9fe0015070 comp rx=0 tx=0).stop 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 -- 172.21.15.38:0/66778457 shutdown_connections 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc8b520 0x55768bc8aee0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc82940 0x55768bc8a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/66778457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82020 0x55768bc971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.482+0000 7f9ff0daf740 1 -- 172.21.15.38:0/66778457 >> 172.21.15.38:0/66778457 conn(0x55768bc7c750 msgr2=0x55768bc7b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- 172.21.15.38:0/66778457 shutdown_connections 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- 172.21.15.38:0/66778457 wait complete. 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- start start 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 Processor -- start 2024-04-22T00:07:33.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc85360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc858a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81b2060 con 0x55768bc82020 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81b1990 con 0x55768bc8b520 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe81aebd0 con 0x55768bc82940 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9ff0daf740 10 client.0 _ll_register_callbacks cb 0x7f9fd80bd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.483+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37022/0 (socket says 172.21.15.38:37022) 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 -- 172.21.15.38:0/4259620233 learned_addr learned my addr 172.21.15.38:0/4259620233 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc85360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe7fff640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc858a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 -- 172.21.15.38:0/4259620233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 msgr2=0x55768bc85360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc85360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 -- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 msgr2=0x55768bc858a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc858a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.484+0000 7f9fe77fe640 1 -- 172.21.15.38:0/4259620233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55768bd009e0 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.485+0000 7f9fe77fe640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80bd9f0 tx=0x7f9fd80045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f89a8af4f1d55b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.485+0000 7f9fc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.485+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80c6440 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.485+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f9fd80bceb0 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.485+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fe0004cc0 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9fd8005690 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 10 client.17424 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55768bd01c80 con 0x55768bc82020 2024-04-22T00:07:33.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0004cc0 con 0x55768bc82020 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9fd80a80b0 con 0x55768bc82020 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 10 client.17424 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 10 client.17424 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 20 client.17424 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bd022a0 0x55768bd046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 4 client.17424 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55768bbeddc0 con 0x55768bd022a0 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.486+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bd022a0 0x55768bd046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.487+0000 7f9fc6ffd640 10 client.17424 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.487+0000 7f9fe6ffd640 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bd022a0 0x55768bd046c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6a0c8aab12974b74 server_cookie=cbff1f8684e4521e in_seq=0 out_seq=0 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9fc6ffd640 1 -- 172.21.15.38:0/4259620233 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55768bbeddc0 con 0x55768bd022a0 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9fc6ffd640 10 client.17424 handle_command_reply: tid=0 2024-04-22T00:07:33.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 client.17424 shutdown 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 msgr2=0x55768bc84e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80bd9f0 tx=0x7f9fd80045f0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 shutdown_connections 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55768bc8b520 0x55768bc858a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55768bc82940 0x55768bc85360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55768bd022a0 0x55768bd046c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 --2- 172.21.15.38:0/4259620233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55768bc82020 0x55768bc84e20 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9fc6ffd640 0 client.17424 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 >> 172.21.15.38:0/4259620233 conn(0x55768bc7c750 msgr2=0x55768bc7da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 shutdown_connections 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 1 -- 172.21.15.38:0/4259620233 wait complete. 2024-04-22T00:07:33.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.488+0000 7f9ff0daf740 20 client.17424 trim_cache size 0 max 16384 2024-04-22T00:07:33.490 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 msgr2=0x7f9fd00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 0x7f9fd00a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0005c40 tx=0x7f9fe0004ab0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 msgr2=0x7f9fe8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009a20 tx=0x7f9fd802dd30 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 shutdown_connections 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9fd00a2d00 0x7f9fd00a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fe810bd00 0x7f9fe8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe81032e0 0x7f9fe8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 --2- 172.21.15.38:0/1814100560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fe81029c0 0x7f9fe8100610 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 >> 172.21.15.38:0/1814100560 conn(0x7f9fe8077740 msgr2=0x7f9fe80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 shutdown_connections 2024-04-22T00:07:33.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.489+0000 7f9fee1be640 1 -- 172.21.15.38:0/1814100560 wait complete. 2024-04-22T00:07:33.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 -- 172.21.15.38:0/511595618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100fd0 msgr2=0x7fb40010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 --2- 172.21.15.38:0/511595618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100fd0 0x7fb40010d4e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009cd0 tx=0x7fb3f802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 -- 172.21.15.38:0/511595618 shutdown_connections 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 --2- 172.21.15.38:0/511595618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100fd0 0x7fb40010d4e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 --2- 172.21.15.38:0/511595618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400100610 0x7fb400100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 --2- 172.21.15.38:0/511595618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4001066b0 0x7fb400106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 -- 172.21.15.38:0/511595618 >> 172.21.15.38:0/511595618 conn(0x7fb400077740 msgr2=0x7fb400077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 -- 172.21.15.38:0/511595618 shutdown_connections 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.610+0000 7fb408535640 1 -- 172.21.15.38:0/511595618 wait complete. 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 Processor -- start 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 -- start start 2024-04-22T00:07:33.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb400100fd0 0x7fb4001992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 0x7fb4001a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001109f0 con 0x7fb400100610 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb400110870 con 0x7fb4001066b0 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb408535640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb400110b70 con 0x7fb400100fd0 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb4062d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb4062d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37036/0 (socket says 172.21.15.38:37036) 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb4062d1640 1 -- 172.21.15.38:0/2183672665 learned_addr learned my addr 172.21.15.38:0/2183672665 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 0x7fb4001a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb405ad0640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb400100fd0 0x7fb4001992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 -- 172.21.15.38:0/2183672665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb400100fd0 msgr2=0x7fb4001992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb400100fd0 0x7fb4001992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 -- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 msgr2=0x7fb400198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb406ad2640 1 -- 172.21.15.38:0/2183672665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4001a4350 con 0x7fb4001066b0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.611+0000 7fb4062d1640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 0x7fb4001a3e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb3f802c630 tx=0x7fb3f8032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb3f77fe640 1 -- 172.21.15.38:0/2183672665 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8041020 con 0x7fb4001066b0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4001a4520 con 0x7fb4001066b0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb3f77fe640 1 -- 172.21.15.38:0/2183672665 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f8034cd0 con 0x7fb4001066b0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb3f77fe640 1 -- 172.21.15.38:0/2183672665 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8039040 con 0x7fb4001066b0 2024-04-22T00:07:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.612+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4001a4980 con 0x7fb4001066b0 2024-04-22T00:07:33.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.614+0000 7fb3f77fe640 1 -- 172.21.15.38:0/2183672665 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb3f8044020 con 0x7fb4001066b0 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.614+0000 7fb40b1f4740 1 Processor -- start 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 -- start start 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3e5f0 0x563312e4b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563312da2530 con 0x563312e3e5f0 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563312d4a830 con 0x563312e3dcd0 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563312d9e4f0 con 0x563312e4b930 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb3f77fe640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 0x7fb3ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb4062d1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3e5f0 0x563312e4b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:35522/0 (socket says 172.21.15.38:35522) 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 -- 172.21.15.38:0/1241059843 learned_addr learned my addr 172.21.15.38:0/1241059843 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 0x7fb3ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb3f77fe640 1 -- 172.21.15.38:0/2183672665 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb3f80f1380 con 0x7fb4001066b0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 -- 172.21.15.38:0/1241059843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 msgr2=0x563312e4dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 -- 172.21.15.38:0/1241059843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3e5f0 msgr2=0x563312e4b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3e5f0 0x563312e4b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.615+0000 7fb405ad0640 1 -- 172.21.15.38:0/1241059843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563312d6e2c0 con 0x563312e3dcd0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.616+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00058c0 tx=0x7fb3f0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98d5d6bb1ab42b5e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.616+0000 7fb406ad2640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 0x7fb3ec0a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80ee950 tx=0x7fb3f80095c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.616+0000 7fb3f4bfe640 1 -- 172.21.15.38:0/1241059843 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0004030 con 0x563312e3dcd0 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.616+0000 7fb3f4bfe640 1 -- 172.21.15.38:0/1241059843 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005500 con 0x563312e3dcd0 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.616+0000 7fb3f4bfe640 1 -- 172.21.15.38:0/1241059843 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0004d30 con 0x563312e3dcd0 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1241059843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 msgr2=0x563312e3e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00058c0 tx=0x7fb3f0005850 comp rx=0 tx=0).stop 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1241059843 shutdown_connections 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3e5f0 0x563312e4b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1241059843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3dcd0 0x563312e3e0b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1241059843 >> 172.21.15.38:0/1241059843 conn(0x563312e38750 msgr2=0x563312e38b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1241059843 shutdown_connections 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1241059843 wait complete. 2024-04-22T00:07:33.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 -- start start 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.617+0000 7fb40b1f4740 1 Processor -- start 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e4b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 0x563312e48650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 0x563312e48bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563312e4e770 con 0x563312e3dcd0 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563312d5df10 con 0x563312e4b930 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563312e4e8f0 con 0x563312e3e5f0 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 10 client.0 _ll_register_callbacks cb 0x563312ab5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e4b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 0x563312e48650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e4b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37056/0 (socket says 172.21.15.38:37056) 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb405ad0640 1 -- 172.21.15.38:0/1483091050 learned_addr learned my addr 172.21.15.38:0/1483091050 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 0x563312e48bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 -- 172.21.15.38:0/1483091050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 msgr2=0x563312e48bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 0x563312e48bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 -- 172.21.15.38:0/1483091050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 msgr2=0x563312e4b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e4b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 -- 172.21.15.38:0/1483091050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563312d2fc80 con 0x563312e3e5f0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 0x563312e48bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb4062d1640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 0x563312e48650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0058c0 tx=0x7fb3fc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b24087288becf27 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc012110 con 0x563312e3e5f0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f0005500 con 0x563312e3e5f0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.618+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb3fc004030 con 0x563312e3e5f0 2024-04-22T00:07:33.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc0122b0 con 0x563312e3e5f0 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 10 client.26659 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563312ab63f0 con 0x563312e3e5f0 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005500 con 0x563312e3e5f0 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb3fc010690 con 0x563312e3e5f0 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 10 client.26659 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 10 client.26659 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 20 client.26659 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312e5bb40 0x563312e5df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 4 client.26659 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563312dafa70 con 0x563312e5bb40 2024-04-22T00:07:33.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.619+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312e5bb40 0x563312e5df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb405ad0640 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312e5bb40 0x563312e5df30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27e16a4ae08dc6e3 server_cookie=91eff337f8d6a65 in_seq=0 out_seq=0 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb3e9ffb640 10 client.26659 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/1483091050 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563312dafa70 con 0x563312e5bb40 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb3e9ffb640 10 client.26659 handle_command_reply: tid=0 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 client.26659 shutdown 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 msgr2=0x563312e48650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 0x563312e48650 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0058c0 tx=0x7fb3fc00f790 comp rx=0 tx=0).stop 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 shutdown_connections 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312e5bb40 0x563312e5df30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e4b930 0x563312e48bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e3e5f0 0x563312e48650 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/1483091050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e4b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb3e9ffb640 0 client.26659 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.620+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 >> 172.21.15.38:0/1483091050 conn(0x563312e38750 msgr2=0x563312e3a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.621+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 shutdown_connections 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.621+0000 7fb40b1f4740 1 -- 172.21.15.38:0/1483091050 wait complete. 2024-04-22T00:07:33.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.621+0000 7fb40b1f4740 20 client.26659 trim_cache size 0 max 16384 2024-04-22T00:07:33.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.634+0000 7fb40b1f4740 1 Processor -- start 2024-04-22T00:07:33.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 -- start start 2024-04-22T00:07:33.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ac7c0 con 0x563312e3dcd0 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ae300 con 0x563312e3e5f0 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ace90 con 0x563312e4b930 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37058/0 (socket says 172.21.15.38:37058) 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 -- 172.21.15.38:0/2461420420 learned_addr learned my addr 172.21.15.38:0/2461420420 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb4062d1640 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 -- 172.21.15.38:0/2461420420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 msgr2=0x563312e4b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 -- 172.21.15.38:0/2461420420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 msgr2=0x563312e4ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 -- 172.21.15.38:0/2461420420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563312b807d0 con 0x563312e3dcd0 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb405ad0640 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0004920 tx=0x7fb3f0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3772f336d9b02f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb3ebfff640 1 -- 172.21.15.38:0/2461420420 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f002e360 con 0x563312e3dcd0 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.635+0000 7fb3ebfff640 1 -- 172.21.15.38:0/2461420420 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f000f8c0 con 0x563312e3dcd0 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb3ebfff640 1 -- 172.21.15.38:0/2461420420 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f001f6c0 con 0x563312e3dcd0 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- 172.21.15.38:0/2461420420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 msgr2=0x563312e576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0004920 tx=0x7fb3f0009fd0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- 172.21.15.38:0/2461420420 shutdown_connections 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/2461420420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e576d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- 172.21.15.38:0/2461420420 >> 172.21.15.38:0/2461420420 conn(0x563312e38750 msgr2=0x563312e3a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- 172.21.15.38:0/2461420420 shutdown_connections 2024-04-22T00:07:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- 172.21.15.38:0/2461420420 wait complete. 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.636+0000 7fb40b1f4740 1 -- start start 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 Processor -- start 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001adc30 con 0x563312e3dcd0 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ad560 con 0x563312e3e5f0 2024-04-22T00:07:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001aa7a0 con 0x563312e4b930 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb40b1f4740 10 client.0 _ll_register_callbacks cb 0x7fb3fc009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37074/0 (socket says 172.21.15.38:37074) 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 -- 172.21.15.38:0/525528007 learned_addr learned my addr 172.21.15.38:0/525528007 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb4062d1640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 -- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 msgr2=0x563312e4f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 -- 172.21.15.38:0/525528007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 msgr2=0x563312e4f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 -- 172.21.15.38:0/525528007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563312e501f0 con 0x563312e3dcd0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb4062d1640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb405ad0640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00059f0 tx=0x7fb3f0009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945fd1196bb4e19c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.637+0000 7fb3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0032070 con 0x563312e3dcd0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f000f8c0 con 0x563312e3dcd0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb3f000fa70 con 0x563312e3dcd0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb3f0011320 con 0x563312e3dcd0 2024-04-22T00:07:33.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 10 client.17442 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563312ec2140 con 0x563312e3dcd0 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f000f8c0 con 0x563312e3dcd0 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb3f0037030 con 0x563312e3dcd0 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 10 client.17442 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 10 client.17442 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 20 client.17442 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312ec2570 0x563312ec4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 4 client.17442 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563312da9dc0 con 0x563312ec2570 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.638+0000 7fb4062d1640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312ec2570 0x563312ec4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb4062d1640 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312ec2570 0x563312ec4960 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=644e0ed5f0297afe server_cookie=c3cd9916cec81d56 in_seq=0 out_seq=0 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb3e9ffb640 10 client.17442 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:07:33.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb3e9ffb640 1 -- 172.21.15.38:0/525528007 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563312da9dc0 con 0x563312ec2570 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb3e9ffb640 10 client.17442 handle_command_reply: tid=0 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 client.17442 shutdown 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 msgr2=0x563312e54070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00059f0 tx=0x7fb3f0009a90 comp rx=0 tx=0).stop 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 shutdown_connections 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x563312ec2570 0x563312ec4960 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563312e4b930 0x563312e4f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563312e3e5f0 0x563312e4f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 --2- 172.21.15.38:0/525528007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563312e3dcd0 0x563312e54070 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb3e9ffb640 0 client.17442 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:07:33.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 >> 172.21.15.38:0/525528007 conn(0x563312e38750 msgr2=0x563312e36fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 shutdown_connections 2024-04-22T00:07:33.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 1 -- 172.21.15.38:0/525528007 wait complete. 2024-04-22T00:07:33.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.639+0000 7fb40b1f4740 20 client.17442 trim_cache size 0 max 16384 2024-04-22T00:07:33.647 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:07:33.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 msgr2=0x7fb3ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 0x7fb3ec0a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80ee950 tx=0x7fb3f80095c0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 msgr2=0x7fb4001a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 0x7fb4001a3e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb3f802c630 tx=0x7fb3f8032f20 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 shutdown_connections 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb3ec0a2d00 0x7fb3ec0a51c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4001066b0 0x7fb4001a3e10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb400100fd0 0x7fb4001992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 --2- 172.21.15.38:0/2183672665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb400100610 0x7fb400198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 >> 172.21.15.38:0/2183672665 conn(0x7fb400077740 msgr2=0x7fb4000ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 shutdown_connections 2024-04-22T00:07:33.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.640+0000 7fb408535640 1 -- 172.21.15.38:0/2183672665 wait complete. 2024-04-22T00:07:33.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:07:33.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:33 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2729558444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 -- 172.21.15.38:0/3466037849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 msgr2=0x7f3aac10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/3466037849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac10b7c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009900 tx=0x7f3a9402bd60 comp rx=0 tx=0).stop 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 -- 172.21.15.38:0/3466037849 shutdown_connections 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/3466037849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 0x7f3aac10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/3466037849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac10b7c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/3466037849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 0x7f3aac102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 -- 172.21.15.38:0/3466037849 >> 172.21.15.38:0/3466037849 conn(0x7f3aac077740 msgr2=0x7f3aac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 -- 172.21.15.38:0/3466037849 shutdown_connections 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.817+0000 7f3ab16fb640 1 -- 172.21.15.38:0/3466037849 wait complete. 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 Processor -- start 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 -- start start 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 0x7f3aac198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 0x7f3aac19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac111b50 con 0x7f3aac1032e0 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1119d0 con 0x7f3aac10bd00 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3ab16fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac111cd0 con 0x7f3aac1029c0 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37098/0 (socket says 172.21.15.38:37098) 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3aa3fff640 1 -- 172.21.15.38:0/167013471 learned_addr learned my addr 172.21.15.38:0/167013471 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.818+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 0x7f3aac19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 0x7f3aac198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 -- 172.21.15.38:0/167013471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 msgr2=0x7f3aac198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 0x7f3aac198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 -- 172.21.15.38:0/167013471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 msgr2=0x7f3aac19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 0x7f3aac19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 -- 172.21.15.38:0/167013471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac19d060 con 0x7f3aac1032e0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009a20 tx=0x7f3a9402dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa8ff9640 1 -- 172.21.15.38:0/167013471 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94043020 con 0x7f3aac1032e0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa8ff9640 1 -- 172.21.15.38:0/167013471 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a940054e0 con 0x7f3aac1032e0 2024-04-22T00:07:33.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aac19d2f0 con 0x7f3aac1032e0 2024-04-22T00:07:33.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3aa8ff9640 1 -- 172.21.15.38:0/167013471 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94003bd0 con 0x7f3aac1032e0 2024-04-22T00:07:33.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.819+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aac1a4be0 con 0x7f3aac1032e0 2024-04-22T00:07:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.821+0000 7f3aa8ff9640 1 -- 172.21.15.38:0/167013471 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3a94046020 con 0x7f3aac1032e0 2024-04-22T00:07:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 Processor -- start 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 -- start start 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b3f1124530 con 0x55b3f11c0940 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b3f10cc830 con 0x55b3f11c9520 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3f11204f0 con 0x55b3f11c0020 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aaaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aaaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37102/0 (socket says 172.21.15.38:37102) 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aaaffd640 1 -- 172.21.15.38:0/1019750941 learned_addr learned my addr 172.21.15.38:0/1019750941 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aa8ff9640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 0x7f3a980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.822+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 0x7f3a980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 -- 172.21.15.38:0/1019750941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 msgr2=0x55b3f11c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 -- 172.21.15.38:0/1019750941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 msgr2=0x55b3f11cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 -- 172.21.15.38:0/1019750941 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3f10f02c0 con 0x55b3f11c0940 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aa8ff9640 1 -- 172.21.15.38:0/167013471 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3a940f17e0 con 0x7f3aac1032e0 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009c90 tx=0x7f3a9c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2521f83b9dd8dd16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 0x7f3a980a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005fd0 tx=0x7f3aa4004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aa17fa640 1 -- 172.21.15.38:0/1019750941 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c012450 con 0x55b3f11c0940 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aa17fa640 1 -- 172.21.15.38:0/1019750941 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a9c012ad0 con 0x55b3f11c0940 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.823+0000 7f3aa17fa640 1 -- 172.21.15.38:0/1019750941 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c00fc30 con 0x55b3f11c0940 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- 172.21.15.38:0/1019750941 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 msgr2=0x55b3f11c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009c90 tx=0x7f3a9c0117b0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- 172.21.15.38:0/1019750941 shutdown_connections 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c8fe0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/1019750941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- 172.21.15.38:0/1019750941 >> 172.21.15.38:0/1019750941 conn(0x55b3f11ba750 msgr2=0x55b3f11bab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- 172.21.15.38:0/1019750941 shutdown_connections 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- 172.21.15.38:0/1019750941 wait complete. 2024-04-22T00:07:33.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.824+0000 7f3ab43c3740 1 -- start start 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 Processor -- start 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b3f11d0720 con 0x55b3f11c0940 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b3f10dff10 con 0x55b3f11c9520 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3f11d08a0 con 0x55b3f11c0020 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37110/0 (socket says 172.21.15.38:37110) 2024-04-22T00:07:33.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 -- 172.21.15.38:0/3864099154 learned_addr learned my addr 172.21.15.38:0/3864099154 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3ab43c3740 10 client.0 _ll_register_callbacks cb 0x55b3f0e37be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 -- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 msgr2=0x55b3f11c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 -- 172.21.15.38:0/3864099154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 msgr2=0x55b3f11c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 -- 172.21.15.38:0/3864099154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3f10b1c80 con 0x55b3f11c0940 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3a940bdb80 tx=0x7f3a940bd9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a270774b8190d223 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.825+0000 7f3a927fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a940d4070 con 0x55b3f11c0940 2024-04-22T00:07:33.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3a940ba260 con 0x55b3f11c0940 2024-04-22T00:07:33.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9c012ad0 con 0x55b3f11c0940 2024-04-22T00:07:33.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3a94004c80 con 0x55b3f11c0940 2024-04-22T00:07:33.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3ab43c3740 10 client.17454 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b3f0e383f0 con 0x55b3f11c0940 2024-04-22T00:07:33.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.826+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a9c012ad0 con 0x55b3f11c0940 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3a940f1480 con 0x55b3f11c0940 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 10 client.17454 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 20 client.17454 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f11dfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 4 client.17454 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.827+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3f1131a70 con 0x55b3f11dd940 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.828+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f11dfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.828+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f11dfd70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c0fc5beeec0a71e server_cookie=b96c7c0e1cda1315 in_seq=0 out_seq=0 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.828+0000 7f3a927fc640 10 client.17454 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3a927fc640 1 -- 172.21.15.38:0/3864099154 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3f1131a70 con 0x55b3f11dd940 2024-04-22T00:07:33.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3a927fc640 10 client.17454 handle_command_reply: tid=0 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 client.17454 shutdown 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 msgr2=0x55b3f11c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3a940bdb80 tx=0x7f3a940bd9c0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 shutdown_connections 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f11dfd70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c9520 0x55b3f11c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c0940 0x55b3f11c61a0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3a927fc640 0 client.17454 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/3864099154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.829+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 >> 172.21.15.38:0/3864099154 conn(0x55b3f11ba750 msgr2=0x55b3f11bc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.830+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 shutdown_connections 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.830+0000 7f3ab43c3740 1 -- 172.21.15.38:0/3864099154 wait complete. 2024-04-22T00:07:33.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.830+0000 7f3ab43c3740 20 client.17454 trim_cache size 0 max 16384 2024-04-22T00:07:33.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:33 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2729558444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 Processor -- start 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 -- start start 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 0x55b3f11d9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0940 0x55b3f11b96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1ac820 con 0x55b3f11c9520 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1ae360 con 0x55b3f11c0020 2024-04-22T00:07:33.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.843+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1acef0 con 0x55b3f11c0940 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37120/0 (socket says 172.21.15.38:37120) 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 -- 172.21.15.38:0/2115076737 learned_addr learned my addr 172.21.15.38:0/2115076737 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0940 0x55b3f11b96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 0x55b3f11d9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 -- 172.21.15.38:0/2115076737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0940 msgr2=0x55b3f11b96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0940 0x55b3f11b96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 -- 172.21.15.38:0/2115076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 msgr2=0x55b3f11d9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 0x55b3f11d9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 -- 172.21.15.38:0/2115076737 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3f0f027d0 con 0x55b3f11c9520 2024-04-22T00:07:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 0x55b3f11d9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4012b50 tx=0x7f3aa4009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40a06bfcc5d4b6e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aa0ff9640 1 -- 172.21.15.38:0/2115076737 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa401a070 con 0x55b3f11c9520 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aa0ff9640 1 -- 172.21.15.38:0/2115076737 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa4026030 con 0x55b3f11c9520 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.844+0000 7f3aa0ff9640 1 -- 172.21.15.38:0/2115076737 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa40227d0 con 0x55b3f11c9520 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- 172.21.15.38:0/2115076737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 msgr2=0x55b3f11c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4012b50 tx=0x7f3aa4009ab0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- 172.21.15.38:0/2115076737 shutdown_connections 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11c88d0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0940 0x55b3f11b96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/2115076737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0020 0x55b3f11d9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- 172.21.15.38:0/2115076737 >> 172.21.15.38:0/2115076737 conn(0x55b3f11ba750 msgr2=0x55b3f11b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- 172.21.15.38:0/2115076737 shutdown_connections 2024-04-22T00:07:33.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- 172.21.15.38:0/2115076737 wait complete. 2024-04-22T00:07:33.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 -- start start 2024-04-22T00:07:33.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.845+0000 7f3ab43c3740 1 Processor -- start 2024-04-22T00:07:33.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11d5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 0x55b3f11d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1adc90 con 0x55b3f11c9520 2024-04-22T00:07:33.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1ad5c0 con 0x55b3f11c0940 2024-04-22T00:07:33.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac1aa800 con 0x55b3f11c0020 2024-04-22T00:07:33.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3ab43c3740 10 client.0 _ll_register_callbacks cb 0x7f3a940035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:33.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37124/0 (socket says 172.21.15.38:37124) 2024-04-22T00:07:33.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 0x55b3f11d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 -- 172.21.15.38:0/4282135492 learned_addr learned my addr 172.21.15.38:0/4282135492 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11d5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 -- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 msgr2=0x55b3f11d5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11d5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 -- 172.21.15.38:0/4282135492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 msgr2=0x55b3f11d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:33.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 0x55b3f11d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 -- 172.21.15.38:0/4282135492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3f11d1b80 con 0x55b3f11c9520 2024-04-22T00:07:33.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aa3fff640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 0x55b3f11d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3aab7fe640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005c40 tx=0x7f3aa4012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83204f86f00b226c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3a927fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:33.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.846+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa401a050 con 0x55b3f11c9520 2024-04-22T00:07:33.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f3aa4011520 con 0x55b3f11c9520 2024-04-22T00:07:33.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa4026030 con 0x55b3f11c9520 2024-04-22T00:07:33.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3aa4011720 con 0x55b3f11c9520 2024-04-22T00:07:33.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3ab43c3740 10 client.17460 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:33.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b3f1243c20 con 0x55b3f11c9520 2024-04-22T00:07:33.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.847+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa4026030 con 0x55b3f11c9520 2024-04-22T00:07:33.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3aa401d070 con 0x55b3f11c9520 2024-04-22T00:07:33.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 10 client.17460 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:33.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 10 client.17460 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:07:33.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 20 client.17460 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:33.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f1246310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 4 client.17460 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:07:33.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3f11c6a70 con 0x55b3f11dd940 2024-04-22T00:07:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f1246310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3a927fc640 10 client.17460 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:07:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.848+0000 7f3aaaffd640 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f1246310 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=332006442a775365 server_cookie=bda68a350ae023b1 in_seq=0 out_seq=0 2024-04-22T00:07:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3a927fc640 1 -- 172.21.15.38:0/4282135492 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3f11c6a70 con 0x55b3f11dd940 2024-04-22T00:07:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3a927fc640 10 client.17460 handle_command_reply: tid=0 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 client.17460 shutdown 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 msgr2=0x55b3f11d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005c40 tx=0x7f3aa4012920 comp rx=0 tx=0).stop 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 shutdown_connections 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55b3f11dd940 0x55b3f1246310 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3f11c9520 0x55b3f11d1640 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3a927fc640 0 client.17460 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:07:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3f11c0940 0x55b3f11d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 --2- 172.21.15.38:0/4282135492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3f11c0020 0x55b3f11d5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 >> 172.21.15.38:0/4282135492 conn(0x55b3f11ba750 msgr2=0x55b3f11b9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 shutdown_connections 2024-04-22T00:07:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 1 -- 172.21.15.38:0/4282135492 wait complete. 2024-04-22T00:07:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab43c3740 20 client.17460 trim_cache size 0 max 16384 2024-04-22T00:07:33.863 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 msgr2=0x7f3a980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 0x7f3a980a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4005fd0 tx=0x7f3aa4004ab0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 msgr2=0x7f3aac1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.849+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009a20 tx=0x7f3a9402dd30 comp rx=0 tx=0).stop 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 shutdown_connections 2024-04-22T00:07:33.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3a980a2d00 0x7f3a980a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aac10bd00 0x7f3aac19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3aac1032e0 0x7f3aac1985b0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 --2- 172.21.15.38:0/167013471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3aac1029c0 0x7f3aac198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 >> 172.21.15.38:0/167013471 conn(0x7f3aac077740 msgr2=0x7f3aac0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 shutdown_connections 2024-04-22T00:07:33.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.850+0000 7f3ab16fb640 1 -- 172.21.15.38:0/167013471 wait complete. 2024-04-22T00:07:33.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 -- 172.21.15.38:0/1101669068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60881032e0 msgr2=0x7f608810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 --2- 172.21.15.38:0/1101669068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60881032e0 0x7f608810b7c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6078009900 tx=0x7f607802bd60 comp rx=0 tx=0).stop 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 -- 172.21.15.38:0/1101669068 shutdown_connections 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 --2- 172.21.15.38:0/1101669068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f608810bd00 0x7f608810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 --2- 172.21.15.38:0/1101669068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f60881032e0 0x7f608810b7c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 --2- 172.21.15.38:0/1101669068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881029c0 0x7f6088102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.994+0000 7f608f42e640 1 -- 172.21.15.38:0/1101669068 >> 172.21.15.38:0/1101669068 conn(0x7f6088077740 msgr2=0x7f6088077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- 172.21.15.38:0/1101669068 shutdown_connections 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- 172.21.15.38:0/1101669068 wait complete. 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 Processor -- start 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- start start 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 0x7f6088198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 0x7f6088198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6088111b50 con 0x7f608810bd00 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60881119d0 con 0x7f60881029c0 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.995+0000 7f608f42e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6088111cd0 con 0x7f60881032e0 2024-04-22T00:07:33.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608c9c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 0x7f6088198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608c9c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 0x7f6088198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45096/0 (socket says 172.21.15.38:45096) 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37134/0 (socket says 172.21.15.38:37134) 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608c9c9640 1 -- 172.21.15.38:0/3365603657 learned_addr learned my addr 172.21.15.38:0/3365603657 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 0x7f6088198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 -- 172.21.15.38:0/3365603657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 msgr2=0x7f6088198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 0x7f6088198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 -- 172.21.15.38:0/3365603657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 msgr2=0x7f6088198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 0x7f6088198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 -- 172.21.15.38:0/3365603657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608819d000 con 0x7f608810bd00 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 0x7f6088198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:33.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608d9cb640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f608000e970 tx=0x7f608000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:33.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f607e7fc640 1 -- 172.21.15.38:0/3365603657 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608000ccc0 con 0x7f608810bd00 2024-04-22T00:07:33.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608819d2f0 con 0x7f608810bd00 2024-04-22T00:07:33.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f607e7fc640 1 -- 172.21.15.38:0/3365603657 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080005020 con 0x7f608810bd00 2024-04-22T00:07:33.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f607e7fc640 1 -- 172.21.15.38:0/3365603657 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080004a00 con 0x7f608810bd00 2024-04-22T00:07:33.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.996+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60881a4b80 con 0x7f608810bd00 2024-04-22T00:07:34.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.999+0000 7f607e7fc640 1 -- 172.21.15.38:0/3365603657 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f60800051d0 con 0x7f608810bd00 2024-04-22T00:07:34.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.999+0000 7f6092014740 1 Processor -- start 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:33.999+0000 7f6092014740 1 -- start start 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febee400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febf9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5640feb52530 con 0x5640febee940 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5640feafa830 con 0x5640febf7520 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f6092014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5640feb4e4f0 con 0x5640febee020 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f607e7fc640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 0x7f60740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37148/0 (socket says 172.21.15.38:37148) 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 -- 172.21.15.38:0/1995468989 learned_addr learned my addr 172.21.15.38:0/1995468989 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608c9c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608c9c9640 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febf9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d9cb640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 0x7f60740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f607e7fc640 1 -- 172.21.15.38:0/3365603657 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f60800cf6c0 con 0x7f608810bd00 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 -- 172.21.15.38:0/1995468989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 msgr2=0x5640febee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febee400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 -- 172.21.15.38:0/1995468989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 msgr2=0x5640febf9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febf9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608d1ca640 1 -- 172.21.15.38:0/1995468989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640feb1e2c0 con 0x5640febee940 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.000+0000 7f608c9c9640 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febf9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.001+0000 7f608d1ca640 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6084011860 tx=0x7f6084011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c2abbb14291aa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.001+0000 7f608d9cb640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 0x7f60740a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60800ccb80 tx=0x7f60800c6430 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.001+0000 7f60737fe640 1 -- 172.21.15.38:0/1995468989 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60840112f0 con 0x5640febee940 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.001+0000 7f60737fe640 1 -- 172.21.15.38:0/1995468989 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6084003e50 con 0x5640febee940 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.001+0000 7f60737fe640 1 -- 172.21.15.38:0/1995468989 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60840044f0 con 0x5640febee940 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- 172.21.15.38:0/1995468989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 msgr2=0x5640febf6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f6084011860 tx=0x7f6084011d20 comp rx=0 tx=0).stop 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- 172.21.15.38:0/1995468989 shutdown_connections 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febf9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf6fe0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 --2- 172.21.15.38:0/1995468989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- 172.21.15.38:0/1995468989 >> 172.21.15.38:0/1995468989 conn(0x5640febe8750 msgr2=0x5640febe8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- 172.21.15.38:0/1995468989 shutdown_connections 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- 172.21.15.38:0/1995468989 wait complete. 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 -- start start 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.002+0000 7f6092014740 1 Processor -- start 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 0x5640febf6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5640febfa310 con 0x5640febee940 2024-04-22T00:07:34.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5640feb0df10 con 0x5640febee020 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5640febfa490 con 0x5640febf7520 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6092014740 10 client.0 _ll_register_callbacks cb 0x5640fe865be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37150/0 (socket says 172.21.15.38:37150) 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 -- 172.21.15.38:0/2355201697 learned_addr learned my addr 172.21.15.38:0/2355201697 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608c9c9640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 0x5640febf6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608c9c9640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 -- 172.21.15.38:0/2355201697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 msgr2=0x5640febf4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 -- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 msgr2=0x5640febf6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 0x5640febf6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 -- 172.21.15.38:0/2355201697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640feadfc80 con 0x5640febee940 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608c9c9640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 0x5640febf6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:34.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f608d1ca640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f608400a010 tx=0x7f6084003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82d0923e8b56e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6070ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.003+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608402a060 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f608400fdb0 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6084003e50 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6084031050 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 10 client.17472 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5640fe8663f0 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6084003e50 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f608401a870 con 0x5640febee940 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 10 client.17472 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 20 client.17472 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec07460 0x5640fec09850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 4 client.17472 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.004+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640feb5fa70 con 0x5640fec07460 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.005+0000 7f608c9c9640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec07460 0x5640fec09850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.005+0000 7f6070ff9640 10 client.17472 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.005+0000 7f608c9c9640 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec07460 0x5640fec09850 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac2149bfca5f0d9b server_cookie=da04942c5d290a7c in_seq=0 out_seq=0 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6070ff9640 1 -- 172.21.15.38:0/2355201697 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640feb5fa70 con 0x5640fec07460 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6070ff9640 10 client.17472 handle_command_reply: tid=0 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 client.17472 shutdown 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 msgr2=0x5640febf4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f608400a010 tx=0x7f6084003c20 comp rx=0 tx=0).stop 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 shutdown_connections 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec07460 0x5640fec09850 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febf4280 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 --2- 172.21.15.38:0/2355201697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee020 0x5640febf6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6070ff9640 0 client.17472 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:34.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 >> 172.21.15.38:0/2355201697 conn(0x5640febe8750 msgr2=0x5640febea4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:34.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 shutdown_connections 2024-04-22T00:07:34.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 1 -- 172.21.15.38:0/2355201697 wait complete. 2024-04-22T00:07:34.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.006+0000 7f6092014740 20 client.17472 trim_cache size 0 max 16384 2024-04-22T00:07:34.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 Processor -- start 2024-04-22T00:07:34.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 -- start start 2024-04-22T00:07:34.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee940 0x5640febea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f60881aea80 con 0x5640febee020 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60881b05c0 con 0x5640febee940 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.023+0000 7f6092014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f60881af150 con 0x5640febf7520 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37164/0 (socket says 172.21.15.38:37164) 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 -- 172.21.15.38:0/3258805391 learned_addr learned my addr 172.21.15.38:0/3258805391 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee940 0x5640febea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 -- 172.21.15.38:0/3258805391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 msgr2=0x5640febf69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 -- 172.21.15.38:0/3258805391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee940 msgr2=0x5640febea270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee940 0x5640febea270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 -- 172.21.15.38:0/3258805391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640febf4c00 con 0x5640febee020 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f607802c950 tx=0x7f6078009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95179b9f2d77ff15 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.024+0000 7f6072ffd640 1 -- 172.21.15.38:0/3258805391 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607804ebd0 con 0x5640febee020 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6072ffd640 1 -- 172.21.15.38:0/3258805391 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6078052030 con 0x5640febee020 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6072ffd640 1 -- 172.21.15.38:0/3258805391 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607804e2c0 con 0x5640febee020 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 -- 172.21.15.38:0/3258805391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 msgr2=0x5640fec031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f607802c950 tx=0x7f6078009260 comp rx=0 tx=0).stop 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 -- 172.21.15.38:0/3258805391 shutdown_connections 2024-04-22T00:07:34.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febf7520 0x5640febf69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febee940 0x5640febea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 --2- 172.21.15.38:0/3258805391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee020 0x5640fec031d0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 -- 172.21.15.38:0/3258805391 >> 172.21.15.38:0/3258805391 conn(0x5640febe8750 msgr2=0x5640febe9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 -- 172.21.15.38:0/3258805391 shutdown_connections 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.025+0000 7f6092014740 1 -- 172.21.15.38:0/3258805391 wait complete. 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 -- start start 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 Processor -- start 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febfb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f60881afef0 con 0x5640febee940 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f60881af820 con 0x5640febf7520 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f60881aca60 con 0x5640febee020 2024-04-22T00:07:34.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f6092014740 10 client.0 _ll_register_callbacks cb 0x7f60840093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37168/0 (socket says 172.21.15.38:37168) 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 -- 172.21.15.38:0/3453113090 learned_addr learned my addr 172.21.15.38:0/3453113090 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febfb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 -- 172.21.15.38:0/3453113090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 msgr2=0x5640febff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febff7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 -- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 msgr2=0x5640febfb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febfb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608d1ca640 1 -- 172.21.15.38:0/3453113090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640fe9307d0 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.026+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febfb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f608d1ca640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x5640febf5850 tx=0x7f6084011400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47785c38bffce1bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608402a040 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f608400fdb0 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6078052030 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f608401ae60 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 10 client.17478 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5640fec6d450 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6078052030 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f60840052f0 con 0x5640febee940 2024-04-22T00:07:34.030 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 10 client.17478 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 20 client.17478 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec6dbf0 0x5640fec70010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 4 client.17478 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.027+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640feb59dc0 con 0x5640fec6dbf0 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec6dbf0 0x5640fec70010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6070ff9640 10 client.17478 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f608c9c9640 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec6dbf0 0x5640fec70010 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fc116f85c9325bd server_cookie=8591a80b251a88bb in_seq=0 out_seq=0 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6070ff9640 1 -- 172.21.15.38:0/3453113090 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640feb59dc0 con 0x5640fec6dbf0 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6070ff9640 10 client.17478 handle_command_reply: tid=0 2024-04-22T00:07:34.031 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 client.17478 shutdown 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 msgr2=0x5640febfac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x5640febf5850 tx=0x7f6084011400 comp rx=0 tx=0).stop 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 shutdown_connections 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640febf7520 0x5640febfb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5640fec6dbf0 0x5640fec70010 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5640febee940 0x5640febfac60 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 --2- 172.21.15.38:0/3453113090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5640febee020 0x5640febff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 >> 172.21.15.38:0/3453113090 conn(0x5640febe8750 msgr2=0x5640febe7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6070ff9640 0 client.17478 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 shutdown_connections 2024-04-22T00:07:34.032 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.028+0000 7f6092014740 1 -- 172.21.15.38:0/3453113090 wait complete. 2024-04-22T00:07:34.033 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f6092014740 20 client.17478 trim_cache size 0 max 16384 2024-04-22T00:07:34.033 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:07:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 msgr2=0x7f60740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 0x7f60740a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f60800ccb80 tx=0x7f60800c6430 comp rx=0 tx=0).stop 2024-04-22T00:07:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 msgr2=0x7f608819c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f608000e970 tx=0x7f608000ee30 comp rx=0 tx=0).stop 2024-04-22T00:07:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 shutdown_connections 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60740a2d00 0x7f60740a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f608810bd00 0x7f608819c8d0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60881032e0 0x7f6088198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 --2- 172.21.15.38:0/3365603657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60881029c0 0x7f6088198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 >> 172.21.15.38:0/3365603657 conn(0x7f6088077740 msgr2=0x7f60880fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 shutdown_connections 2024-04-22T00:07:34.035 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:34.029+0000 7f608f42e640 1 -- 172.21.15.38:0/3365603657 wait complete. 2024-04-22T00:07:34.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:34 smithi038 ceph-mon[27972]: pgmap v167: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.20k op/s 2024-04-22T00:07:34.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:34 smithi203 ceph-mon[31431]: pgmap v167: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.20k op/s 2024-04-22T00:07:34.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:34 smithi111 ceph-mon[31543]: pgmap v167: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 13 MiB/s wr, 3.20k op/s 2024-04-22T00:07:35.037 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:07:35.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:07:35.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:07:35.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.182+0000 7f96c86a7640 1 -- 172.21.15.38:0/2076685313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0112620 msgr2=0x7f96c0114a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 --2- 172.21.15.38:0/2076685313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0112620 0x7f96c0114a10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009cd0 tx=0x7f96b802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 -- 172.21.15.38:0/2076685313 shutdown_connections 2024-04-22T00:07:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 --2- 172.21.15.38:0/2076685313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0112620 0x7f96c0114a10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 --2- 172.21.15.38:0/2076685313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96c0076590 0x7f96c01120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 --2- 172.21.15.38:0/2076685313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0075c70 0x7f96c0076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 -- 172.21.15.38:0/2076685313 >> 172.21.15.38:0/2076685313 conn(0x7f96c00ff4d0 msgr2=0x7f96c01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 -- 172.21.15.38:0/2076685313 shutdown_connections 2024-04-22T00:07:35.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.183+0000 7f96c86a7640 1 -- 172.21.15.38:0/2076685313 wait complete. 2024-04-22T00:07:35.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.184+0000 7f96c86a7640 1 Processor -- start 2024-04-22T00:07:35.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 -- start start 2024-04-22T00:07:35.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96c0075c70 0x7f96c019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 0x7f96c019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 0x7f96c01a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96c0118140 con 0x7f96c0112620 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96c0117fc0 con 0x7f96c0076590 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c86a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01182c0 con 0x7f96c0075c70 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 0x7f96c019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 0x7f96c019ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:40884/0 (socket says 172.21.15.38:40884) 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.185+0000 7f96c5c42640 1 -- 172.21.15.38:0/3399754885 learned_addr learned my addr 172.21.15.38:0/3399754885 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 0x7f96c01a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 -- 172.21.15.38:0/3399754885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96c0075c70 msgr2=0x7f96c019e7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96c0075c70 0x7f96c019e7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 -- 172.21.15.38:0/3399754885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 msgr2=0x7f96c019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 0x7f96c019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 -- 172.21.15.38:0/3399754885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c01a2730 con 0x7f96c0112620 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.186+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 0x7f96c01a2000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f96b802c700 tx=0x7f96b8032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.187+0000 7f96b77fe640 1 -- 172.21.15.38:0/3399754885 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8041020 con 0x7f96c0112620 2024-04-22T00:07:35.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.187+0000 7f96b77fe640 1 -- 172.21.15.38:0/3399754885 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b8034cd0 con 0x7f96c0112620 2024-04-22T00:07:35.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.188+0000 7f96b77fe640 1 -- 172.21.15.38:0/3399754885 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8039040 con 0x7f96c0112620 2024-04-22T00:07:35.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.188+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c01a29c0 con 0x7f96c0112620 2024-04-22T00:07:35.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.188+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c01ab390 con 0x7f96c0112620 2024-04-22T00:07:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96b77fe640 1 -- 172.21.15.38:0/3399754885 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f96b8044020 con 0x7f96c0112620 2024-04-22T00:07:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 Processor -- start 2024-04-22T00:07:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 -- start start 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b845a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b8463db0 0x5565b84661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565b83be170 con 0x5565b845aa40 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5565b83667c0 con 0x5565b845b720 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.191+0000 7f96cb36b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5565b83ba130 con 0x5565b8463db0 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37198/0 (socket says 172.21.15.38:37198) 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 -- 172.21.15.38:0/882101816 learned_addr learned my addr 172.21.15.38:0/882101816 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b845a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c5c42640 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b8463db0 0x5565b84661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 -- 172.21.15.38:0/882101816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b8463db0 msgr2=0x5565b84661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b8463db0 0x5565b84661a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 -- 172.21.15.38:0/882101816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 msgr2=0x5565b845a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b845a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 -- 172.21.15.38:0/882101816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565b8377e60 con 0x5565b845aa40 2024-04-22T00:07:35.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96c6443640 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f96b000ba80 tx=0x7f96b000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb787f535a373d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96b4ff9640 1 -- 172.21.15.38:0/882101816 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0016070 con 0x5565b845aa40 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.192+0000 7f96b77fe640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 0x7f96ac0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96b4ff9640 1 -- 172.21.15.38:0/882101816 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b0005500 con 0x5565b845aa40 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96b77fe640 1 -- 172.21.15.38:0/3399754885 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96b80f11a0 con 0x7f96c0112620 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96c5c42640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 0x7f96ac0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96b4ff9640 1 -- 172.21.15.38:0/882101816 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0010460 con 0x5565b845aa40 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 -- 172.21.15.38:0/882101816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 msgr2=0x5565b845aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f96b000ba80 tx=0x7f96b000bf40 comp rx=0 tx=0).stop 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 -- 172.21.15.38:0/882101816 shutdown_connections 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b8463db0 0x5565b84661a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845aa40 0x5565b845aec0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 --2- 172.21.15.38:0/882101816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b845a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.193+0000 7f96cb36b740 1 -- 172.21.15.38:0/882101816 >> 172.21.15.38:0/882101816 conn(0x5565b84214f0 msgr2=0x5565b8455490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.194+0000 7f96c5c42640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 0x7f96ac0a5340 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0097f0 tx=0x7f96bc009780 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.194+0000 7f96cb36b740 1 -- 172.21.15.38:0/882101816 shutdown_connections 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.194+0000 7f96cb36b740 1 -- 172.21.15.38:0/882101816 wait complete. 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.194+0000 7f96cb36b740 1 -- start start 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.194+0000 7f96cb36b740 1 Processor -- start 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8460c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845b720 0x5565b8461180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 0x5565b84616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565b8466c70 con 0x5565b845b720 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5565b8379d40 con 0x5565b8463db0 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6443640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8460c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6443640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8460c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45182/0 (socket says 172.21.15.38:45182) 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6443640 1 -- 172.21.15.38:0/3384102169 learned_addr learned my addr 172.21.15.38:0/3384102169 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c5c42640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845b720 0x5565b8461180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5565b8466df0 con 0x5565b845aa40 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96cb36b740 10 client.0 _ll_register_callbacks cb 0x5565b83b5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 0x5565b84616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 -- 172.21.15.38:0/3384102169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 msgr2=0x5565b8460c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8460c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 -- 172.21.15.38:0/3384102169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845b720 msgr2=0x5565b8461180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845b720 0x5565b8461180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.195+0000 7f96c6c44640 1 -- 172.21.15.38:0/3384102169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565b819a7e0 con 0x5565b8463db0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 0x5565b84616c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96b80ee6c0 tx=0x7f96b80bd320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440d7e4913872881 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8003e30 con 0x5565b8463db0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96b80c5c70 con 0x5565b8463db0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b0005500 con 0x5565b8463db0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.196+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f96b80bdb90 con 0x5565b8463db0 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 10 client.26855 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:35.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5565b81faf70 con 0x5565b8463db0 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b80098b0 con 0x5565b8463db0 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f96b803c030 con 0x5565b8463db0 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 10 client.26855 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 10 client.26855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 20 client.26855 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b8474040 0x5565b84764e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 4 client.26855 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f96b0005500 con 0x5565b8474040 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.197+0000 7f96c6443640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b8474040 0x5565b84764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.198+0000 7f96a9ffb640 10 client.26855 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.198+0000 7f96c6443640 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b8474040 0x5565b84764e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8b17972b475bee2 server_cookie=7f09776190d572e2 in_seq=0 out_seq=0 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3384102169 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f96b0005500 con 0x5565b8474040 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96a9ffb640 10 client.26855 handle_command_reply: tid=0 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 client.26855 shutdown 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 msgr2=0x5565b84616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 0x5565b84616c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96b80ee6c0 tx=0x7f96b80bd320 comp rx=0 tx=0).stop 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 shutdown_connections 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b8463db0 0x5565b84616c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b8474040 0x5565b84764e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b845b720 0x5565b8461180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3384102169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8460c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 >> 172.21.15.38:0/3384102169 conn(0x5565b84214f0 msgr2=0x5565b84571e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96a9ffb640 0 client.26855 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 shutdown_connections 2024-04-22T00:07:35.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 1 -- 172.21.15.38:0/3384102169 wait complete. 2024-04-22T00:07:35.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.199+0000 7f96cb36b740 20 client.26855 trim_cache size 0 max 16384 2024-04-22T00:07:35.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 Processor -- start 2024-04-22T00:07:35.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 -- start start 2024-04-22T00:07:35.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8463370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b846fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b5050 con 0x5565b8463db0 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b6b90 con 0x5565b845b720 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96cb36b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b5720 con 0x5565b845aa40 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96c6443640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8463370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.225+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b846fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37224/0 (socket says 172.21.15.38:37224) 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 -- 172.21.15.38:0/1033669392 learned_addr learned my addr 172.21.15.38:0/1033669392 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 -- 172.21.15.38:0/1033669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 msgr2=0x5565b8463370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8463370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 -- 172.21.15.38:0/1033669392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 msgr2=0x5565b846fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b846fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 -- 172.21.15.38:0/1033669392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565b8092170 con 0x5565b8463db0 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c5c42640 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b846fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96c6c44640 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f96b80eebb0 tx=0x7f96b80bb9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bf2124c538fab8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96abfff640 1 -- 172.21.15.38:0/1033669392 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b80bcdf0 con 0x5565b8463db0 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96abfff640 1 -- 172.21.15.38:0/1033669392 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b8003e30 con 0x5565b8463db0 2024-04-22T00:07:35.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.226+0000 7f96abfff640 1 -- 172.21.15.38:0/1033669392 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b80bcb40 con 0x5565b8463db0 2024-04-22T00:07:35.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 -- 172.21.15.38:0/1033669392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 msgr2=0x5565b8454370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f96b80eebb0 tx=0x7f96b80bb9a0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 -- 172.21.15.38:0/1033669392 shutdown_connections 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b8454370 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845b720 0x5565b846fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 --2- 172.21.15.38:0/1033669392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845aa40 0x5565b8463370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 -- 172.21.15.38:0/1033669392 >> 172.21.15.38:0/1033669392 conn(0x5565b84214f0 msgr2=0x5565b8453c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 -- 172.21.15.38:0/1033669392 shutdown_connections 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.227+0000 7f96cb36b740 1 -- 172.21.15.38:0/1033669392 wait complete. 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 -- start start 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 Processor -- start 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b84d9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c6443640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c6443640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:40928/0 (socket says 172.21.15.38:40928) 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c5c42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45198/0 (socket says 172.21.15.38:45198) 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c5c42640 1 -- 172.21.15.38:0/3690425179 learned_addr learned my addr 172.21.15.38:0/3690425179 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b64c0 con 0x5565b8463db0 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b5df0 con 0x5565b845aa40 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01b3030 con 0x5565b845b720 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b84d9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.228+0000 7f96cb36b740 10 client.0 _ll_register_callbacks cb 0x7f96b8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 -- 172.21.15.38:0/3690425179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 msgr2=0x5565b845e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 -- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 msgr2=0x5565b845def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565b84d9b60 con 0x5565b8463db0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c5c42640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6443640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96c6c44640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b84d9620 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f96b80b9e60 tx=0x7f96b802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f1f36f5d83ca76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b80bcdf0 con 0x5565b8463db0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f96b80b91e0 con 0x5565b8463db0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.229+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b8003e30 con 0x5565b8463db0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f96b80098b0 con 0x5565b8463db0 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 10 client.17496 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:35.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5565b84d9b60 con 0x5565b8463db0 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b80c1840 con 0x5565b8463db0 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f96b80c29d0 con 0x5565b8463db0 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 10 client.17496 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 10 client.17496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 20 client.17496 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b84db020 0x5565b84dd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 4 client.17496 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f96b8003e30 con 0x5565b84db020 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.230+0000 7f96c6443640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b84db020 0x5565b84dd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.231+0000 7f96a9ffb640 10 client.17496 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.231+0000 7f96c6443640 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b84db020 0x5565b84dd410 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af2ac5db059f032c server_cookie=b4129ecc14a710de in_seq=0 out_seq=0 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96a9ffb640 1 -- 172.21.15.38:0/3690425179 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f96b8003e30 con 0x5565b84db020 2024-04-22T00:07:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96a9ffb640 10 client.17496 handle_command_reply: tid=0 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 client.17496 shutdown 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 msgr2=0x5565b84d9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b84d9620 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f96b80b9e60 tx=0x7f96b802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 shutdown_connections 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5565b84db020 0x5565b84dd410 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96a9ffb640 0 client.17496 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565b8463db0 0x5565b84d9620 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5565b845b720 0x5565b845e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 --2- 172.21.15.38:0/3690425179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565b845aa40 0x5565b845def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 >> 172.21.15.38:0/3690425179 conn(0x5565b84214f0 msgr2=0x5565b84544f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 shutdown_connections 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 1 -- 172.21.15.38:0/3690425179 wait complete. 2024-04-22T00:07:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.232+0000 7f96cb36b740 20 client.17496 trim_cache size 0 max 16384 2024-04-22T00:07:35.233 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:07:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:07:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "d4e634b0-bce2-4d9a-ae2c-642e8f550d9e", 2024-04-22T00:07:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:07:35.233 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:07:35.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 msgr2=0x7f96ac0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 0x7f96ac0a5340 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0097f0 tx=0x7f96bc009780 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 msgr2=0x7f96c01a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 0x7f96c01a2000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f96b802c700 tx=0x7f96b8032ec0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 shutdown_connections 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f96ac0a2e80 0x7f96ac0a5340 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96c0112620 0x7f96c01a2000 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96c0076590 0x7f96c019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 --2- 172.21.15.38:0/3399754885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96c0075c70 0x7f96c019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.233+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 >> 172.21.15.38:0/3399754885 conn(0x7f96c00ff4d0 msgr2=0x7f96c01018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.234+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 shutdown_connections 2024-04-22T00:07:35.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.234+0000 7f96c86a7640 1 -- 172.21.15.38:0/3399754885 wait complete. 2024-04-22T00:07:35.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:07:35.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:07:35.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 -- 172.21.15.38:0/934840666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc806a6e0 msgr2=0x7efdc810dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/934840666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc806a6e0 0x7efdc810dc10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009900 tx=0x7efdb002bc30 comp rx=0 tx=0).stop 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 -- 172.21.15.38:0/934840666 shutdown_connections 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/934840666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc810e150 0x7efdc8110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/934840666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc806a6e0 0x7efdc810dc10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/934840666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc806a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 -- 172.21.15.38:0/934840666 >> 172.21.15.38:0/934840666 conn(0x7efdc80776a0 msgr2=0x7efdc8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 -- 172.21.15.38:0/934840666 shutdown_connections 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.382+0000 7efdcdb7a640 1 -- 172.21.15.38:0/934840666 wait complete. 2024-04-22T00:07:35.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 Processor -- start 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 -- start start 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 0x7efdc819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efdc8113d10 con 0x7efdc810e150 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc8113b90 con 0x7efdc806a6e0 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdcdb7a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdc8113e90 con 0x7efdc8069dc0 2024-04-22T00:07:35.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 0x7efdc819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37246/0 (socket says 172.21.15.38:37246) 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 0x7efdc819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:40950/0 (socket says 172.21.15.38:40950) 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45214/0 (socket says 172.21.15.38:45214) 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.383+0000 7efdc7fff640 1 -- 172.21.15.38:0/1259780638 learned_addr learned my addr 172.21.15.38:0/1259780638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 -- 172.21.15.38:0/1259780638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 msgr2=0x7efdc819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 -- 172.21.15.38:0/1259780638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 msgr2=0x7efdc819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 0x7efdc819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 -- 172.21.15.38:0/1259780638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc819f250 con 0x7efdc810e150 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc7fff640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efdc000e9d0 tx=0x7efdc000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc4ff9640 1 -- 172.21.15.38:0/1259780638 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc000cd80 con 0x7efdc810e150 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc4ff9640 1 -- 172.21.15.38:0/1259780638 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc0005020 con 0x7efdc810e150 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc819f540 con 0x7efdc810e150 2024-04-22T00:07:35.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdc4ff9640 1 -- 172.21.15.38:0/1259780638 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00116d0 con 0x7efdc810e150 2024-04-22T00:07:35.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.384+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc81a6dd0 con 0x7efdc810e150 2024-04-22T00:07:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.386+0000 7efdc4ff9640 1 -- 172.21.15.38:0/1259780638 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efdc00051d0 con 0x7efdc810e150 2024-04-22T00:07:35.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 Processor -- start 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 -- start start 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc4ff9640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 0x7efdb40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920673ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a9205cf560 con 0x55a920674430 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a920577730 con 0x55a92066af30 2024-04-22T00:07:35.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdd0766740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a9205cb520 con 0x55a92066b850 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc77fe640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 0x7efdb40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:40962/0 (socket says 172.21.15.38:40962) 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37250/0 (socket says 172.21.15.38:37250) 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc4ff9640 1 -- 172.21.15.38:0/1259780638 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efdc00cfbb0 con 0x7efdc810e150 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 -- 172.21.15.38:0/2113322495 learned_addr learned my addr 172.21.15.38:0/2113322495 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920673ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 -- 172.21.15.38:0/2113322495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 msgr2=0x55a920673ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920673ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 -- 172.21.15.38:0/2113322495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 msgr2=0x55a920676820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc6ffd640 1 -- 172.21.15.38:0/2113322495 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9203ad7d0 con 0x55a92066af30 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.387+0000 7efdc7fff640 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.388+0000 7efdc77fe640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 0x7efdb40a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009bd0 tx=0x7efdb8005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.388+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55a92066aa00 tx=0x7efdb0005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ff29b7b41b38366 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.388+0000 7efdbdffb640 1 -- 172.21.15.38:0/2113322495 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0004030 con 0x55a92066af30 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.388+0000 7efdbdffb640 1 -- 172.21.15.38:0/2113322495 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb0004ba0 con 0x55a92066af30 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.388+0000 7efdbdffb640 1 -- 172.21.15.38:0/2113322495 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0005350 con 0x55a92066af30 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- 172.21.15.38:0/2113322495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 msgr2=0x55a92066b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55a92066aa00 tx=0x7efdb0005e40 comp rx=0 tx=0).stop 2024-04-22T00:07:35.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- 172.21.15.38:0/2113322495 shutdown_connections 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a920674430 0x55a920676820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920673ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- 172.21.15.38:0/2113322495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066af30 0x55a92066b310 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- 172.21.15.38:0/2113322495 >> 172.21.15.38:0/2113322495 conn(0x55a920665660 msgr2=0x55a920665a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- 172.21.15.38:0/2113322495 shutdown_connections 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- 172.21.15.38:0/2113322495 wait complete. 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- start start 2024-04-22T00:07:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 Processor -- start 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920671a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a920671fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a9206771c0 con 0x55a92066af30 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a92058ae00 con 0x55a92066b850 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a920677340 con 0x55a920674430 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.389+0000 7efdd0766740 10 client.0 _ll_register_callbacks cb 0x55a9205c5eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:35.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37266/0 (socket says 172.21.15.38:37266) 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 -- 172.21.15.38:0/2583110515 learned_addr learned my addr 172.21.15.38:0/2583110515 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc7fff640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a920671fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920671a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 -- 172.21.15.38:0/2583110515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 msgr2=0x55a920671fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a920671fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 -- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 msgr2=0x55a920671a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920671a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 -- 172.21.15.38:0/2583110515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9202e33f0 con 0x55a92066af30 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdc77fe640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efdb8003940 tx=0x7efdb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c35766d4325965 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efda6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb800ecc0 con 0x55a92066af30 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efdb8005020 con 0x55a92066af30 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdb0004ba0 con 0x55a92066af30 2024-04-22T00:07:35.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdb802e050 con 0x55a92066af30 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdd0766740 10 client.17508 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.390+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a9202a30f0 con 0x55a92066af30 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb0004ba0 con 0x55a92066af30 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efdb801e420 con 0x55a92066af30 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 10 client.17508 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 20 client.17508 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:35.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206848f0 0x55a920686d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 4 client.17508 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9205dcaa0 con 0x55a9206848f0 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206848f0 0x55a920686d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206848f0 0x55a920686d40 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79c0ebf0ea824bcc server_cookie=753cbb32f75af63e in_seq=0 out_seq=0 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.391+0000 7efda6ffd640 10 client.17508 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efda6ffd640 1 -- 172.21.15.38:0/2583110515 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9205dcaa0 con 0x55a9206848f0 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efda6ffd640 10 client.17508 handle_command_reply: tid=0 2024-04-22T00:07:35.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 client.17508 shutdown 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 msgr2=0x55a920671550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efdb8003940 tx=0x7efdb8004730 comp rx=0 tx=0).stop 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 shutdown_connections 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a920671fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920671a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206848f0 0x55a920686d40 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 --2- 172.21.15.38:0/2583110515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920671550 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efda6ffd640 0 client.17508 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 >> 172.21.15.38:0/2583110515 conn(0x55a920665660 msgr2=0x55a9206673b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 shutdown_connections 2024-04-22T00:07:35.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.392+0000 7efdd0766740 1 -- 172.21.15.38:0/2583110515 wait complete. 2024-04-22T00:07:35.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.393+0000 7efdd0766740 20 client.17508 trim_cache size 0 max 16384 2024-04-22T00:07:35.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 Processor -- start 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 -- start start 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920680080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a920674430 0x55a920660440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81b0c60 con 0x55a92066af30 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81b27a0 con 0x55a920674430 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdd0766740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81b1330 con 0x55a92066b850 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37270/0 (socket says 172.21.15.38:37270) 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdc77fe640 1 -- 172.21.15.38:0/3522505982 learned_addr learned my addr 172.21.15.38:0/3522505982 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.410+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920680080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc7fff640 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a920674430 0x55a920660440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 -- 172.21.15.38:0/3522505982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 msgr2=0x55a920680080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920680080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 -- 172.21.15.38:0/3522505982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a920674430 msgr2=0x55a920660440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a920674430 0x55a920660440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 -- 172.21.15.38:0/3522505982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a920670500 con 0x55a92066af30 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdc77fe640 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efdb8003690 tx=0x7efdb8005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b922509a136344 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdbd7fa640 1 -- 172.21.15.38:0/3522505982 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8004590 con 0x55a92066af30 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdbd7fa640 1 -- 172.21.15.38:0/3522505982 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb8036030 con 0x55a92066af30 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.411+0000 7efdbd7fa640 1 -- 172.21.15.38:0/3522505982 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8032660 con 0x55a92066af30 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- 172.21.15.38:0/3522505982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 msgr2=0x55a920673c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efdb8003690 tx=0x7efdb8005c70 comp rx=0 tx=0).stop 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- 172.21.15.38:0/3522505982 shutdown_connections 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a920674430 0x55a920660440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a92066b850 0x55a920680080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 --2- 172.21.15.38:0/3522505982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a920673c80 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- 172.21.15.38:0/3522505982 >> 172.21.15.38:0/3522505982 conn(0x55a920665660 msgr2=0x55a920663e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- 172.21.15.38:0/3522505982 shutdown_connections 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- 172.21.15.38:0/3522505982 wait complete. 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 -- start start 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.412+0000 7efdd0766740 1 Processor -- start 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920678810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81b20d0 con 0x55a92066af30 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81b1a00 con 0x55a92066b850 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdc81aec40 con 0x55a920674430 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdd0766740 10 client.0 _ll_register_callbacks cb 0x7efdb8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920678810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37282/0 (socket says 172.21.15.38:37282) 2024-04-22T00:07:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:45262/0 (socket says 172.21.15.38:45262) 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920678810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:40996/0 (socket says 172.21.15.38:40996) 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 -- 172.21.15.38:0/325093311 learned_addr learned my addr 172.21.15.38:0/325093311 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 -- 172.21.15.38:0/325093311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 msgr2=0x55a9206e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 -- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 msgr2=0x55a920678810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920678810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 -- 172.21.15.38:0/325093311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9206790e0 con 0x55a92066af30 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc7fff640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efdc77fe640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efdb80059f0 tx=0x7efdb801ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ca8987f2ed62c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efda6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.413+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb803f070 con 0x55a92066af30 2024-04-22T00:07:35.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdb8036030 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efdb800eac0 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdb804b050 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 10 client.17514 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a9206eb4a0 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb8036030 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efdb8004590 con 0x55a92066af30 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 10 client.17514 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 20 client.17514 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206ebab0 0x55a9206eded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 4 client.17514 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a9205d6df0 con 0x55a9206ebab0 2024-04-22T00:07:35.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.414+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206ebab0 0x55a9206eded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efda6ffd640 10 client.17514 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdc6ffd640 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206ebab0 0x55a9206eded0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a6e4f65768b5e6 server_cookie=2a16e9e4e45079ab in_seq=0 out_seq=0 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efda6ffd640 1 -- 172.21.15.38:0/325093311 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a9205d6df0 con 0x55a9206ebab0 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efda6ffd640 10 client.17514 handle_command_reply: tid=0 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 client.17514 shutdown 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 msgr2=0x55a9206782d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efdb80059f0 tx=0x7efdb801ed80 comp rx=0 tx=0).stop 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 shutdown_connections 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a920674430 0x55a9206e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a92066b850 0x55a920678810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a9206ebab0 0x55a9206eded0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 --2- 172.21.15.38:0/325093311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a92066af30 0x55a9206782d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 >> 172.21.15.38:0/325093311 conn(0x55a920665660 msgr2=0x55a9206646e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efda6ffd640 0 client.17514 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 shutdown_connections 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 1 -- 172.21.15.38:0/325093311 wait complete. 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.415+0000 7efdd0766740 20 client.17514 trim_cache size 0 max 16384 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (155 inodes in the stack)", 2024-04-22T00:07:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "d4e634b0-bce2-4d9a-ae2c-642e8f550d9e": { 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "d4e634b0-bce2-4d9a-ae2c-642e8f550d9e", 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:07:35.423 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 msgr2=0x7efdb40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 0x7efdb40a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009bd0 tx=0x7efdb8005f90 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 msgr2=0x7efdc819eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efdc000e9d0 tx=0x7efdc000ee90 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 shutdown_connections 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efdb40a2d80 0x7efdb40a5240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efdc810e150 0x7efdc819eb20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc806a6e0 0x7efdc819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 --2- 172.21.15.38:0/1259780638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdc8069dc0 0x7efdc819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 >> 172.21.15.38:0/1259780638 conn(0x7efdc80776a0 msgr2=0x7efdc810f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 shutdown_connections 2024-04-22T00:07:35.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:07:35.416+0000 7efdcdb7a640 1 -- 172.21.15.38:0/1259780638 wait complete. 2024-04-22T00:07:35.426 INFO:tasks.cephfs.filesystem:scrub status for tag:d4e634b0-bce2-4d9a-ae2c-642e8f550d9e - {'path': '/', 'tag': 'd4e634b0-bce2-4d9a-ae2c-642e8f550d9e', 'options': 'recursive,force'} 2024-04-22T00:07:36.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:36 smithi038 ceph-mon[27972]: pgmap v168: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:07:36.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:36 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:07:36.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:36 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:36.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:36 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:36.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:36 smithi203 ceph-mon[31431]: pgmap v168: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:07:36.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:36 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:07:36.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:36 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:36.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:36 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:36 smithi111 ceph-mon[31543]: pgmap v168: 129 pgs: 129 active+clean; 14 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:07:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:36 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:07:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:36 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:36 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:38.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:38 smithi038 ceph-mon[27972]: pgmap v169: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:07:38.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:38 smithi203 ceph-mon[31431]: pgmap v169: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:07:38.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:38 smithi111 ceph-mon[31543]: pgmap v169: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:07:40.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:40 smithi038 ceph-mon[27972]: pgmap v170: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:40.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:40 smithi203 ceph-mon[31431]: pgmap v170: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:40.828 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:40 smithi111 ceph-mon[31543]: pgmap v170: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:41.829 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:07:41 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:07:41.376+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v500>,<0x100000001fb/client.0 v476>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v478>,<0x10000000001/sv_1 v495>,<0x10000000000/qa v477>,<0x1/volumes v459>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: pgmap v171: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:42.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:42 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: pgmap v171: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:07:42.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:42 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: pgmap v171: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:07:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:07:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:42 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:43 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:43.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:43 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:43 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:43.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:43 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:43 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:43.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:43 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:43 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:43.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:43 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:43 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:43.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:43 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:43.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:43 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:43.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:43 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:44.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:44 smithi038 ceph-mon[27972]: pgmap v172: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.58k op/s 2024-04-22T00:07:44.730 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:44 smithi111 ceph-mon[31543]: pgmap v172: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.58k op/s 2024-04-22T00:07:44.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:44 smithi203 ceph-mon[31431]: pgmap v172: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 14 MiB/s wr, 3.58k op/s 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 -- 172.21.15.38:0/234872143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f010a0e0 msgr2=0x7f52f010a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 --2- 172.21.15.38:0/234872143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f010a0e0 0x7f52f010a4c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009900 tx=0x7f52e002bcd0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 -- 172.21.15.38:0/234872143 shutdown_connections 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 --2- 172.21.15.38:0/234872143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f0113690 0x7f52f0115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 --2- 172.21.15.38:0/234872143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f010aa00 0x7f52f0113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 --2- 172.21.15.38:0/234872143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f010a0e0 0x7f52f010a4c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.514+0000 7f52f6113640 1 -- 172.21.15.38:0/234872143 >> 172.21.15.38:0/234872143 conn(0x7f52f006c810 msgr2=0x7f52f006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.515+0000 7f52f6113640 1 -- 172.21.15.38:0/234872143 shutdown_connections 2024-04-22T00:07:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.515+0000 7f52f6113640 1 -- 172.21.15.38:0/234872143 wait complete. 2024-04-22T00:07:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.515+0000 7f52f6113640 1 Processor -- start 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.515+0000 7f52f6113640 1 -- start start 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f010a0e0 0x7f52f019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f010aa00 0x7f52f019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52f0118160 con 0x7f52f0113690 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52f0117fe0 con 0x7f52f010a0e0 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52f6113640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01182e0 con 0x7f52f010aa00 2024-04-22T00:07:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32954/0 (socket says 172.21.15.38:32954) 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 -- 172.21.15.38:0/1519727810 learned_addr learned my addr 172.21.15.38:0/1519727810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f010a0e0 0x7f52f019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 -- 172.21.15.38:0/1519727810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f010aa00 msgr2=0x7f52f019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f010aa00 0x7f52f019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 -- 172.21.15.38:0/1519727810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f010a0e0 msgr2=0x7f52f019e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f010a0e0 0x7f52f019e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 -- 172.21.15.38:0/1519727810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f01a3570 con 0x7f52f0113690 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.516+0000 7f52effff640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009ed0 tx=0x7f52e80104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.517+0000 7f52ecff9640 1 -- 172.21.15.38:0/1519727810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8004b20 con 0x7f52f0113690 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.517+0000 7f52ecff9640 1 -- 172.21.15.38:0/1519727810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e8004cc0 con 0x7f52f0113690 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.517+0000 7f52ecff9640 1 -- 172.21.15.38:0/1519727810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80166f0 con 0x7f52f0113690 2024-04-22T00:07:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.517+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f01a3860 con 0x7f52f0113690 2024-04-22T00:07:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.517+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f01ab0f0 con 0x7f52f0113690 2024-04-22T00:07:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.519+0000 7f52ecff9640 1 -- 172.21.15.38:0/1519727810 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f52e801d070 con 0x7f52f0113690 2024-04-22T00:07:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 Processor -- start 2024-04-22T00:07:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 -- start start 2024-04-22T00:07:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389ab20 0x5621f389af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5621f37feb70 con 0x5621f38a4210 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5621f37a7470 con 0x5621f389ab20 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5621f37fab30 con 0x5621f389b440 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52ecff9640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 0x7f52d80a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52effff640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 0x7f52d80a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32960/0 (socket says 172.21.15.38:32960) 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 -- 172.21.15.38:0/1910690006 learned_addr learned my addr 172.21.15.38:0/1910690006 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389ab20 0x5621f389af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52ecff9640 1 -- 172.21.15.38:0/1519727810 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52e80cef30 con 0x7f52f0113690 2024-04-22T00:07:45.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 -- 172.21.15.38:0/1910690006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 msgr2=0x5621f38a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.520+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52eeffd640 1 -- 172.21.15.38:0/1910690006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389ab20 msgr2=0x5621f389af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389ab20 0x5621f389af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52eeffd640 1 -- 172.21.15.38:0/1910690006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f3513210 con 0x5621f38a4210 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52effff640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 0x7f52d80a9670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f52e80cc500 tx=0x7f52e801d000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f52e40137e0 tx=0x7f52e4009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78557e5288956d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52d5ffb640 1 -- 172.21.15.38:0/1910690006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40132d0 con 0x5621f38a4210 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52d5ffb640 1 -- 172.21.15.38:0/1910690006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e4013e20 con 0x5621f38a4210 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.521+0000 7f52d5ffb640 1 -- 172.21.15.38:0/1910690006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40035f0 con 0x5621f38a4210 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1910690006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 msgr2=0x5621f38a6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f52e40137e0 tx=0x7f52e4009650 comp rx=0 tx=0).stop 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1910690006 shutdown_connections 2024-04-22T00:07:45.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f38a4210 0x5621f38a6600 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1910690006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389ab20 0x5621f389af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1910690006 >> 172.21.15.38:0/1910690006 conn(0x5621f3895440 msgr2=0x5621f3895820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1910690006 shutdown_connections 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1910690006 wait complete. 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 -- start start 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 Processor -- start 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.522+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38a1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5621f38a6de0 con 0x5621f389ab20 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5621f37baa10 con 0x5621f389b440 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5621f38a6f60 con 0x5621f38a4210 2024-04-22T00:07:45.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32968/0 (socket says 172.21.15.38:32968) 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 10 client.0 _ll_register_callbacks cb 0x5621f37f5b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 -- 172.21.15.38:0/120586519 learned_addr learned my addr 172.21.15.38:0/120586519 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 -- 172.21.15.38:0/120586519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 msgr2=0x5621f38a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 -- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 msgr2=0x5621f38a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 -- 172.21.15.38:0/120586519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f35df620 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52eeffd640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f52e400f010 tx=0x7f52e40058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f54d6dc987aabae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40132d0 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52e4005020 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e4013e20 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f52e4013470 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 10 client.17526 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.523+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5621f38b3080 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e4013e20 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f52e401f480 con 0x5621f389ab20 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 10 client.17526 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 10 client.17526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 20 client.17526 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f38b4430 0x5621f38b6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 4 client.17526 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621f380c0b0 con 0x5621f38b4430 2024-04-22T00:07:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.524+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f38b4430 0x5621f38b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.530+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f38b4430 0x5621f38b6820 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=721f1f9c6a2df60b server_cookie=3448b629d23cdf12 in_seq=0 out_seq=0 2024-04-22T00:07:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.530+0000 7f52ceffd640 10 client.17526 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52ceffd640 1 -- 172.21.15.38:0/120586519 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621f380c0b0 con 0x5621f38b4430 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52ceffd640 10 client.17526 handle_command_reply: tid=0 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 client.17526 shutdown 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 msgr2=0x5621f38a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f52e400f010 tx=0x7f52e40058c0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 shutdown_connections 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f38b4430 0x5621f38b6820 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52ceffd640 0 client.17526 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/120586519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a1250 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 >> 172.21.15.38:0/120586519 conn(0x5621f3895440 msgr2=0x5621f3897190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 shutdown_connections 2024-04-22T00:07:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 1 -- 172.21.15.38:0/120586519 wait complete. 2024-04-22T00:07:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.532+0000 7f52f8dd9740 20 client.17526 trim_cache size 0 max 16384 2024-04-22T00:07:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.547+0000 7f52f8dd9740 1 Processor -- start 2024-04-22T00:07:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.547+0000 7f52f8dd9740 1 -- start start 2024-04-22T00:07:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b5090 con 0x5621f389ab20 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b6bd0 con 0x5621f389b440 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b5760 con 0x5621f38a4210 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32980/0 (socket says 172.21.15.38:32980) 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 -- 172.21.15.38:0/1639272812 learned_addr learned my addr 172.21.15.38:0/1639272812 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 -- 172.21.15.38:0/1639272812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 msgr2=0x5621f38a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 -- 172.21.15.38:0/1639272812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 msgr2=0x5621f38afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 -- 172.21.15.38:0/1639272812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621f38a6930 con 0x5621f389ab20 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52eeffd640 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f52e4005e00 tx=0x7f52e4016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edc497be548759f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52d57fa640 1 -- 172.21.15.38:0/1639272812 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e403b070 con 0x5621f389ab20 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52d57fa640 1 -- 172.21.15.38:0/1639272812 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e4004520 con 0x5621f389ab20 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.548+0000 7f52d57fa640 1 -- 172.21.15.38:0/1639272812 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e401fac0 con 0x5621f389ab20 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1639272812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 msgr2=0x5621f38a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f52e4005e00 tx=0x7f52e4016040 comp rx=0 tx=0).stop 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1639272812 shutdown_connections 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f38a4210 0x5621f38a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f389b440 0x5621f38afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/1639272812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38a3980 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1639272812 >> 172.21.15.38:0/1639272812 conn(0x5621f3895440 msgr2=0x5621f38970d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1639272812 shutdown_connections 2024-04-22T00:07:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- 172.21.15.38:0/1639272812 wait complete. 2024-04-22T00:07:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.549+0000 7f52f8dd9740 1 -- start start 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 Processor -- start 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f38a4210 0x5621f38a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b6500 con 0x5621f389ab20 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b5e30 con 0x5621f38a4210 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52f01b3070 con 0x5621f389b440 2024-04-22T00:07:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52f8dd9740 10 client.0 _ll_register_callbacks cb 0x7f52e4009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32988/0 (socket says 172.21.15.38:32988) 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 -- 172.21.15.38:0/4268297439 learned_addr learned my addr 172.21.15.38:0/4268297439 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f38a4210 0x5621f38a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 -- 172.21.15.38:0/4268297439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 msgr2=0x5621f38a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 -- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f38a4210 msgr2=0x5621f38a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f38a4210 0x5621f38a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.550+0000 7f52eeffd640 1 -- 172.21.15.38:0/4268297439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621f38a8720 con 0x5621f389ab20 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52eeffd640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f52e401f7d0 tx=0x7f52e401f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255f697f20912558 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e403b070 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f52e401fac0 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e4004520 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f52e4048050 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52f8dd9740 10 client.17532 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5621f3922ef0 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.551+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e4004520 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f52e4003af0 con 0x5621f389ab20 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 10 client.17532 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 10 client.17532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 20 client.17532 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f3923540 0x5621f3925930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 4 client.17532 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621f3890900 con 0x5621f3923540 2024-04-22T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f3923540 0x5621f3925930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52ef7fe640 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f3923540 0x5621f3925930 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8faf3de4b556307a server_cookie=ec45dd9b0c9c244c in_seq=0 out_seq=0 2024-04-22T00:07:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.552+0000 7f52ceffd640 10 client.17532 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:45.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:45 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:07:45.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:45 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:45.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:45 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:07:45.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:45 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:45.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:45 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:07:45.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:45 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.837+0000 7f52ceffd640 1 -- 172.21.15.38:0/4268297439 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621f3890900 con 0x5621f3923540 2024-04-22T00:07:45.832 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.837+0000 7f52ceffd640 10 client.17532 handle_command_reply: tid=0 2024-04-22T00:07:45.833 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.837+0000 7f52f8dd9740 1 client.17532 shutdown 2024-04-22T00:07:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 msgr2=0x5621f38ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f52e401f7d0 tx=0x7f52e401f650 comp rx=0 tx=0).stop 2024-04-22T00:07:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 shutdown_connections 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5621f38a4210 0x5621f38a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621f389b440 0x5621f38a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5621f3923540 0x5621f3925930 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52ceffd640 0 client.17532 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.838+0000 7f52f8dd9740 1 --2- 172.21.15.38:0/4268297439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5621f389ab20 0x5621f38ac5d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.839+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 >> 172.21.15.38:0/4268297439 conn(0x5621f3895440 msgr2=0x5621f38a1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.839+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 shutdown_connections 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.839+0000 7f52f8dd9740 1 -- 172.21.15.38:0/4268297439 wait complete. 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.839+0000 7f52f8dd9740 20 client.17532 trim_cache size 0 max 16384 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:07:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:07:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:07:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 msgr2=0x7f52d80a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 0x7f52d80a9670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f52e80cc500 tx=0x7f52e801d000 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 msgr2=0x7f52f01a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009ed0 tx=0x7f52e80104b0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 shutdown_connections 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f52d80a71b0 0x7f52d80a9670 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52f0113690 0x7f52f01a2e40 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52f010aa00 0x7f52f019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 --2- 172.21.15.38:0/1519727810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52f010a0e0 0x7f52f019e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 >> 172.21.15.38:0/1519727810 conn(0x7f52f006c810 msgr2=0x7f52f0070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.840+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 shutdown_connections 2024-04-22T00:07:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:45.841+0000 7f52f6113640 1 -- 172.21.15.38:0/1519727810 wait complete. 2024-04-22T00:07:46.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:46 smithi038 ceph-mon[27972]: pgmap v173: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:46.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:46 smithi203 ceph-mon[31431]: pgmap v173: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:46.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:46 smithi111 ceph-mon[31543]: pgmap v173: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 2.80k op/s 2024-04-22T00:07:48.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:48 smithi038 ceph-mon[27972]: pgmap v174: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-22T00:07:48.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:48 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:48.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:48 smithi203 ceph-mon[31431]: pgmap v174: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-22T00:07:48.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:48 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:48.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:48 smithi111 ceph-mon[31543]: pgmap v174: 129 pgs: 129 active+clean; 14 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 14 MiB/s wr, 3.52k op/s 2024-04-22T00:07:48.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:48 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:07:50.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:50 smithi038 ceph-mon[27972]: pgmap v175: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-04-22T00:07:50.781 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:50 smithi203 ceph-mon[31431]: pgmap v175: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-04-22T00:07:50.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:50 smithi111 ceph-mon[31543]: pgmap v175: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 3.11k op/s 2024-04-22T00:07:52.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:52 smithi038 ceph-mon[27972]: pgmap v176: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:07:52.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:52 smithi203 ceph-mon[31431]: pgmap v176: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:07:52.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:52 smithi111 ceph-mon[31543]: pgmap v176: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:07:54.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:54 smithi038 ceph-mon[27972]: pgmap v177: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 15 MiB/s wr, 3.69k op/s 2024-04-22T00:07:54.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:54 smithi203 ceph-mon[31431]: pgmap v177: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 15 MiB/s wr, 3.69k op/s 2024-04-22T00:07:54.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:54 smithi111 ceph-mon[31543]: pgmap v177: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 15 MiB/s wr, 3.69k op/s 2024-04-22T00:07:56.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:56 smithi038 ceph-mon[27972]: pgmap v178: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 2.67k op/s 2024-04-22T00:07:56.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:56 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:56.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:56 smithi203 ceph-mon[31431]: pgmap v178: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 2.67k op/s 2024-04-22T00:07:56.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:56 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:56.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:56 smithi111 ceph-mon[31543]: pgmap v178: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 2.67k op/s 2024-04-22T00:07:56.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:56 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:57.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:57.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:57.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:57.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:57 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:07:57.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:57.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:57.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:57.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:57 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:07:57.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:57.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:57.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:57.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:57 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:07:58.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:07:58 smithi038 ceph-mon[27972]: pgmap v179: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-04-22T00:07:58.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:07:58 smithi203 ceph-mon[31431]: pgmap v179: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-04-22T00:07:58.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:07:58 smithi111 ceph-mon[31543]: pgmap v179: 129 pgs: 129 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 3.21k op/s 2024-04-22T00:07:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.978+0000 7f7a25afb640 1 -- 172.21.15.38:0/3783669591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 msgr2=0x7f7a2010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.978+0000 7f7a25afb640 1 --2- 172.21.15.38:0/3783669591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a2010b7d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7a14009900 tx=0x7f7a1402bd60 comp rx=0 tx=0).stop 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 -- 172.21.15.38:0/3783669591 shutdown_connections 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 --2- 172.21.15.38:0/3783669591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 0x7f7a2010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 --2- 172.21.15.38:0/3783669591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a2010b7d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 --2- 172.21.15.38:0/3783669591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 -- 172.21.15.38:0/3783669591 >> 172.21.15.38:0/3783669591 conn(0x7f7a200777c0 msgr2=0x7f7a20077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 -- 172.21.15.38:0/3783669591 shutdown_connections 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 -- 172.21.15.38:0/3783669591 wait complete. 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 Processor -- start 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.979+0000 7f7a25afb640 1 -- start start 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 0x7f7a201010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a20111b60 con 0x7f7a201032f0 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201119e0 con 0x7f7a201029d0 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a25afb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a20111ce0 con 0x7f7a2010bd10 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38540/0 (socket says 172.21.15.38:38540) 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1effd640 1 -- 172.21.15.38:0/113869806 learned_addr learned my addr 172.21.15.38:0/113869806 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 0x7f7a201010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/113869806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 msgr2=0x7f7a201010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 0x7f7a201010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 msgr2=0x7f7a20100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/113869806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a201a6520 con 0x7f7a201029d0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.980+0000 7f7a1effd640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20100620 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e9f0 tx=0x7f7a1000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a1cff9640 1 -- 172.21.15.38:0/113869806 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1000ccc0 con 0x7f7a201029d0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a201a6810 con 0x7f7a201029d0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a1cff9640 1 -- 172.21.15.38:0/113869806 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a10010910 con 0x7f7a201029d0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a1cff9640 1 -- 172.21.15.38:0/113869806 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10018960 con 0x7f7a201029d0 2024-04-22T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.981+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a201a6d60 con 0x7f7a201029d0 2024-04-22T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.983+0000 7f7a1cff9640 1 -- 172.21.15.38:0/113869806 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7a10010ac0 con 0x7f7a201029d0 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.983+0000 7f7a1cff9640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 0x7f7a080a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a1effd640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 0x7f7a080a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a1cff9640 1 -- 172.21.15.38:0/113869806 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7a100cfb20 con 0x7f7a201029d0 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 Processor -- start 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a1effd640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 0x7f7a080a5260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a140096c0 tx=0x7f7a14005a20 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 -- start start 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38ac440 0x55a2e38b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 0x55a2e38b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a2e380fb70 con 0x55a2e38abb20 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a2e37b8470 con 0x55a2e38b5210 2024-04-22T00:07:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.984+0000 7f7a287c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a2e380bb30 con 0x55a2e38ac440 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 0x55a2e38b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38544/0 (socket says 172.21.15.38:38544) 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/3349451451 learned_addr learned my addr 172.21.15.38:0/3349451451 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1effd640 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38ac440 0x55a2e38b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/3349451451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38ac440 msgr2=0x55a2e38b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38ac440 0x55a2e38b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/3349451451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 msgr2=0x55a2e38b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 0x55a2e38b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/3349451451 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2e3524210 con 0x55a2e38abb20 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 0x55a2e38b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a10098210 tx=0x7f7a1009b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bc982da065fbd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a05ffb640 1 -- 172.21.15.38:0/3349451451 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a100cd1b0 con 0x55a2e38abb20 2024-04-22T00:07:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a05ffb640 1 -- 172.21.15.38:0/3349451451 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a10098420 con 0x55a2e38abb20 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.985+0000 7f7a05ffb640 1 -- 172.21.15.38:0/3349451451 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10018d60 con 0x55a2e38abb20 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 -- 172.21.15.38:0/3349451451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 msgr2=0x55a2e38abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a10098210 tx=0x7f7a1009b810 comp rx=0 tx=0).stop 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 -- 172.21.15.38:0/3349451451 shutdown_connections 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38b5210 0x55a2e38b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38ac440 0x55a2e38b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 --2- 172.21.15.38:0/3349451451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38abf00 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 -- 172.21.15.38:0/3349451451 >> 172.21.15.38:0/3349451451 conn(0x55a2e38a6440 msgr2=0x55a2e38a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 -- 172.21.15.38:0/3349451451 shutdown_connections 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.986+0000 7f7a287c0740 1 -- 172.21.15.38:0/3349451451 wait complete. 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 -- start start 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 Processor -- start 2024-04-22T00:07:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38b2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e38b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a2e38b7de0 con 0x55a2e38abb20 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.987+0000 7f7a287c0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a2e37cba10 con 0x55a2e38ac440 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a287c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a2e38b7f60 con 0x55a2e38b5210 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38546/0 (socket says 172.21.15.38:38546) 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/1069056031 learned_addr learned my addr 172.21.15.38:0/1069056031 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1effd640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e38b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a287c0740 10 client.0 _ll_register_callbacks cb 0x55a2e3806b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/1069056031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 msgr2=0x55a2e38b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e38b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 msgr2=0x55a2e38b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/1069056031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2e35f0620 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7a100a4440 tx=0x7f7a100a4af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4af98dc1602ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f79feffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a100b8070 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7a10010640 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.988+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a10098420 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7a1007abd0 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 10 client.17544 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a2e38c4100 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a10098420 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7a10086050 con 0x55a2e38abb20 2024-04-22T00:07:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 10 client.17544 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 10 client.17544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 20 client.17544 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e38c5450 0x55a2e38c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 4 client.17544 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2e381d0b0 con 0x55a2e38c5450 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.989+0000 7f7a1effd640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e38c5450 0x55a2e38c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.990+0000 7f79feffd640 10 client.17544 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.990+0000 7f7a1effd640 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e38c5450 0x55a2e38c7840 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a1063ddcc0f284 server_cookie=c863a430e9d5e734 in_seq=0 out_seq=0 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.990+0000 7f79feffd640 1 -- 172.21.15.38:0/1069056031 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2e381d0b0 con 0x55a2e38c5450 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.990+0000 7f79feffd640 10 client.17544 handle_command_reply: tid=0 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 client.17544 shutdown 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 msgr2=0x55a2e38b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7a100a4440 tx=0x7f7a100a4af0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 shutdown_connections 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e38b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e38c5450 0x55a2e38c7840 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f79feffd640 0 client.17544 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1069056031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38b2300 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 >> 172.21.15.38:0/1069056031 conn(0x55a2e38a6440 msgr2=0x55a2e38a8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 shutdown_connections 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 1 -- 172.21.15.38:0/1069056031 wait complete. 2024-04-22T00:07:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:58.991+0000 7f7a287c0740 20 client.17544 trim_cache size 0 max 16384 2024-04-22T00:07:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 Processor -- start 2024-04-22T00:07:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 -- start start 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 0x55a2e38b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38b5210 0x55a2e38afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201b0d70 con 0x55a2e38b5210 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201b28b0 con 0x55a2e38ac440 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a287c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201b1440 con 0x55a2e38abb20 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 0x55a2e38b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38b5210 0x55a2e38afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:59030/0 (socket says 172.21.15.38:59030) 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 0x55a2e38b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:47894/0 (socket says 172.21.15.38:47894) 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.011+0000 7f7a1effd640 1 -- 172.21.15.38:0/4182977164 learned_addr learned my addr 172.21.15.38:0/4182977164 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.012+0000 7f7a1effd640 1 -- 172.21.15.38:0/4182977164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 msgr2=0x55a2e38b4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.012+0000 7f7a1effd640 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 0x55a2e38b4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.012+0000 7f7a1effd640 1 -- 172.21.15.38:0/4182977164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38b5210 msgr2=0x55a2e38afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.012+0000 7f7a1effd640 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38b5210 0x55a2e38afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.012+0000 7f7a1effd640 1 -- 172.21.15.38:0/4182977164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2e38add40 con 0x55a2e38ac440 2024-04-22T00:07:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.013+0000 7f7a1effd640 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a1402c370 tx=0x7f7a14042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25428eb0d4423a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.014+0000 7f7a057fa640 1 -- 172.21.15.38:0/4182977164 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1405a070 con 0x55a2e38ac440 2024-04-22T00:07:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.014+0000 7f7a057fa640 1 -- 172.21.15.38:0/4182977164 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a14054d90 con 0x55a2e38ac440 2024-04-22T00:07:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.014+0000 7f7a057fa640 1 -- 172.21.15.38:0/4182977164 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a14054790 con 0x55a2e38ac440 2024-04-22T00:07:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.015+0000 7f7a287c0740 1 -- 172.21.15.38:0/4182977164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 msgr2=0x55a2e38af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.015+0000 7f7a287c0740 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a1402c370 tx=0x7f7a14042000 comp rx=0 tx=0).stop 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.016+0000 7f7a287c0740 1 -- 172.21.15.38:0/4182977164 shutdown_connections 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38b5210 0x55a2e38afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38af580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 --2- 172.21.15.38:0/4182977164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38abb20 0x55a2e38b4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 -- 172.21.15.38:0/4182977164 >> 172.21.15.38:0/4182977164 conn(0x55a2e38a6440 msgr2=0x55a2e38ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 -- 172.21.15.38:0/4182977164 shutdown_connections 2024-04-22T00:07:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 -- 172.21.15.38:0/4182977164 wait complete. 2024-04-22T00:07:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.017+0000 7f7a287c0740 1 -- start start 2024-04-22T00:07:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.018+0000 7f7a287c0740 1 Processor -- start 2024-04-22T00:07:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.018+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e3932e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201b21e0 con 0x55a2e38abb20 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201b1b10 con 0x55a2e38ac440 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a201aed50 con 0x55a2e38b5210 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.019+0000 7f7a287c0740 10 client.0 _ll_register_callbacks cb 0x7f7a18009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e3932e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38570/0 (socket says 172.21.15.38:38570) 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1f7fe640 1 -- 172.21.15.38:0/1484773107 learned_addr learned my addr 172.21.15.38:0/1484773107 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:07:59.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1effd640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 -- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 msgr2=0x55a2e38c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 -- 172.21.15.38:0/1484773107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 msgr2=0x55a2e38c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:07:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.020+0000 7f7a1ffff640 1 -- 172.21.15.38:0/1484773107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2e39333d0 con 0x55a2e38b5210 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f7a1ffff640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e3932e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55a2e38a1470 tx=0x7f7a18003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78704a9dc6a62f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f79feffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18022060 con 0x55a2e38b5210 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f7a18012b20 con 0x55a2e38b5210 2024-04-22T00:07:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a14054d90 con 0x55a2e38b5210 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7a18029050 con 0x55a2e38b5210 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f7a287c0740 10 client.26758 fetch_fsmap learned FSMap version 35 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.021+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a2e3934740 con 0x55a2e38b5210 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a14054d90 con 0x55a2e38b5210 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7a1801c3e0 con 0x55a2e38b5210 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 10 client.26758 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 10 client.26758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 20 client.26758 populate_metadata read hostname 'smithi038' 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e3934ec0 0x55a2e39372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:07:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 4 client.26758 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:07:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2e3817400 con 0x55a2e3934ec0 2024-04-22T00:07:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.022+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e3934ec0 0x55a2e39372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:07:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.023+0000 7f79feffd640 10 client.26758 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.023+0000 7f7a1f7fe640 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e3934ec0 0x55a2e39372e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be74d8d2d8d9abe server_cookie=5ba3e0c4d9652c8b in_seq=0 out_seq=0 2024-04-22T00:07:59.109 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f79feffd640 1 -- 172.21.15.38:0/1484773107 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2e3817400 con 0x55a2e3934ec0 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f79feffd640 10 client.26758 handle_command_reply: tid=0 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 client.26758 shutdown 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 msgr2=0x55a2e3932e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e3932e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55a2e38a1470 tx=0x7f7a18003f90 comp rx=0 tx=0).stop 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 shutdown_connections 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a2e38b5210 0x55a2e3932e40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a2e38ac440 0x55a2e38c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a2e3934ec0 0x55a2e39372e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f79feffd640 0 client.26758 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 --2- 172.21.15.38:0/1484773107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a2e38abb20 0x55a2e38c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.112+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 >> 172.21.15.38:0/1484773107 conn(0x55a2e38a6440 msgr2=0x55a2e38a4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.113+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 shutdown_connections 2024-04-22T00:07:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.113+0000 7f7a287c0740 1 -- 172.21.15.38:0/1484773107 wait complete. 2024-04-22T00:07:59.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.113+0000 7f7a287c0740 20 client.26758 trim_cache size 0 max 16384 2024-04-22T00:07:59.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:07:59.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:07:59.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:07:59.111 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:07:59.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 msgr2=0x7f7a080a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:59.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 0x7f7a080a5260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a140096c0 tx=0x7f7a14005a20 comp rx=0 tx=0).stop 2024-04-22T00:07:59.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 msgr2=0x7f7a20100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:07:59.112 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20100620 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e9f0 tx=0x7f7a1000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 shutdown_connections 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a080a2da0 0x7f7a080a5260 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a2010bd10 0x7f7a201010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a201032f0 0x7f7a20100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 --2- 172.21.15.38:0/113869806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a201029d0 0x7f7a20100620 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.116+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 >> 172.21.15.38:0/113869806 conn(0x7f7a200777c0 msgr2=0x7f7a200fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.117+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 shutdown_connections 2024-04-22T00:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:07:59.117+0000 7f7a25afb640 1 -- 172.21.15.38:0/113869806 wait complete. 2024-04-22T00:08:00.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:00 smithi038 ceph-mon[27972]: pgmap v180: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.90k op/s 2024-04-22T00:08:00.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:00 smithi203 ceph-mon[31431]: pgmap v180: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.90k op/s 2024-04-22T00:08:00.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:00 smithi111 ceph-mon[31543]: pgmap v180: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 2.90k op/s 2024-04-22T00:08:02.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:02 smithi038 ceph-mon[27972]: pgmap v181: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 2.43k op/s 2024-04-22T00:08:02.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:02 smithi203 ceph-mon[31431]: pgmap v181: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 2.43k op/s 2024-04-22T00:08:02.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:02 smithi111 ceph-mon[31543]: pgmap v181: 129 pgs: 129 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 9.7 MiB/s wr, 2.43k op/s 2024-04-22T00:08:04.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:04 smithi038 ceph-mon[27972]: pgmap v182: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.44k op/s 2024-04-22T00:08:04.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:04 smithi203 ceph-mon[31431]: pgmap v182: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.44k op/s 2024-04-22T00:08:04.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:04 smithi111 ceph-mon[31543]: pgmap v182: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.44k op/s 2024-04-22T00:08:05.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:08:05.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:08:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 -- 172.21.15.38:0/2599884700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 msgr2=0x7f1204101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 --2- 172.21.15.38:0/2599884700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f1204101b80 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009930 tx=0x7f11f402bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 -- 172.21.15.38:0/2599884700 shutdown_connections 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 --2- 172.21.15.38:0/2599884700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12041020c0 0x7f120410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 --2- 172.21.15.38:0/2599884700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f1204101b80 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 --2- 172.21.15.38:0/2599884700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1204107710 0x7f1204107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 -- 172.21.15.38:0/2599884700 >> 172.21.15.38:0/2599884700 conn(0x7f12040fd1b0 msgr2=0x7f12040ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 -- 172.21.15.38:0/2599884700 shutdown_connections 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.554+0000 7f120c257640 1 -- 172.21.15.38:0/2599884700 wait complete. 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 Processor -- start 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 -- start start 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12041020c0 0x7f1204198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1204107710 0x7f120419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1204111b70 con 0x7f1204101700 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f12041119f0 con 0x7f12041020c0 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120c257640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1204111cf0 con 0x7f1204107710 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60914/0 (socket says 172.21.15.38:60914) 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 -- 172.21.15.38:0/2414201226 learned_addr learned my addr 172.21.15.38:0/2414201226 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f12097f2640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12041020c0 0x7f1204198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f120a7f4640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1204107710 0x7f120419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 -- 172.21.15.38:0/2414201226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1204107710 msgr2=0x7f120419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1204107710 0x7f120419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 -- 172.21.15.38:0/2414201226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12041020c0 msgr2=0x7f1204198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12041020c0 0x7f1204198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 -- 172.21.15.38:0/2414201226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120419d0c0 con 0x7f1204101700 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.555+0000 7f1209ff3640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f120000d950 tx=0x7f120000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.556+0000 7f11faffd640 1 -- 172.21.15.38:0/2414201226 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200017030 con 0x7f1204101700 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.556+0000 7f11faffd640 1 -- 172.21.15.38:0/2414201226 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1200004020 con 0x7f1204101700 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.556+0000 7f11faffd640 1 -- 172.21.15.38:0/2414201226 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12000047a0 con 0x7f1204101700 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.556+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120419d3b0 con 0x7f1204101700 2024-04-22T00:08:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.556+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12041a4c40 con 0x7f1204101700 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f11faffd640 1 -- 172.21.15.38:0/2414201226 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f12000041d0 con 0x7f1204101700 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 Processor -- start 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 -- start start 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e73b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e743ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56264e69f560 con 0x56264e744430 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56264e647730 con 0x56264e73af30 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120ee3d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56264e69b520 con 0x56264e73b850 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f12097f2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f12097f2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60918/0 (socket says 172.21.15.38:60918) 2024-04-22T00:08:05.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f12097f2640 1 -- 172.21.15.38:0/723313680 learned_addr learned my addr 172.21.15.38:0/723313680 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f120a7f4640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e73b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.559+0000 7f1209ff3640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e743ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 -- 172.21.15.38:0/723313680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 msgr2=0x56264e743ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e743ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f11faffd640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 0x7f11f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 -- 172.21.15.38:0/723313680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 msgr2=0x56264e746820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 -- 172.21.15.38:0/723313680 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264e47d7d0 con 0x56264e73af30 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f12097f2640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f11faffd640 1 -- 172.21.15.38:0/2414201226 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f12000cf030 con 0x7f1204101700 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e73b310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009f30 tx=0x7f11fc010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c71db0edc072d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.560+0000 7f120a7f4640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 0x7f11f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.561+0000 7f120a7f4640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 0x7f11f00a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11fc011010 tx=0x7f11fc00b000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.561+0000 7f11effff640 1 -- 172.21.15.38:0/723313680 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc022030 con 0x56264e73af30 2024-04-22T00:08:05.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.561+0000 7f11effff640 1 -- 172.21.15.38:0/723313680 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11fc012da0 con 0x56264e73af30 2024-04-22T00:08:05.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.561+0000 7f11effff640 1 -- 172.21.15.38:0/723313680 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc016070 con 0x56264e73af30 2024-04-22T00:08:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.562+0000 7f120ee3d740 1 -- 172.21.15.38:0/723313680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 msgr2=0x56264e73b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.562+0000 7f120ee3d740 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e73b310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009f30 tx=0x7f11fc010800 comp rx=0 tx=0).stop 2024-04-22T00:08:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 -- 172.21.15.38:0/723313680 shutdown_connections 2024-04-22T00:08:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e746820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e743ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 --2- 172.21.15.38:0/723313680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e73b310 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 -- 172.21.15.38:0/723313680 >> 172.21.15.38:0/723313680 conn(0x56264e735660 msgr2=0x56264e735a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 -- 172.21.15.38:0/723313680 shutdown_connections 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.563+0000 7f120ee3d740 1 -- 172.21.15.38:0/723313680 wait complete. 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 -- start start 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 Processor -- start 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 0x56264e74e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e748ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56264e747220 con 0x56264e744430 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56264e65ae00 con 0x56264e73b850 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56264e7473a0 con 0x56264e73af30 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120ee3d740 10 client.0 _ll_register_callbacks cb 0x56264e695eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f12097f2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f1209ff3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 0x56264e74e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f12097f2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58468/0 (socket says 172.21.15.38:58468) 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f12097f2640 1 -- 172.21.15.38:0/4059593061 learned_addr learned my addr 172.21.15.38:0/4059593061 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.564+0000 7f120a7f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e748ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 -- 172.21.15.38:0/4059593061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 msgr2=0x56264e74e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 0x56264e74e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 -- 172.21.15.38:0/4059593061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 msgr2=0x56264e748ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e748ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 -- 172.21.15.38:0/4059593061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264e3b33f0 con 0x56264e73b850 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f1209ff3640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 0x56264e74e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f12097f2640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009230 tx=0x7f11f4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8569d363b7c559a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f11ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f402cc10 con 0x56264e73b850 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f11f40046c0 con 0x56264e73b850 2024-04-22T00:08:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11fc012da0 con 0x56264e73b850 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f11f40048c0 con 0x56264e73b850 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f120ee3d740 10 client.26927 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.565+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56264e3730f0 con 0x56264e73b850 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11fc012da0 con 0x56264e73b850 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f11f4003680 con 0x56264e73b850 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 10 client.26927 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 20 client.26927 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e758c40 0x56264e75b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 4 client.26927 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56264e6acaa0 con 0x56264e758c40 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.566+0000 7f1209ff3640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e758c40 0x56264e75b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f11ed7fa640 10 client.26927 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f1209ff3640 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e758c40 0x56264e75b030 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b194f0c10c431e22 server_cookie=1c6ea54ed69394a7 in_seq=0 out_seq=0 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f11ed7fa640 1 -- 172.21.15.38:0/4059593061 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56264e6acaa0 con 0x56264e758c40 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f11ed7fa640 10 client.26927 handle_command_reply: tid=0 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 client.26927 shutdown 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 msgr2=0x56264e74ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009230 tx=0x7f11f4009680 comp rx=0 tx=0).stop 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 shutdown_connections 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e758c40 0x56264e75b030 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e748ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f11ed7fa640 0 client.26927 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e74ec10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 --2- 172.21.15.38:0/4059593061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73af30 0x56264e74e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.567+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 >> 172.21.15.38:0/4059593061 conn(0x56264e735660 msgr2=0x56264e7373b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.568+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 shutdown_connections 2024-04-22T00:08:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.568+0000 7f120ee3d740 1 -- 172.21.15.38:0/4059593061 wait complete. 2024-04-22T00:08:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.568+0000 7f120ee3d740 20 client.26927 trim_cache size 0 max 16384 2024-04-22T00:08:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 Processor -- start 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 -- start start 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e7608c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e760e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f12041ac9e0 con 0x56264e744430 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f12041ae520 con 0x56264e73af30 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120ee3d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12041ad0b0 con 0x56264e73b850 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60928/0 (socket says 172.21.15.38:60928) 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 -- 172.21.15.38:0/3325139893 learned_addr learned my addr 172.21.15.38:0/3325139893 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f12097f2640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e760e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f1209ff3640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e7608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 -- 172.21.15.38:0/3325139893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 msgr2=0x56264e760e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e760e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 -- 172.21.15.38:0/3325139893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 msgr2=0x56264e7608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.588+0000 7f120a7f4640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e7608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f120a7f4640 1 -- 172.21.15.38:0/3325139893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264e75b3a0 con 0x56264e744430 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f12097f2640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e760e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f120a7f4640 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f11fc01c720 tx=0x7f11fc01c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10351205d7cde40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f11ef7fe640 1 -- 172.21.15.38:0/3325139893 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc035070 con 0x56264e744430 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f11ef7fe640 1 -- 172.21.15.38:0/3325139893 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11fc003e50 con 0x56264e744430 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.589+0000 7f11ef7fe640 1 -- 172.21.15.38:0/3325139893 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc016070 con 0x56264e744430 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- 172.21.15.38:0/3325139893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 msgr2=0x56264e742b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f11fc01c720 tx=0x7f11fc01c6b0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- 172.21.15.38:0/3325139893 shutdown_connections 2024-04-22T00:08:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e744430 0x56264e742b10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e73b850 0x56264e760e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 --2- 172.21.15.38:0/3325139893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73af30 0x56264e7608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- 172.21.15.38:0/3325139893 >> 172.21.15.38:0/3325139893 conn(0x56264e735660 msgr2=0x56264e748370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- 172.21.15.38:0/3325139893 shutdown_connections 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- 172.21.15.38:0/3325139893 wait complete. 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 -- start start 2024-04-22T00:08:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.590+0000 7f120ee3d740 1 Processor -- start 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e7529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e744430 0x56264e752f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f12041ade50 con 0x56264e73af30 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f12041ad780 con 0x56264e73b850 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12041aa9c0 con 0x56264e744430 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 10 client.0 _ll_register_callbacks cb 0x7f11f4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f1209ff3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f1209ff3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60930/0 (socket says 172.21.15.38:60930) 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e7529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f1209ff3640 1 -- 172.21.15.38:0/1990733113 learned_addr learned my addr 172.21.15.38:0/1990733113 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120a7f4640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e744430 0x56264e752f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 -- 172.21.15.38:0/1990733113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e744430 msgr2=0x56264e752f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e744430 0x56264e752f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 -- 172.21.15.38:0/1990733113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 msgr2=0x56264e752480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 -- 172.21.15.38:0/1990733113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264e7be170 con 0x56264e73b850 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f1209ff3640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f12097f2640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e7529c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009850 tx=0x7f11f4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615ebd4f2043d1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f11ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40044d0 con 0x56264e73b850 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f11f4003800 con 0x56264e73b850 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.591+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11fc003e50 con 0x56264e73b850 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f11f4065050 con 0x56264e73b850 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 10 client.26933 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56264e7bf440 con 0x56264e73b850 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11fc003e50 con 0x56264e73b850 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f11f4060030 con 0x56264e73b850 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 10 client.26933 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 20 client.26933 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e7bfc40 0x56264e7c2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 4 client.26933 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56264e6a6df0 con 0x56264e7bfc40 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.592+0000 7f1209ff3640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e7bfc40 0x56264e7c2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f1209ff3640 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e7bfc40 0x56264e7c2060 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b382fd0ceb5d9215 server_cookie=22bd2f40d9e1a69 in_seq=0 out_seq=0 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f11ed7fa640 10 client.26933 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f11ed7fa640 1 -- 172.21.15.38:0/1990733113 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56264e6a6df0 con 0x56264e7bfc40 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f11ed7fa640 10 client.26933 handle_command_reply: tid=0 2024-04-22T00:08:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 client.26933 shutdown 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 msgr2=0x56264e7529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e7529c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009850 tx=0x7f11f4000bc0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 shutdown_connections 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56264e744430 0x56264e752f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56264e73b850 0x56264e7529c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56264e7bfc40 0x56264e7c2060 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 --2- 172.21.15.38:0/1990733113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56264e73af30 0x56264e752480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f11ed7fa640 0 client.26933 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 >> 172.21.15.38:0/1990733113 conn(0x56264e735660 msgr2=0x56264e74e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 shutdown_connections 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 1 -- 172.21.15.38:0/1990733113 wait complete. 2024-04-22T00:08:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.593+0000 7f120ee3d740 20 client.26933 trim_cache size 0 max 16384 2024-04-22T00:08:05.596 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:05.596 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:08:05.596 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:08:05.596 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 msgr2=0x7f11f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 0x7f11f00a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f11fc011010 tx=0x7f11fc00b000 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 msgr2=0x7f12041980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f120000d950 tx=0x7f120000de10 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 shutdown_connections 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f11f00a2d00 0x7f11f00a51c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1204107710 0x7f120419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12041020c0 0x7f1204198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 --2- 172.21.15.38:0/2414201226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1204101700 0x7f12041980d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 >> 172.21.15.38:0/2414201226 conn(0x7f12040fd1b0 msgr2=0x7f12040fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 shutdown_connections 2024-04-22T00:08:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.594+0000 7f120c257640 1 -- 172.21.15.38:0/2414201226 wait complete. 2024-04-22T00:08:05.600 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:08:05.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 -- 172.21.15.38:0/719955916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff404102160 msgr2=0x7ff40410e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/719955916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff404102160 0x7ff40410e5a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009ca0 tx=0x7ff3fc02c180 comp rx=0 tx=0).stop 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 -- 172.21.15.38:0/719955916 shutdown_connections 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/719955916 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff404102160 0x7ff40410e5a0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/719955916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4041017c0 0x7ff404101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/719955916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 0x7ff404107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 -- 172.21.15.38:0/719955916 >> 172.21.15.38:0/719955916 conn(0x7ff4040788d0 msgr2=0x7ff404078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:05.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 -- 172.21.15.38:0/719955916 shutdown_connections 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.710+0000 7ff40b5f7640 1 -- 172.21.15.38:0/719955916 wait complete. 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 Processor -- start 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 -- start start 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff404102160 0x7ff404198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 0x7ff40419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff404111b10 con 0x7ff4041017c0 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff404111990 con 0x7ff404107750 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff40b5f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff404111c90 con 0x7ff404102160 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff409393640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff409393640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60950/0 (socket says 172.21.15.38:60950) 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff409393640 1 -- 172.21.15.38:0/2929574937 learned_addr learned my addr 172.21.15.38:0/2929574937 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.711+0000 7ff408b92640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff404102160 0x7ff404198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409b94640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 0x7ff40419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 -- 172.21.15.38:0/2929574937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff404102160 msgr2=0x7ff404198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff404102160 0x7ff404198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 -- 172.21.15.38:0/2929574937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 msgr2=0x7ff40419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 0x7ff40419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 -- 172.21.15.38:0/2929574937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40419d000 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff409393640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff40000e9f0 tx=0x7ff40000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40000cd80 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff400010910 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40419d2f0 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400018990 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.712+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4041a4b80 con 0x7ff4041017c0 2024-04-22T00:08:05.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.713+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4040689b0 con 0x7ff4041017c0 2024-04-22T00:08:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.714+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff400010ac0 con 0x7ff4041017c0 2024-04-22T00:08:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.715+0000 7ff3fa7fc640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 0x7ff3f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.715+0000 7ff408b92640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 0x7ff3f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.715+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff4000cfe70 con 0x7ff4041017c0 2024-04-22T00:08:05.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.715+0000 7ff408b92640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 0x7ff3f00a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff404078ea0 tx=0x7ff3f400b410 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:05.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:05.718+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff40009c750 con 0x7ff4041017c0 2024-04-22T00:08:06.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.056+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff404199360 con 0x7ff4041017c0 2024-04-22T00:08:06.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.059+0000 7ff3fa7fc640 1 -- 172.21.15.38:0/2929574937 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7ff40009c050 con 0x7ff4041017c0 2024-04-22T00:08:06.055 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:08:06.055 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:08:06.055 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:08:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 msgr2=0x7ff3f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 0x7ff3f00a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff404078ea0 tx=0x7ff3f400b410 comp rx=0 tx=0).stop 2024-04-22T00:08:06.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 msgr2=0x7ff404198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff40000e9f0 tx=0x7ff40000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 shutdown_connections 2024-04-22T00:08:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3f00a2d00 0x7ff3f00a51c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff404107750 0x7ff40419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff404102160 0x7ff404198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 --2- 172.21.15.38:0/2929574937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4041017c0 0x7ff404198010 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 >> 172.21.15.38:0/2929574937 conn(0x7ff4040788d0 msgr2=0x7ff404100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 shutdown_connections 2024-04-22T00:08:06.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.063+0000 7ff40b5f7640 1 -- 172.21.15.38:0/2929574937 wait complete. 2024-04-22T00:08:06.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:08:06.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 -- 172.21.15.38:0/1354901597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c1066b0 msgr2=0x7fb86c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 --2- 172.21.15.38:0/1354901597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c1066b0 0x7fb86c106a90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fb86800fb40 tx=0x7fb86802fdb0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 -- 172.21.15.38:0/1354901597 shutdown_connections 2024-04-22T00:08:06.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 --2- 172.21.15.38:0/1354901597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 0x7fb86c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 --2- 172.21.15.38:0/1354901597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c100610 0x7fb86c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 --2- 172.21.15.38:0/1354901597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c1066b0 0x7fb86c106a90 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.244+0000 7fb8738a7640 1 -- 172.21.15.38:0/1354901597 >> 172.21.15.38:0/1354901597 conn(0x7fb86c077740 msgr2=0x7fb86c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.245+0000 7fb8738a7640 1 -- 172.21.15.38:0/1354901597 shutdown_connections 2024-04-22T00:08:06.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.245+0000 7fb8738a7640 1 -- 172.21.15.38:0/1354901597 wait complete. 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.245+0000 7fb8738a7640 1 Processor -- start 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.245+0000 7fb8738a7640 1 -- start start 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 0x7fb86c1962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 0x7fb86c19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c10f920 con 0x7fb86c100610 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c10f7a0 con 0x7fb86c1066b0 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb8738a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c10faa0 con 0x7fb86c100fd0 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60978/0 (socket says 172.21.15.38:60978) 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 -- 172.21.15.38:0/2297067842 learned_addr learned my addr 172.21.15.38:0/2297067842 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb870e42640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 0x7fb86c1962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871e44640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 0x7fb86c19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 -- 172.21.15.38:0/2297067842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 msgr2=0x7fb86c1962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 0x7fb86c1962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 -- 172.21.15.38:0/2297067842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 msgr2=0x7fb86c19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 0x7fb86c19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871643640 1 -- 172.21.15.38:0/2297067842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86c19ad50 con 0x7fb86c100610 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.246+0000 7fb871e44640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 0x7fb86c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb871643640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb868000c00 tx=0x7fb86800f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb8627fc640 1 -- 172.21.15.38:0/2297067842 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868005250 con 0x7fb86c100610 2024-04-22T00:08:06.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86c19afe0 con 0x7fb86c100610 2024-04-22T00:08:06.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb8627fc640 1 -- 172.21.15.38:0/2297067842 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8680053f0 con 0x7fb86c100610 2024-04-22T00:08:06.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb86c1a28d0 con 0x7fb86c100610 2024-04-22T00:08:06.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.247+0000 7fb8627fc640 1 -- 172.21.15.38:0/2297067842 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868004cc0 con 0x7fb86c100610 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.250+0000 7fb87656b740 1 Processor -- start 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.250+0000 7fb87656b740 1 -- start start 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb8627fc640 1 -- 172.21.15.38:0/2297067842 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb868049020 con 0x7fb86c100610 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db567400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db56ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e5db4cb530 con 0x55e5db567020 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e5db473830 con 0x55e5db570520 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb87656b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5db4c74f0 con 0x55e5db567940 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb871e44640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db567400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb871643640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db56ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58532/0 (socket says 172.21.15.38:58532) 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 -- 172.21.15.38:0/3577014431 learned_addr learned my addr 172.21.15.38:0/3577014431 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb871643640 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db56ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:33978/0 (socket says 172.21.15.38:33978) 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 -- 172.21.15.38:0/3577014431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 msgr2=0x55e5db56ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db56ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 -- 172.21.15.38:0/3577014431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 msgr2=0x55e5db567400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db567400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.251+0000 7fb870e42640 1 -- 172.21.15.38:0/3577014431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5db4972c0 con 0x55e5db570520 2024-04-22T00:08:06.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb870e42640 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00bc90 tx=0x7fb85c00b3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bca4d4b2e096f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb8577fe640 1 -- 172.21.15.38:0/3577014431 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004230 con 0x55e5db570520 2024-04-22T00:08:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb8577fe640 1 -- 172.21.15.38:0/3577014431 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb85c0043d0 con 0x55e5db570520 2024-04-22T00:08:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb8577fe640 1 -- 172.21.15.38:0/3577014431 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004230 con 0x55e5db570520 2024-04-22T00:08:06.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb8627fc640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 0x7fb8580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb8627fc640 1 -- 172.21.15.38:0/2297067842 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb8680f4c30 con 0x7fb86c100610 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.252+0000 7fb871e44640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 0x7fb8580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- 172.21.15.38:0/3577014431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 msgr2=0x55e5db572910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00bc90 tx=0x7fb85c00b3f0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- 172.21.15.38:0/3577014431 shutdown_connections 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db572910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db56ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 --2- 172.21.15.38:0/3577014431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db567400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- 172.21.15.38:0/3577014431 >> 172.21.15.38:0/3577014431 conn(0x55e5db561750 msgr2=0x55e5db561b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb871e44640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 0x7fb8580a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb864005fd0 tx=0x7fb864009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- 172.21.15.38:0/3577014431 shutdown_connections 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- 172.21.15.38:0/3577014431 wait complete. 2024-04-22T00:08:06.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 -- start start 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.253+0000 7fb87656b740 1 Processor -- start 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 0x55e5db56d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e5db573360 con 0x55e5db567020 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e5db486f10 con 0x55e5db567940 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5db5734e0 con 0x55e5db570520 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb87656b740 10 client.0 _ll_register_callbacks cb 0x55e5db1debe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32772/0 (socket says 172.21.15.38:32772) 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb871643640 1 -- 172.21.15.38:0/1309142896 learned_addr learned my addr 172.21.15.38:0/1309142896 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb870e42640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 0x55e5db56d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb871e44640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb870e42640 1 -- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 msgr2=0x55e5db56d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.253 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb870e42640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb870e42640 1 -- 172.21.15.38:0/1309142896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 msgr2=0x55e5db56fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.254+0000 7fb870e42640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb870e42640 1 -- 172.21.15.38:0/1309142896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5db458c80 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb871643640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb870e42640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 0x55e5db56d160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00bc90 tx=0x7fb85c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f0b8ad7a167473 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c004230 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb85c0043d0 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb85c005260 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb85c032050 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb87656b740 10 client.26951 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e5db1df3f0 con 0x55e5db567940 2024-04-22T00:08:06.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.255+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb85c0043d0 con 0x55e5db567940 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb85c004cc0 con 0x55e5db567940 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 10 client.26951 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 10 client.26951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 20 client.26951 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db580440 0x55e5db582830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 4 client.26951 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5db4d8a70 con 0x55e5db580440 2024-04-22T00:08:06.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.256+0000 7fb871643640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db580440 0x55e5db582830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.257+0000 7fb871643640 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db580440 0x55e5db582830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a3c89e86011f19ed server_cookie=ce1fbe47538efb17 in_seq=0 out_seq=0 2024-04-22T00:08:06.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.257+0000 7fb854ff9640 10 client.26951 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:06.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb854ff9640 1 -- 172.21.15.38:0/1309142896 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5db4d8a70 con 0x55e5db580440 2024-04-22T00:08:06.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb854ff9640 10 client.26951 handle_command_reply: tid=0 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 client.26951 shutdown 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 msgr2=0x55e5db56d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 0x55e5db56d160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00bc90 tx=0x7fb85c00bea0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 shutdown_connections 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db580440 0x55e5db582830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567940 0x55e5db56d160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 --2- 172.21.15.38:0/1309142896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db56fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 >> 172.21.15.38:0/1309142896 conn(0x55e5db561750 msgr2=0x55e5db5634a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb854ff9640 0 client.26951 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 shutdown_connections 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 1 -- 172.21.15.38:0/1309142896 wait complete. 2024-04-22T00:08:06.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.258+0000 7fb87656b740 20 client.26951 trim_cache size 0 max 16384 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 Processor -- start 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 -- start start 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567940 0x55e5db568690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1ac870 con 0x55e5db567940 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb871643640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1ae3b0 con 0x55e5db567020 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb87656b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1acf40 con 0x55e5db570520 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb870e42640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567940 0x55e5db568690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb871643640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58554/0 (socket says 172.21.15.38:58554) 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb871643640 1 -- 172.21.15.38:0/3048912099 learned_addr learned my addr 172.21.15.38:0/3048912099 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.276+0000 7fb871e44640 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 -- 172.21.15.38:0/3048912099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 msgr2=0x55e5db56f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 -- 172.21.15.38:0/3048912099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567940 msgr2=0x55e5db568690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567940 0x55e5db568690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 -- 172.21.15.38:0/3048912099 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5db2a97d0 con 0x55e5db567020 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871e44640 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb871643640 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb8680f2640 tx=0x7fb8680f8780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fed607cfafe189a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb856ffd640 1 -- 172.21.15.38:0/3048912099 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8680bd1c0 con 0x55e5db567020 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb856ffd640 1 -- 172.21.15.38:0/3048912099 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb868034040 con 0x55e5db567020 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.277+0000 7fb856ffd640 1 -- 172.21.15.38:0/3048912099 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868004e60 con 0x55e5db567020 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- 172.21.15.38:0/3048912099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 msgr2=0x55e5db57c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb8680f2640 tx=0x7fb8680f8780 comp rx=0 tx=0).stop 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- 172.21.15.38:0/3048912099 shutdown_connections 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db570520 0x55e5db56f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567940 0x55e5db568690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 --2- 172.21.15.38:0/3048912099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db567020 0x55e5db57c220 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- 172.21.15.38:0/3048912099 >> 172.21.15.38:0/3048912099 conn(0x55e5db561750 msgr2=0x55e5db567f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- 172.21.15.38:0/3048912099 shutdown_connections 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- 172.21.15.38:0/3048912099 wait complete. 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.278+0000 7fb87656b740 1 -- start start 2024-04-22T00:08:06.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb87656b740 1 Processor -- start 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb87656b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb87656b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db578c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb871643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32790/0 (socket says 172.21.15.38:32790) 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb87656b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db5693d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.279+0000 7fb871643640 1 -- 172.21.15.38:0/1904407302 learned_addr learned my addr 172.21.15.38:0/1904407302 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1adce0 con 0x55e5db567020 2024-04-22T00:08:06.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1ad610 con 0x55e5db570520 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb86c1aa850 con 0x55e5db567940 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb870e42640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db578c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871e44640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db5693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 -- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 msgr2=0x55e5db578c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db578c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 -- 172.21.15.38:0/1904407302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 msgr2=0x55e5db5693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db5693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5db569910 con 0x55e5db567020 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb87656b740 10 client.0 _ll_register_callbacks cb 0x7fb8640099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb868034040 con 0x55e5db567020 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871e44640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db5693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb871643640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb868006010 tx=0x7fb8680bd1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aaf65197771b015 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb854ff9640 10 client.17592 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb854ff9640 1 -- 172.21.15.38:0/1904407302 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8680c0880 con 0x55e5db567020 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb854ff9640 1 -- 172.21.15.38:0/1904407302 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb8680f22c0 con 0x55e5db567020 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.280+0000 7fb854ff9640 1 -- 172.21.15.38:0/1904407302 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb8680c0a60 con 0x55e5db567020 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 10 client.17592 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55e5db5e5d50 con 0x55e5db567020 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb854ff9640 1 -- 172.21.15.38:0/1904407302 <== mon.0 v2:172.21.15.38:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb8680c4020 con 0x55e5db567020 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 10 client.17592 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 10 client.17592 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 20 client.17592 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db5e6370 0x55e5db5e87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 4 client.17592 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5db56da30 con 0x55e5db5e6370 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb870e42640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db5e6370 0x55e5db5e87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb870e42640 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db5e6370 0x55e5db5e87d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c77198b8c801343 server_cookie=499cbf197d1a9bfa in_seq=0 out_seq=0 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.281+0000 7fb854ff9640 10 client.17592 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb854ff9640 1 -- 172.21.15.38:0/1904407302 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5db56da30 con 0x55e5db5e6370 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb854ff9640 10 client.17592 handle_command_reply: tid=0 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 client.17592 shutdown 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 msgr2=0x55e5db578730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb868006010 tx=0x7fb8680bd1c0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 shutdown_connections 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e5db5e6370 0x55e5db5e87d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e5db570520 0x55e5db5693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5db567940 0x55e5db578c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb854ff9640 0 client.17592 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 --2- 172.21.15.38:0/1904407302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e5db567020 0x55e5db578730 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 >> 172.21.15.38:0/1904407302 conn(0x55e5db561750 msgr2=0x55e5db568750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 shutdown_connections 2024-04-22T00:08:06.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 1 -- 172.21.15.38:0/1904407302 wait complete. 2024-04-22T00:08:06.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.282+0000 7fb87656b740 20 client.17592 trim_cache size 0 max 16384 2024-04-22T00:08:06.284 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 msgr2=0x7fb8580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 0x7fb8580a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb864005fd0 tx=0x7fb864009500 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 msgr2=0x7fb86c195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb868000c00 tx=0x7fb86800f480 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 shutdown_connections 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb8580a2d00 0x7fb8580a51c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb86c1066b0 0x7fb86c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb86c100fd0 0x7fb86c1962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 --2- 172.21.15.38:0/2297067842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb86c100610 0x7fb86c195d60 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 >> 172.21.15.38:0/2297067842 conn(0x7fb86c077740 msgr2=0x7fb86c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 shutdown_connections 2024-04-22T00:08:06.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.283+0000 7fb8738a7640 1 -- 172.21.15.38:0/2297067842 wait complete. 2024-04-22T00:08:06.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:08:06.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 -- 172.21.15.38:0/2767657580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40100fe0 msgr2=0x7faa4010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 --2- 172.21.15.38:0/2767657580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40100fe0 0x7faa4010d4f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7faa38009cd0 tx=0x7faa3802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 -- 172.21.15.38:0/2767657580 shutdown_connections 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 --2- 172.21.15.38:0/2767657580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40100fe0 0x7faa4010d4f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 --2- 172.21.15.38:0/2767657580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 0x7faa40100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 --2- 172.21.15.38:0/2767657580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40106630 0x7faa40106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 -- 172.21.15.38:0/2767657580 >> 172.21.15.38:0/2767657580 conn(0x7faa400776a0 msgr2=0x7faa40077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 -- 172.21.15.38:0/2767657580 shutdown_connections 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.428+0000 7faa473e5640 1 -- 172.21.15.38:0/2767657580 wait complete. 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 Processor -- start 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 -- start start 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 0x7faa40198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 0x7faa4019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa40111b40 con 0x7faa40106630 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa401119c0 con 0x7faa40100620 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa473e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa40111cc0 con 0x7faa40100fe0 2024-04-22T00:08:06.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa45982640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 0x7faa4019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34018/0 (socket says 172.21.15.38:34018) 2024-04-22T00:08:06.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 -- 172.21.15.38:0/2517256060 learned_addr learned my addr 172.21.15.38:0/2517256060 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa45181640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 0x7faa40198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 -- 172.21.15.38:0/2517256060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 msgr2=0x7faa40198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 0x7faa40198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 -- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 msgr2=0x7faa4019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 0x7faa4019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa44980640 1 -- 172.21.15.38:0/2517256060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4019d070 con 0x7faa40100fe0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.429+0000 7faa45982640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 0x7faa4019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa44980640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d950 tx=0x7faa3000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa367fc640 1 -- 172.21.15.38:0/2517256060 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30017030 con 0x7faa40100fe0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa367fc640 1 -- 172.21.15.38:0/2517256060 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa30004020 con 0x7faa40100fe0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4019d360 con 0x7faa40100fe0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa367fc640 1 -- 172.21.15.38:0/2517256060 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300047a0 con 0x7faa40100fe0 2024-04-22T00:08:06.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.430+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa401a4bf0 con 0x7faa40100fe0 2024-04-22T00:08:06.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.432+0000 7faa367fc640 1 -- 172.21.15.38:0/2517256060 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7faa300041d0 con 0x7faa40100fe0 2024-04-22T00:08:06.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 Processor -- start 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 -- start start 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 0x55b3479e23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479e2930 0x55b3479e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b347939530 con 0x55b3479d4cd0 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b3478e1830 con 0x55b3479e2930 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa49fcf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3479354f0 con 0x55b3479d55f0 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa367fc640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 0x7faa2c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45982640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45982640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32816/0 (socket says 172.21.15.38:32816) 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45982640 1 -- 172.21.15.38:0/3161088700 learned_addr learned my addr 172.21.15.38:0/3161088700 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 0x55b3479e23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45982640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 0x7faa2c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa367fc640 1 -- 172.21.15.38:0/2517256060 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7faa300cf080 con 0x7faa40100fe0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa44980640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479e2930 0x55b3479e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 -- 172.21.15.38:0/3161088700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479e2930 msgr2=0x55b3479e4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479e2930 0x55b3479e4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 -- 172.21.15.38:0/3161088700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 msgr2=0x55b3479d50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45181640 1 -- 172.21.15.38:0/3161088700 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3479052c0 con 0x55b3479d55f0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.433+0000 7faa45982640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa45181640 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 0x55b3479e23f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e581d970c9d12992 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa45982640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 0x7faa2c0a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa38004af0 tx=0x7faa38005c50 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa2b7fe640 1 -- 172.21.15.38:0/3161088700 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c004050 con 0x55b3479d55f0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa2b7fe640 1 -- 172.21.15.38:0/3161088700 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c005500 con 0x55b3479d55f0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa2b7fe640 1 -- 172.21.15.38:0/3161088700 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c004db0 con 0x55b3479d55f0 2024-04-22T00:08:06.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa49fcf740 1 -- 172.21.15.38:0/3161088700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 msgr2=0x55b3479e23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.434+0000 7faa49fcf740 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 0x55b3479e23f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c011810 comp rx=0 tx=0).stop 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 -- 172.21.15.38:0/3161088700 shutdown_connections 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479e2930 0x55b3479e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479d55f0 0x55b3479e23f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 --2- 172.21.15.38:0/3161088700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479d50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 -- 172.21.15.38:0/3161088700 >> 172.21.15.38:0/3161088700 conn(0x55b3479cf750 msgr2=0x55b3479cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 -- 172.21.15.38:0/3161088700 shutdown_connections 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 -- 172.21.15.38:0/3161088700 wait complete. 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 -- start start 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.435+0000 7faa49fcf740 1 Processor -- start 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 0x55b3479e2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b3479e5770 con 0x55b3479d55f0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b3478f4f10 con 0x55b3479d4cd0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3479e58f0 con 0x55b3479e2930 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa49fcf740 10 client.0 _ll_register_callbacks cb 0x55b34764cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32824/0 (socket says 172.21.15.38:32824) 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 -- 172.21.15.38:0/2827159858 learned_addr learned my addr 172.21.15.38:0/2827159858 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa45982640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa45181640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 0x55b3479e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 -- 172.21.15.38:0/2827159858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 msgr2=0x55b3479dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 -- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 msgr2=0x55b3479e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 0x55b3479e2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa44980640 1 -- 172.21.15.38:0/2827159858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3478c6c80 con 0x55b3479d55f0 2024-04-22T00:08:06.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.436+0000 7faa45181640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 0x55b3479e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa44980640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7faa30009610 tx=0x7faa30097620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e1f406e1322d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3009b260 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7faa300a3a40 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa3c005500 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faa3009f610 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 10 client.17604 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b34764d3f0 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c005500 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faa300bb0b0 con 0x55b3479d55f0 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 10 client.17604 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 10 client.17604 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 20 client.17604 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b3479f2940 0x55b3479f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 4 client.17604 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.437+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b347946a70 con 0x55b3479f2940 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.438+0000 7faa45181640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b3479f2940 0x55b3479f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.438+0000 7faa28ff9640 10 client.17604 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.438+0000 7faa45181640 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b3479f2940 0x55b3479f4d50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29f155af1e9bffe8 server_cookie=b500920b9ce9226 in_seq=0 out_seq=0 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa28ff9640 1 -- 172.21.15.38:0/2827159858 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b347946a70 con 0x55b3479f2940 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa28ff9640 10 client.17604 handle_command_reply: tid=0 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 client.17604 shutdown 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 msgr2=0x55b3479df600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7faa30009610 tx=0x7faa30097620 comp rx=0 tx=0).stop 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 shutdown_connections 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b3479f2940 0x55b3479f4d50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d55f0 0x55b3479df600 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa28ff9640 0 client.17604 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 --2- 172.21.15.38:0/2827159858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d4cd0 0x55b3479e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 >> 172.21.15.38:0/2827159858 conn(0x55b3479cf750 msgr2=0x55b3479d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 shutdown_connections 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 1 -- 172.21.15.38:0/2827159858 wait complete. 2024-04-22T00:08:06.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.439+0000 7faa49fcf740 20 client.17604 trim_cache size 0 max 16384 2024-04-22T00:08:06.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 Processor -- start 2024-04-22T00:08:06.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 -- start start 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479e1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa401ac840 con 0x55b3479d4cd0 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa401ae380 con 0x55b3479d55f0 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa49fcf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa401acf10 con 0x55b3479e2930 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32840/0 (socket says 172.21.15.38:32840) 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 -- 172.21.15.38:0/1176308283 learned_addr learned my addr 172.21.15.38:0/1176308283 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa44980640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45982640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479e1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 -- 172.21.15.38:0/1176308283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 msgr2=0x55b3479e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479e1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 -- 172.21.15.38:0/1176308283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 msgr2=0x55b3479daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479daaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 -- 172.21.15.38:0/1176308283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3477177d0 con 0x55b3479d4cd0 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa44980640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.453+0000 7faa45181640 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7faa3c014040 tx=0x7faa3c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d1db3c53dab08a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.457+0000 7faa2affd640 1 -- 172.21.15.38:0/1176308283 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c02bb50 con 0x55b3479d4cd0 2024-04-22T00:08:06.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.457+0000 7faa2affd640 1 -- 172.21.15.38:0/1176308283 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c030070 con 0x55b3479d4cd0 2024-04-22T00:08:06.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.457+0000 7faa2affd640 1 -- 172.21.15.38:0/1176308283 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c019690 con 0x55b3479d4cd0 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 -- 172.21.15.38:0/1176308283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 msgr2=0x55b3479ee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7faa3c014040 tx=0x7faa3c011810 comp rx=0 tx=0).stop 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 -- 172.21.15.38:0/1176308283 shutdown_connections 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479e1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 --2- 172.21.15.38:0/1176308283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479ee670 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 -- 172.21.15.38:0/1176308283 >> 172.21.15.38:0/1176308283 conn(0x55b3479cf750 msgr2=0x55b3479da390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 -- 172.21.15.38:0/1176308283 shutdown_connections 2024-04-22T00:08:06.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.458+0000 7faa49fcf740 1 -- 172.21.15.38:0/1176308283 wait complete. 2024-04-22T00:08:06.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 -- start start 2024-04-22T00:08:06.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 Processor -- start 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479dbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa401adcb0 con 0x55b3479d4cd0 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa401ad5e0 con 0x55b3479d55f0 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa401aa820 con 0x55b3479e2930 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa49fcf740 10 client.0 _ll_register_callbacks cb 0x7faa3009b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa45181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa45181640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32846/0 (socket says 172.21.15.38:32846) 2024-04-22T00:08:06.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa45181640 1 -- 172.21.15.38:0/4190362865 learned_addr learned my addr 172.21.15.38:0/4190362865 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa44980640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479dbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.459+0000 7faa45982640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 -- 172.21.15.38:0/4190362865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 msgr2=0x55b3479dc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 -- 172.21.15.38:0/4190362865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 msgr2=0x55b3479eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 -- 172.21.15.38:0/4190362865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3479dcd20 con 0x55b3479d55f0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa45181640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa44980640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479dbfb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faa3009b1f0 tx=0x7faa30004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35815d70aa39a3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3009b260 con 0x55b3479d55f0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7faa300bb0b0 con 0x55b3479d55f0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa3c030070 con 0x55b3479d55f0 2024-04-22T00:08:06.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faa3009abc0 con 0x55b3479d55f0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa49fcf740 10 client.26972 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.460+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b347a58ec0 con 0x55b3479d55f0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c030070 con 0x55b3479d55f0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faa3009a120 con 0x55b3479d55f0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 10 client.26972 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 10 client.26972 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 20 client.26972 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b347a59450 0x55b347a5b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 4 client.26972 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3479dfed0 con 0x55b347a59450 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa45181640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b347a59450 0x55b347a5b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa45181640 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b347a59450 0x55b347a5b840 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f0d02da223a0ac2 server_cookie=7e8323732d0e23e7 in_seq=0 out_seq=0 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa28ff9640 10 client.26972 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:06.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa28ff9640 1 -- 172.21.15.38:0/4190362865 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3479dfed0 con 0x55b347a59450 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa28ff9640 10 client.26972 handle_command_reply: tid=0 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.461+0000 7faa49fcf740 1 client.26972 shutdown 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 msgr2=0x55b3479dbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479dbfb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faa3009b1f0 tx=0x7faa30004730 comp rx=0 tx=0).stop 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 shutdown_connections 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3479e2930 0x55b3479dc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3479d55f0 0x55b3479dbfb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b347a59450 0x55b347a5b840 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 --2- 172.21.15.38:0/4190362865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b3479d4cd0 0x55b3479eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa28ff9640 0 client.26972 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 >> 172.21.15.38:0/4190362865 conn(0x55b3479cf750 msgr2=0x55b3479ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 shutdown_connections 2024-04-22T00:08:06.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 1 -- 172.21.15.38:0/4190362865 wait complete. 2024-04-22T00:08:06.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa49fcf740 20 client.26972 trim_cache size 0 max 16384 2024-04-22T00:08:06.463 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 msgr2=0x7faa2c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 0x7faa2c0a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa38004af0 tx=0x7faa38005c50 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 msgr2=0x7faa401985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d950 tx=0x7faa3000de10 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 shutdown_connections 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa2c0a2d80 0x7faa2c0a5240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa40106630 0x7faa4019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa40100fe0 0x7faa401985c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 --2- 172.21.15.38:0/2517256060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa40100620 0x7faa40198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.462+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 >> 172.21.15.38:0/2517256060 conn(0x7faa400776a0 msgr2=0x7faa40103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.463+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 shutdown_connections 2024-04-22T00:08:06.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.463+0000 7faa473e5640 1 -- 172.21.15.38:0/2517256060 wait complete. 2024-04-22T00:08:06.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 -- 172.21.15.38:0/3951975791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c1077a0 msgr2=0x7f835c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 --2- 172.21.15.38:0/3951975791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c1077a0 0x7f835c107b80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f834c009930 tx=0x7f834c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 -- 172.21.15.38:0/3951975791 shutdown_connections 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 --2- 172.21.15.38:0/3951975791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c102150 0x7f835c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 --2- 172.21.15.38:0/3951975791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 --2- 172.21.15.38:0/3951975791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c1077a0 0x7f835c107b80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 -- 172.21.15.38:0/3951975791 >> 172.21.15.38:0/3951975791 conn(0x7f835c0fd2a0 msgr2=0x7f835c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 -- 172.21.15.38:0/3951975791 shutdown_connections 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.606+0000 7f83614e2640 1 -- 172.21.15.38:0/3951975791 wait complete. 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 Processor -- start 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 -- start start 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c102150 0x7f835c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f835c111b70 con 0x7f835c1077a0 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1119f0 con 0x7f835c101790 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f83614e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f835c111cf0 con 0x7f835c102150 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f835b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f835b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32866/0 (socket says 172.21.15.38:32866) 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.607+0000 7f835b7fe640 1 -- 172.21.15.38:0/1285720644 learned_addr learned my addr 172.21.15.38:0/1285720644 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c102150 0x7f835c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835affd640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 -- 172.21.15.38:0/1285720644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c102150 msgr2=0x7f835c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c102150 0x7f835c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 -- 172.21.15.38:0/1285720644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 msgr2=0x7f835c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 -- 172.21.15.38:0/1285720644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835c19d0f0 con 0x7f835c1077a0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835affd640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f835b7fe640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f835400efd0 tx=0x7f835400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f8343fff640 1 -- 172.21.15.38:0/1285720644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8354003b70 con 0x7f835c1077a0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f8343fff640 1 -- 172.21.15.38:0/1285720644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8354003d10 con 0x7f835c1077a0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835c19d3e0 con 0x7f835c1077a0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f8343fff640 1 -- 172.21.15.38:0/1285720644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83540109f0 con 0x7f835c1077a0 2024-04-22T00:08:06.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.608+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835c1a4c70 con 0x7f835c1077a0 2024-04-22T00:08:06.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f8343fff640 1 -- 172.21.15.38:0/1285720644 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8354004020 con 0x7f835c1077a0 2024-04-22T00:08:06.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 Processor -- start 2024-04-22T00:08:06.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 -- start start 2024-04-22T00:08:06.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839da00 0x5600e839dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839e3b0 0x5600e83ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600e8302530 con 0x5600e83ab930 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600e82aa830 con 0x5600e839da00 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f83641aa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600e82fe4f0 con 0x5600e839e3b0 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32868/0 (socket says 172.21.15.38:32868) 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 -- 172.21.15.38:0/3825911270 learned_addr learned my addr 172.21.15.38:0/3825911270 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839da00 0x5600e839dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835affd640 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839e3b0 0x5600e83ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f8343fff640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 0x7f83440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 -- 172.21.15.38:0/3825911270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839e3b0 msgr2=0x5600e83ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839e3b0 0x5600e83ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 -- 172.21.15.38:0/3825911270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839da00 msgr2=0x5600e839dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839da00 0x5600e839dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 -- 172.21.15.38:0/3825911270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600e82ce2c0 con 0x5600e83ab930 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835b7fe640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 0x7f83440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.611+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f83500093e0 tx=0x7f8350009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9660c3edfc3dd297 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f8343fff640 1 -- 172.21.15.38:0/1285720644 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f83540cfc00 con 0x7f835c1077a0 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83417fa640 1 -- 172.21.15.38:0/3825911270 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835000f7c0 con 0x5600e83ab930 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83417fa640 1 -- 172.21.15.38:0/3825911270 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835000fe40 con 0x5600e83ab930 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83417fa640 1 -- 172.21.15.38:0/3825911270 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835001fdb0 con 0x5600e83ab930 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 -- 172.21.15.38:0/3825911270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 msgr2=0x5600e83add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f835b7fe640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 0x7f83440a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83540cd0c0 tx=0x7f835406dda0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f83500093e0 tx=0x7f8350009370 comp rx=0 tx=0).stop 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 -- 172.21.15.38:0/3825911270 shutdown_connections 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e83ab930 0x5600e83add20 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839e3b0 0x5600e83ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 --2- 172.21.15.38:0/3825911270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839da00 0x5600e839dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.612+0000 7f83641aa740 1 -- 172.21.15.38:0/3825911270 >> 172.21.15.38:0/3825911270 conn(0x5600e8398750 msgr2=0x5600e8398b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- 172.21.15.38:0/3825911270 shutdown_connections 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- 172.21.15.38:0/3825911270 wait complete. 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- start start 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 Processor -- start 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 0x5600e83a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e83ab930 0x5600e83a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5600e83ae6f0 con 0x5600e839e3b0 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5600e82bdf10 con 0x5600e83ab930 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5600e83ae870 con 0x5600e839da00 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f83641aa740 10 client.0 _ll_register_callbacks cb 0x5600e8015be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f835affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 0x5600e83a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34070/0 (socket says 172.21.15.38:34070) 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f835a7fc640 1 -- 172.21.15.38:0/1794303063 learned_addr learned my addr 172.21.15.38:0/1794303063 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.613+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e83ab930 0x5600e83a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 -- 172.21.15.38:0/1794303063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e83ab930 msgr2=0x5600e83a8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e83ab930 0x5600e83a8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 -- 172.21.15.38:0/1794303063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 msgr2=0x5600e83a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 0x5600e83a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 -- 172.21.15.38:0/1794303063 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600e828fc80 con 0x5600e839da00 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835affd640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 0x5600e83a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8350009fd0 tx=0x7f835001f8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75631d6c26b685df server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83500040f0 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f835000fe40 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8350019070 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8350036050 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f83641aa740 10 client.26830 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5600e80163f0 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835000fe40 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.614+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f835001d020 con 0x5600e839da00 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 10 client.26830 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 10 client.26830 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 20 client.26830 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e83bb900 0x5600e83bdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 4 client.26830 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:06.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600e830fa70 con 0x5600e83bb900 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f835affd640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e83bb900 0x5600e83bdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f835affd640 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e83bb900 0x5600e83bdcf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3138f13bdde82522 server_cookie=6b6f2f36734e3fb3 in_seq=0 out_seq=0 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.615+0000 7f833eff5640 10 client.26830 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f833eff5640 1 -- 172.21.15.38:0/1794303063 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600e830fa70 con 0x5600e83bb900 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f833eff5640 10 client.26830 handle_command_reply: tid=0 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 client.26830 shutdown 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 msgr2=0x5600e83ab310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8350009fd0 tx=0x7f835001f8f0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 shutdown_connections 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e83bb900 0x5600e83bdcf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e83ab930 0x5600e83a8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f833eff5640 0 client.26830 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839e3b0 0x5600e83a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 --2- 172.21.15.38:0/1794303063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e839da00 0x5600e83ab310 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 >> 172.21.15.38:0/1794303063 conn(0x5600e8398750 msgr2=0x5600e839a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 shutdown_connections 2024-04-22T00:08:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 1 -- 172.21.15.38:0/1794303063 wait complete. 2024-04-22T00:08:06.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.616+0000 7f83641aa740 20 client.26830 trim_cache size 0 max 16384 2024-04-22T00:08:06.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 Processor -- start 2024-04-22T00:08:06.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 -- start start 2024-04-22T00:08:06.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1ac950 con 0x5600e839da00 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1ae490 con 0x5600e839e3b0 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f83641aa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1ad020 con 0x5600e83ab930 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32894/0 (socket says 172.21.15.38:32894) 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 -- 172.21.15.38:0/1646763556 learned_addr learned my addr 172.21.15.38:0/1646763556 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835affd640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 -- 172.21.15.38:0/1646763556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 msgr2=0x5600e83aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 -- 172.21.15.38:0/1646763556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 msgr2=0x5600e83a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83a76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835a7fc640 1 -- 172.21.15.38:0/1646763556 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600e839e150 con 0x5600e839da00 2024-04-22T00:08:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.631+0000 7f835affd640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83a76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:06.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f835a7fc640 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8350011520 tx=0x7f8350009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcca0f06781c80d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f8340ff9640 1 -- 172.21.15.38:0/1646763556 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350003d80 con 0x5600e839da00 2024-04-22T00:08:06.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f8340ff9640 1 -- 172.21.15.38:0/1646763556 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835001fa50 con 0x5600e839da00 2024-04-22T00:08:06.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f8340ff9640 1 -- 172.21.15.38:0/1646763556 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83500052a0 con 0x5600e839da00 2024-04-22T00:08:06.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f83641aa740 1 -- 172.21.15.38:0/1646763556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 msgr2=0x5600e83b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.632+0000 7f83641aa740 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f8350011520 tx=0x7f8350009370 comp rx=0 tx=0).stop 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- 172.21.15.38:0/1646763556 shutdown_connections 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83a76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- 172.21.15.38:0/1646763556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b75f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- 172.21.15.38:0/1646763556 >> 172.21.15.38:0/1646763556 conn(0x5600e8398750 msgr2=0x5600e8397860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- 172.21.15.38:0/1646763556 shutdown_connections 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- 172.21.15.38:0/1646763556 wait complete. 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- start start 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 Processor -- start 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83b4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1addc0 con 0x5600e839da00 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1ad6f0 con 0x5600e839e3b0 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f835c1aa930 con 0x5600e83ab930 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f83641aa740 10 client.0 _ll_register_callbacks cb 0x7f8350009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f835a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32902/0 (socket says 172.21.15.38:32902) 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f835a7fc640 1 -- 172.21.15.38:0/3871608744 learned_addr learned my addr 172.21.15.38:0/3871608744 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f835affd640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.633+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 -- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 msgr2=0x5600e83a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 -- 172.21.15.38:0/3871608744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 msgr2=0x5600e83b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 -- 172.21.15.38:0/3871608744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600e80e07d0 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f835a7fc640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8350005fd0 tx=0x7f8350005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e562d8bbe38e3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f833eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350009a90 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f8350009c30 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f835001fa50 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83500040f0 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f83641aa740 10 client.17628 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5600e8421380 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.634+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f835001fa50 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f835001d030 con 0x5600e839da00 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 10 client.17628 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 10 client.17628 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:08:06.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 20 client.17628 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e8421990 0x5600e8423db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 4 client.17628 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5600e8309dc0 con 0x5600e8421990 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f835affd640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e8421990 0x5600e8423db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f833eff5640 10 client.17628 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f835affd640 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e8421990 0x5600e8423db0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c5fb8635f065c92e server_cookie=f7e5e0bdca780157 in_seq=0 out_seq=0 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f833eff5640 1 -- 172.21.15.38:0/3871608744 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5600e8309dc0 con 0x5600e8421990 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f833eff5640 10 client.17628 handle_command_reply: tid=0 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.635+0000 7f83641aa740 1 client.17628 shutdown 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 msgr2=0x5600e83b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8350005fd0 tx=0x7f8350005810 comp rx=0 tx=0).stop 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 shutdown_connections 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x5600e8421990 0x5600e8423db0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5600e83ab930 0x5600e83a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f833eff5640 0 client.17628 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5600e839e3b0 0x5600e83b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 --2- 172.21.15.38:0/3871608744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5600e839da00 0x5600e83b3b00 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 >> 172.21.15.38:0/3871608744 conn(0x5600e8398750 msgr2=0x5600e83a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 shutdown_connections 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 1 -- 172.21.15.38:0/3871608744 wait complete. 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83641aa740 20 client.17628 trim_cache size 0 max 16384 2024-04-22T00:08:06.638 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 msgr2=0x7f83440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 0x7f83440a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83540cd0c0 tx=0x7f835406dda0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 msgr2=0x7f835c19c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f835400efd0 tx=0x7f835400c6a0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 shutdown_connections 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83440a2d00 0x7f83440a51c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f835c1077a0 0x7f835c19c9c0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f835c102150 0x7f835c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 --2- 172.21.15.38:0/1285720644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f835c101790 0x7f835c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 >> 172.21.15.38:0/1285720644 conn(0x7f835c0fd2a0 msgr2=0x7f835c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.636+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 shutdown_connections 2024-04-22T00:08:06.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.637+0000 7f83614e2640 1 -- 172.21.15.38:0/1285720644 wait complete. 2024-04-22T00:08:06.642 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:06 smithi038 ceph-mon[27972]: pgmap v183: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:08:06.642 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:06 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2929574937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:06.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3067256132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100610 msgr2=0x7fc4b0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3067256132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100610 0x7fc4b0100a90 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc498009930 tx=0x7fc49802bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3067256132 shutdown_connections 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3067256132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100fd0 0x7fc4b010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3067256132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100610 0x7fc4b0100a90 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3067256132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 0x7fc4b0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3067256132 >> 172.21.15.38:0/3067256132 conn(0x7fc4b0077740 msgr2=0x7fc4b0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3067256132 shutdown_connections 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.764+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3067256132 wait complete. 2024-04-22T00:08:06.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 Processor -- start 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 -- start start 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100610 0x7fc4b01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 0x7fc4b019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0110a50 con 0x7fc4b0100fd0 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01108d0 con 0x7fc4b0100610 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.765+0000 7fc4b4f66640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0110bd0 con 0x7fc4b01066b0 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32924/0 (socket says 172.21.15.38:32924) 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 -- 172.21.15.38:0/3977419057 learned_addr learned my addr 172.21.15.38:0/3977419057 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100610 0x7fc4b01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 0x7fc4b019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 -- 172.21.15.38:0/3977419057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 msgr2=0x7fc4b019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 0x7fc4b019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 -- 172.21.15.38:0/3977419057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100610 msgr2=0x7fc4b01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100610 0x7fc4b01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 -- 172.21.15.38:0/3977419057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b019d090 con 0x7fc4b0100fd0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc498009230 tx=0x7fc49802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4a77fe640 1 -- 172.21.15.38:0/3977419057 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498041070 con 0x7fc4b0100fd0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b019d320 con 0x7fc4b0100fd0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4a77fe640 1 -- 172.21.15.38:0/3977419057 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc498004450 con 0x7fc4b0100fd0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4a77fe640 1 -- 172.21.15.38:0/3977419057 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498034440 con 0x7fc4b0100fd0 2024-04-22T00:08:06.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.766+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b01a4c10 con 0x7fc4b0100fd0 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4a77fe640 1 -- 172.21.15.38:0/3977419057 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc498044070 con 0x7fc4b0100fd0 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 Processor -- start 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 -- start start 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 0x555c0ac48400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac50fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c0abac530 con 0x555c0ac51520 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c0ab54830 con 0x555c0ac48020 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c0aba84f0 con 0x555c0ac48940 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4a77fe640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 0x7fc49c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4aed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 0x555c0ac48400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.769+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32936/0 (socket says 172.21.15.38:32936) 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 -- 172.21.15.38:0/1118383025 learned_addr learned my addr 172.21.15.38:0/1118383025 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac50fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 0x7fc49c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4a77fe640 1 -- 172.21.15.38:0/3977419057 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc4980f0fd0 con 0x7fc4b0100fd0 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 -- 172.21.15.38:0/1118383025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 msgr2=0x555c0ac50fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac50fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 -- 172.21.15.38:0/1118383025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 msgr2=0x555c0ac48400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 0x555c0ac48400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 -- 172.21.15.38:0/1118383025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c0ab782c0 con 0x555c0ac51520 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 0x555c0ac48400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc4980c5a00 tx=0x7fc498003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c660a5a6ccb7719a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4a4ff9640 1 -- 172.21.15.38:0/1118383025 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49804b070 con 0x555c0ac51520 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 0x7fc49c0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009fd0 tx=0x7fc4a0004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4a4ff9640 1 -- 172.21.15.38:0/1118383025 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc498004c80 con 0x555c0ac51520 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.770+0000 7fc4a4ff9640 1 -- 172.21.15.38:0/1118383025 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980b99e0 con 0x555c0ac51520 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1118383025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 msgr2=0x555c0ac53910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc4980c5a00 tx=0x7fc498003990 comp rx=0 tx=0).stop 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1118383025 shutdown_connections 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac51520 0x555c0ac53910 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac50fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1118383025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac48020 0x555c0ac48400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1118383025 >> 172.21.15.38:0/1118383025 conn(0x555c0ac42750 msgr2=0x555c0ac42b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1118383025 shutdown_connections 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1118383025 wait complete. 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 -- start start 2024-04-22T00:08:06.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.771+0000 7fc4b7c20740 1 Processor -- start 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac4e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac4e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555c0abac530 con 0x555c0ac48020 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555c0ab67f10 con 0x555c0ac51520 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555c0ac540d0 con 0x555c0ac48940 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4b7c20740 10 client.0 _ll_register_callbacks cb 0x555c0a8bfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32940/0 (socket says 172.21.15.38:32940) 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 -- 172.21.15.38:0/3999921788 learned_addr learned my addr 172.21.15.38:0/3999921788 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac4e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac4e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 -- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 msgr2=0x555c0ac4e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac4e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 -- 172.21.15.38:0/3999921788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 msgr2=0x555c0ac4e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac4e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 -- 172.21.15.38:0/3999921788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c0ab39c80 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac4e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80126e0 tx=0x7fc4a8012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae70c6d22292198 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.772+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004050 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc4a80041f0 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc498004c80 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4a80043f0 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 10 client.17640 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555c0a8c03f0 con 0x555c0ac48020 2024-04-22T00:08:06.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc498004c80 con 0x555c0ac48020 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc4a8003ca0 con 0x555c0ac48020 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 10 client.17640 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 10 client.17640 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 20 client.17640 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0ac612a0 0x555c0ac63720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 4 client.17640 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c0abb9a70 con 0x555c0ac612a0 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.773+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0ac612a0 0x555c0ac63720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.774+0000 7fc4add9b640 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0ac612a0 0x555c0ac63720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcccf5d5f511b63 server_cookie=1571589bb31bc8be in_seq=0 out_seq=0 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.774+0000 7fc495ffb640 10 client.17640 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.774+0000 7fc495ffb640 1 -- 172.21.15.38:0/3999921788 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c0abb9a70 con 0x555c0ac612a0 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.774+0000 7fc495ffb640 10 client.17640 handle_command_reply: tid=0 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.774+0000 7fc4b7c20740 1 client.17640 shutdown 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 msgr2=0x555c0ac50ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80126e0 tx=0x7fc4a8012ba0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 shutdown_connections 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0ac612a0 0x555c0ac63720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac4e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48940 0x555c0ac4e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc495ffb640 0 client.17640 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/3999921788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48020 0x555c0ac50ef0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 >> 172.21.15.38:0/3999921788 conn(0x555c0ac42750 msgr2=0x555c0ac444a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 shutdown_connections 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 1 -- 172.21.15.38:0/3999921788 wait complete. 2024-04-22T00:08:06.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.775+0000 7fc4b7c20740 20 client.17640 trim_cache size 0 max 16384 2024-04-22T00:08:06.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:06 smithi203 ceph-mon[31431]: pgmap v183: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:08:06.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:06 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2929574937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 Processor -- start 2024-04-22T00:08:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 -- start start 2024-04-22T00:08:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac5ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac50ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01aead0 con 0x555c0ac48940 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01b0610 con 0x555c0ac51520 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01af1a0 con 0x555c0ac48020 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32944/0 (socket says 172.21.15.38:32944) 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4add9b640 1 -- 172.21.15.38:0/1858567106 learned_addr learned my addr 172.21.15.38:0/1858567106 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.788+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac5ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 -- 172.21.15.38:0/1858567106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 msgr2=0x555c0ac5ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac5ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 -- 172.21.15.38:0/1858567106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 msgr2=0x555c0ac50ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac50ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac50ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 -- 172.21.15.38:0/1858567106 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c0ac48770 con 0x555c0ac48940 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac50ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc49802bc70 tx=0x7fc498000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a841b362c1cbfbc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc497fff640 1 -- 172.21.15.38:0/1858567106 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49804b040 con 0x555c0ac48940 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc497fff640 1 -- 172.21.15.38:0/1858567106 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc498004c80 con 0x555c0ac48940 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc497fff640 1 -- 172.21.15.38:0/1858567106 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980b99e0 con 0x555c0ac48940 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1858567106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 msgr2=0x555c0ac509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc49802bc70 tx=0x7fc498000bc0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1858567106 shutdown_connections 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac50ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac509b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1858567106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac5ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1858567106 >> 172.21.15.38:0/1858567106 conn(0x555c0ac42750 msgr2=0x555c0ac41850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.789+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1858567106 shutdown_connections 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1858567106 wait complete. 2024-04-22T00:08:06.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 -- start start 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 Processor -- start 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac597d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac55130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01aff40 con 0x555c0ac48940 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01af870 con 0x555c0ac51520 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01acab0 con 0x555c0ac48020 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4b7c20740 10 client.0 _ll_register_callbacks cb 0x7fc4a8009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32956/0 (socket says 172.21.15.38:32956) 2024-04-22T00:08:06.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 -- 172.21.15.38:0/1371301305 learned_addr learned my addr 172.21.15.38:0/1371301305 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4aed9d640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac55130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 -- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 msgr2=0x555c0ac597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 -- 172.21.15.38:0/1371301305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 msgr2=0x555c0ac55130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac55130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.790+0000 7fc4add9b640 1 -- 172.21.15.38:0/1371301305 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c0a98a7d0 con 0x555c0ac48940 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4add9b640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc49802c820 tx=0x7fc4980bd370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c68516717d0e3ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49804b040 con 0x555c0ac48940 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fc4980035f0 con 0x555c0ac48940 2024-04-22T00:08:06.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc498004c80 con 0x555c0ac48940 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4980bccd0 con 0x555c0ac48940 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 10 client.17646 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555c0accbad0 con 0x555c0ac48940 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc498004c80 con 0x555c0ac48940 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc4980dd0d0 con 0x555c0ac48940 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 10 client.17646 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 10 client.17646 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 20 client.17646 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0accbe70 0x555c0acce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 4 client.17646 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:06.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c0abb3dc0 con 0x555c0accbe70 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.791+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0accbe70 0x555c0acce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc495ffb640 10 client.17646 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4ae59c640 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0accbe70 0x555c0acce260 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f004a7c5c350801 server_cookie=ea0d51107858a27c in_seq=0 out_seq=0 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc495ffb640 1 -- 172.21.15.38:0/1371301305 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c0abb3dc0 con 0x555c0accbe70 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc495ffb640 10 client.17646 handle_command_reply: tid=0 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 client.17646 shutdown 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 msgr2=0x555c0ac54b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc49802c820 tx=0x7fc4980bd370 comp rx=0 tx=0).stop 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 shutdown_connections 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555c0accbe70 0x555c0acce260 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555c0ac51520 0x555c0ac55130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc495ffb640 0 client.17646 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555c0ac48940 0x555c0ac54b90 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 --2- 172.21.15.38:0/1371301305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555c0ac48020 0x555c0ac597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 >> 172.21.15.38:0/1371301305 conn(0x555c0ac42750 msgr2=0x555c0ac48e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 shutdown_connections 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 1 -- 172.21.15.38:0/1371301305 wait complete. 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.792+0000 7fc4b7c20740 20 client.17646 trim_cache size 0 max 16384 2024-04-22T00:08:06.799 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 msgr2=0x7fc49c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 0x7fc49c0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009fd0 tx=0x7fc4a0004a40 comp rx=0 tx=0).stop 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 msgr2=0x7fc4b01985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc498009230 tx=0x7fc49802cba0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 shutdown_connections 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc49c0a2d00 0x7fc49c0a51c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4b01066b0 0x7fc4b019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4b0100fd0 0x7fc4b01985e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 --2- 172.21.15.38:0/3977419057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0100610 0x7fc4b01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 >> 172.21.15.38:0/3977419057 conn(0x7fc4b0077740 msgr2=0x7fc4b00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 shutdown_connections 2024-04-22T00:08:06.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.793+0000 7fc4b4f66640 1 -- 172.21.15.38:0/3977419057 wait complete. 2024-04-22T00:08:06.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:08:06.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:06 smithi111 ceph-mon[31543]: pgmap v183: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.60k op/s 2024-04-22T00:08:06.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:06 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2929574937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- 172.21.15.38:0/3855033507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc81066b0 msgr2=0x7f4cc8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- 172.21.15.38:0/3855033507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc81066b0 0x7f4cc8106a90 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009930 tx=0x7f4cb802bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:06.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- 172.21.15.38:0/3855033507 shutdown_connections 2024-04-22T00:08:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- 172.21.15.38:0/3855033507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8100fd0 0x7f4cc810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- 172.21.15.38:0/3855033507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- 172.21.15.38:0/3855033507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc81066b0 0x7f4cc8106a90 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- 172.21.15.38:0/3855033507 >> 172.21.15.38:0/3855033507 conn(0x7f4cc8077740 msgr2=0x7f4cc8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- 172.21.15.38:0/3855033507 shutdown_connections 2024-04-22T00:08:06.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- 172.21.15.38:0/3855033507 wait complete. 2024-04-22T00:08:06.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 Processor -- start 2024-04-22T00:08:06.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- start start 2024-04-22T00:08:06.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 0x7f4cc819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc8111b30 con 0x7f4cc8100fd0 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81119b0 con 0x7f4cc8100610 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.926+0000 7f4ccd415640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc8111cb0 con 0x7f4cc81066b0 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cbffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58740/0 (socket says 172.21.15.38:58740) 2024-04-22T00:08:06.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cbffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:32978/0 (socket says 172.21.15.38:32978) 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/1756051793 learned_addr learned my addr 172.21.15.38:0/1756051793 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 0x7f4cc819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/1756051793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 msgr2=0x7f4cc819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 0x7f4cc819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 msgr2=0x7f4cc81984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/1756051793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc819cf60 con 0x7f4cc8100610 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cbffff640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 0x7f4cc819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009230 tx=0x7f4cb802cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc4ff9640 1 -- 172.21.15.38:0/1756051793 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8041070 con 0x7f4cc8100610 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc819d1f0 con 0x7f4cc8100610 2024-04-22T00:08:06.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc4ff9640 1 -- 172.21.15.38:0/1756051793 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb8004450 con 0x7f4cc8100610 2024-04-22T00:08:06.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4cc4ff9640 1 -- 172.21.15.38:0/1756051793 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003d90 con 0x7f4cc8100610 2024-04-22T00:08:06.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.927+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc81a4ae0 con 0x7f4cc8100610 2024-04-22T00:08:06.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.929+0000 7f4cc4ff9640 1 -- 172.21.15.38:0/1756051793 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4cb8044070 con 0x7f4cc8100610 2024-04-22T00:08:06.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 Processor -- start 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 -- start start 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05cd50 0x56027f05d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f06cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cc4ff9640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 0x7f4cb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56027efc1530 con 0x56027f06a930 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56027ef69830 con 0x56027f05d670 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cd0002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56027efbd4f0 con 0x56027f05cd50 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.930+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 0x7f4cb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58746/0 (socket says 172.21.15.38:58746) 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/2137829978 learned_addr learned my addr 172.21.15.38:0/2137829978 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05cd50 0x56027f05d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cbffff640 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f06cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc4ff9640 1 -- 172.21.15.38:0/1756051793 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4cb80f11f0 con 0x7f4cc8100610 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/2137829978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05cd50 msgr2=0x56027f05d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05cd50 0x56027f05d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/2137829978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 msgr2=0x56027f06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f06cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/2137829978 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027ef8d2c0 con 0x56027f05d670 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 0x7f4cb40a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009270 tx=0x7f4cc0004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802cfd0 tx=0x7f4cb808f440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafd3ab0023dc375 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cbd7fa640 1 -- 172.21.15.38:0/2137829978 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb804b070 con 0x56027f05d670 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cbd7fa640 1 -- 172.21.15.38:0/2137829978 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb804c260 con 0x56027f05d670 2024-04-22T00:08:06.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.931+0000 7f4cbd7fa640 1 -- 172.21.15.38:0/2137829978 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80c60f0 con 0x56027f05d670 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- 172.21.15.38:0/2137829978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 msgr2=0x56027f06a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4cb802cfd0 tx=0x7f4cb808f440 comp rx=0 tx=0).stop 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- 172.21.15.38:0/2137829978 shutdown_connections 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f06cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05d670 0x56027f06a3f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 --2- 172.21.15.38:0/2137829978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05cd50 0x56027f05d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- 172.21.15.38:0/2137829978 >> 172.21.15.38:0/2137829978 conn(0x56027f057750 msgr2=0x56027f057b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- 172.21.15.38:0/2137829978 shutdown_connections 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- 172.21.15.38:0/2137829978 wait complete. 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 -- start start 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.932+0000 7f4cd0002740 1 Processor -- start 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05d670 0x56027f067670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f06a930 0x56027f067c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56027ef69830 con 0x56027f05d670 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56027ef7cf10 con 0x56027f05cd50 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56027f06d4b0 con 0x56027f06a930 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 10 client.0 _ll_register_callbacks cb 0x56027ecd4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cbffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05d670 0x56027f067670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58752/0 (socket says 172.21.15.38:58752) 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/124714760 learned_addr learned my addr 172.21.15.38:0/124714760 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f06a930 0x56027f067c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/124714760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f06a930 msgr2=0x56027f067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f06a930 0x56027f067c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/124714760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05d670 msgr2=0x56027f067670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05d670 0x56027f067670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/124714760 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027ef4ec80 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80c5bc0 tx=0x7f4cb8004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7bb19785a2904c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80d6060 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4cb80d6200 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.933+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56027efcea70 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4cb804b070 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 10 client.27008 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56027ecd53f0 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56027efcea70 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4cb80c5c60 con 0x56027f05cd50 2024-04-22T00:08:06.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 10 client.27008 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 10 client.27008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 20 client.27008 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f07a4d0 0x56027f07c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 4 client.27008 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56027efc8dc0 con 0x56027f07a4d0 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cbffff640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f07a4d0 0x56027f07c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cbffff640 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f07a4d0 0x56027f07c8c0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92adb6654a8f0bd7 server_cookie=874d9309ceacb393 in_seq=0 out_seq=0 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.934+0000 7f4cae7fc640 10 client.27008 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cae7fc640 1 -- 172.21.15.38:0/124714760 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56027efc8dc0 con 0x56027f07a4d0 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cae7fc640 10 client.27008 handle_command_reply: tid=0 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 client.27008 shutdown 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 msgr2=0x56027f06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80c5bc0 tx=0x7f4cb8004ee0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 shutdown_connections 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f06a930 0x56027f067c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f07a4d0 0x56027f07c8c0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05d670 0x56027f067670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 --2- 172.21.15.38:0/124714760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f06a2e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cae7fc640 0 client.27008 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 >> 172.21.15.38:0/124714760 conn(0x56027f057750 msgr2=0x56027f0594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 shutdown_connections 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 1 -- 172.21.15.38:0/124714760 wait complete. 2024-04-22T00:08:06.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.935+0000 7f4cd0002740 20 client.27008 trim_cache size 0 max 16384 2024-04-22T00:08:06.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 Processor -- start 2024-04-22T00:08:06.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 -- start start 2024-04-22T00:08:06.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f056850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81ac830 con 0x56027f06a930 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81ae370 con 0x56027f05cd50 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cd0002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81acf00 con 0x56027f05d670 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33010/0 (socket says 172.21.15.38:33010) 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 -- 172.21.15.38:0/3938059245 learned_addr learned my addr 172.21.15.38:0/3938059245 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cbffff640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f056850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 -- 172.21.15.38:0/3938059245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 msgr2=0x56027f056850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f056850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 -- 172.21.15.38:0/3938059245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 msgr2=0x56027f076260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f076260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc77fe640 1 -- 172.21.15.38:0/3938059245 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027f05d4a0 con 0x56027f06a930 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.950+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f076260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.951+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009e30 tx=0x7f4cc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c953bd920c3f858 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.951+0000 7f4cbcff9640 1 -- 172.21.15.38:0/3938059245 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0012e60 con 0x56027f06a930 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.951+0000 7f4cbcff9640 1 -- 172.21.15.38:0/3938059245 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cc002b920 con 0x56027f06a930 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.951+0000 7f4cbcff9640 1 -- 172.21.15.38:0/3938059245 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc0010b10 con 0x56027f06a930 2024-04-22T00:08:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.951+0000 7f4cd0002740 1 -- 172.21.15.38:0/3938059245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 msgr2=0x56027f069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009e30 tx=0x7f4cc0017000 comp rx=0 tx=0).stop 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 -- 172.21.15.38:0/3938059245 shutdown_connections 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f06a930 0x56027f069da0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f056850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- 172.21.15.38:0/3938059245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f05cd50 0x56027f076260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 -- 172.21.15.38:0/3938059245 >> 172.21.15.38:0/3938059245 conn(0x56027f057750 msgr2=0x56027f057b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 -- 172.21.15.38:0/3938059245 shutdown_connections 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 -- 172.21.15.38:0/3938059245 wait complete. 2024-04-22T00:08:06.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 -- start start 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 Processor -- start 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f063ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.952+0000 7f4cd0002740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f06a930 0x56027f064420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cd0002740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81adca0 con 0x56027f05cd50 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cd0002740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81ad5d0 con 0x56027f06a930 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cd0002740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc81aa810 con 0x56027f05d670 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cd0002740 10 client.0 _ll_register_callbacks cb 0x7f4cb0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33026/0 (socket says 172.21.15.38:33026) 2024-04-22T00:08:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/659866884 learned_addr learned my addr 172.21.15.38:0/659866884 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc77fe640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f06a930 0x56027f064420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cbffff640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f063ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/659866884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 msgr2=0x56027f063ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f063ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f06a930 msgr2=0x56027f064420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f06a930 0x56027f064420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 -- 172.21.15.38:0/659866884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56027ed9f7d0 con 0x56027f05cd50 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cc6ffd640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80bd410 tx=0x7f4cb802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8ea40c1f2ee498 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80d6040 con 0x56027f05cd50 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4cb80d61e0 con 0x56027f05cd50 2024-04-22T00:08:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.953+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cc002b920 con 0x56027f05cd50 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4cb80eea30 con 0x56027f05cd50 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cd0002740 10 client.17664 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56027ed9f7d0 con 0x56027f05cd50 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cc002b920 con 0x56027f05cd50 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4cb80bc8d0 con 0x56027f05cd50 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cd0002740 10 client.17664 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cd0002740 10 client.17664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.954+0000 7f4cd0002740 20 client.17664 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f0e09c0 0x56027f0e2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 4 client.17664 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:06.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56027ef466e0 con 0x56027f0e09c0 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cbffff640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f0e09c0 0x56027f0e2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cae7fc640 10 client.17664 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cbffff640 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f0e09c0 0x56027f0e2db0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fb1ee39f65855fe server_cookie=3415b7ec2b65a6f1 in_seq=0 out_seq=0 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cae7fc640 1 -- 172.21.15.38:0/659866884 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56027ef466e0 con 0x56027f0e09c0 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cae7fc640 10 client.17664 handle_command_reply: tid=0 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 1 client.17664 shutdown 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 msgr2=0x56027f0729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.955+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80bd410 tx=0x7f4cb802bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 shutdown_connections 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56027f06a930 0x56027f064420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56027f05d670 0x56027f063ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56027f0e09c0 0x56027f0e2db0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 --2- 172.21.15.38:0/659866884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56027f05cd50 0x56027f0729d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 >> 172.21.15.38:0/659866884 conn(0x56027f057750 msgr2=0x56027f056850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cae7fc640 0 client.17664 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:06.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.956+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 shutdown_connections 2024-04-22T00:08:06.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.957+0000 7f4cd0002740 1 -- 172.21.15.38:0/659866884 wait complete. 2024-04-22T00:08:06.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.957+0000 7f4cd0002740 20 client.17664 trim_cache size 0 max 16384 2024-04-22T00:08:06.961 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:06.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.958+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 msgr2=0x7f4cb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.958+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 0x7f4cb40a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009270 tx=0x7f4cc0004ab0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 msgr2=0x7f4cc8197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009230 tx=0x7f4cb802cb20 comp rx=0 tx=0).stop 2024-04-22T00:08:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 shutdown_connections 2024-04-22T00:08:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4cb40a2d00 0x7f4cb40a51c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc81066b0 0x7f4cc819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4cc8100fd0 0x7f4cc81984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 --2- 172.21.15.38:0/1756051793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc8100610 0x7f4cc8197f70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.959+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 >> 172.21.15.38:0/1756051793 conn(0x7f4cc8077740 msgr2=0x7f4cc80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.960+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 shutdown_connections 2024-04-22T00:08:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:06.960+0000 7f4ccd415640 1 -- 172.21.15.38:0/1756051793 wait complete. 2024-04-22T00:08:07.974 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:08:07.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:08:07.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 -- 172.21.15.38:0/3545498794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 msgr2=0x7ff52c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 --2- 172.21.15.38:0/3545498794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c107b00 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009930 tx=0x7ff51c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 -- 172.21.15.38:0/3545498794 shutdown_connections 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 --2- 172.21.15.38:0/3545498794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c1020d0 0x7ff52c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 --2- 172.21.15.38:0/3545498794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c101710 0x7ff52c101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 --2- 172.21.15.38:0/3545498794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c107b00 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.106+0000 7ff530e37640 1 -- 172.21.15.38:0/3545498794 >> 172.21.15.38:0/3545498794 conn(0x7ff52c0fd180 msgr2=0x7ff52c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- 172.21.15.38:0/3545498794 shutdown_connections 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- 172.21.15.38:0/3545498794 wait complete. 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 Processor -- start 2024-04-22T00:08:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- start start 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c101710 0x7ff52c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c1020d0 0x7ff52c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c111b30 con 0x7ff52c107720 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1119b0 con 0x7ff52c101710 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff530e37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c111cb0 con 0x7ff52c1020d0 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.107+0000 7ff52ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33038/0 (socket says 172.21.15.38:33038) 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 -- 172.21.15.38:0/629716261 learned_addr learned my addr 172.21.15.38:0/629716261 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff529d9b640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c1020d0 0x7ff52c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52a59c640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c101710 0x7ff52c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 -- 172.21.15.38:0/629716261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c1020d0 msgr2=0x7ff52c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c1020d0 0x7ff52c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 -- 172.21.15.38:0/629716261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c101710 msgr2=0x7ff52c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c101710 0x7ff52c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 -- 172.21.15.38:0/629716261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c1a6510 con 0x7ff52c107720 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cce0 tx=0x7ff52400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff51b7fe640 1 -- 172.21.15.38:0/629716261 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52400f450 con 0x7ff52c107720 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c1a6800 con 0x7ff52c107720 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff51b7fe640 1 -- 172.21.15.38:0/629716261 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff52400fad0 con 0x7ff52c107720 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff51b7fe640 1 -- 172.21.15.38:0/629716261 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff524018c10 con 0x7ff52c107720 2024-04-22T00:08:08.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.108+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a6d00 con 0x7ff52c107720 2024-04-22T00:08:08.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff51b7fe640 1 -- 172.21.15.38:0/629716261 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff52400f5f0 con 0x7ff52c107720 2024-04-22T00:08:08.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 Processor -- start 2024-04-22T00:08:08.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 -- start start 2024-04-22T00:08:08.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff51b7fe640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 0x7ff5140a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b3fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff52a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff51b7fe640 1 -- 172.21.15.38:0/629716261 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff5240cfd10 con 0x7ff52c107720 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff529d9b640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 0x7ff5140a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff52a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33054/0 (socket says 172.21.15.38:33054) 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff52a59c640 1 -- 172.21.15.38:0/1207003436 learned_addr learned my addr 172.21.15.38:0/1207003436 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b4b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f94aa3170 con 0x559f94b40720 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559f94a4b7c0 con 0x559f94b3fa40 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.111+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f94a9f130 con 0x559f94b48db0 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b3fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b4b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 -- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 msgr2=0x559f94b4b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b4b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 -- 172.21.15.38:0/1207003436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 msgr2=0x559f94b3fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b3fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 -- 172.21.15.38:0/1207003436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f94a5ce60 con 0x559f94b40720 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b4b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:08.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff529d9b640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 0x7ff5140a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff520004650 tx=0x7ff52000bef0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02cfd0 tx=0x7ff51c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2cb749fcb99ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff518ff9640 1 -- 172.21.15.38:0/1207003436 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c042070 con 0x559f94b40720 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff518ff9640 1 -- 172.21.15.38:0/1207003436 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51c02ca80 con 0x559f94b40720 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.112+0000 7ff518ff9640 1 -- 172.21.15.38:0/1207003436 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004f50 con 0x559f94b40720 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 msgr2=0x559f94b3f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02cfd0 tx=0x7ff51c02c5d0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 shutdown_connections 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b4b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b3fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 --2- 172.21.15.38:0/1207003436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b3f470 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 >> 172.21.15.38:0/1207003436 conn(0x559f94b064f0 msgr2=0x559f94b3a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 shutdown_connections 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- 172.21.15.38:0/1207003436 wait complete. 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 -- start start 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.113+0000 7ff533a17740 1 Processor -- start 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b45d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f94b4bbc0 con 0x559f94b40720 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559f94a5ed40 con 0x559f94b3fa40 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f94b4bd40 con 0x559f94b48db0 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 10 client.0 _ll_register_callbacks cb 0x559f94a9a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:08.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33060/0 (socket says 172.21.15.38:33060) 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 -- 172.21.15.38:0/1322120292 learned_addr learned my addr 172.21.15.38:0/1322120292 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff52ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b45d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 -- 172.21.15.38:0/1322120292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 msgr2=0x559f94b46780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b46780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 -- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 msgr2=0x559f94b45d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b45d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 -- 172.21.15.38:0/1322120292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f9487f7e0 con 0x559f94b40720 2024-04-22T00:08:08.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b46780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff520003f80 tx=0x7ff520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678ad6b17a009643 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520005020 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff5200051c0 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff51c02ca80 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff52002d050 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 10 client.17676 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.114+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f948dff70 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51c02ca80 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff520020670 con 0x559f94b40720 2024-04-22T00:08:08.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 10 client.17676 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 20 client.17676 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94b58df0 0x559f94b5b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 4 client.17676 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f94a77450 con 0x559f94b58df0 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94b58df0 0x559f94b5b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff511ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.115+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94b58df0 0x559f94b5b210 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8d08f8f45c30acf server_cookie=3fc22b37ca3eeae3 in_seq=0 out_seq=0 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff511ffb640 1 -- 172.21.15.38:0/1322120292 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f94a77450 con 0x559f94b58df0 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff511ffb640 10 client.17676 handle_command_reply: tid=0 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 client.17676 shutdown 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 msgr2=0x559f94b46240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff520003f80 tx=0x7ff520004730 comp rx=0 tx=0).stop 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 shutdown_connections 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b48db0 0x559f94b46780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94b58df0 0x559f94b5b210 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b40720 0x559f94b46240 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff511ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 --2- 172.21.15.38:0/1322120292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b3fa40 0x559f94b45d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 >> 172.21.15.38:0/1322120292 conn(0x559f94b064f0 msgr2=0x559f94b3c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 shutdown_connections 2024-04-22T00:08:08.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 1 -- 172.21.15.38:0/1322120292 wait complete. 2024-04-22T00:08:08.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.116+0000 7ff533a17740 20 client.17676 trim_cache size 0 max 16384 2024-04-22T00:08:08.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 Processor -- start 2024-04-22T00:08:08.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 -- start start 2024-04-22T00:08:08.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b3fa40 0x559f94b48430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 0x559f94b54af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1b0b10 con 0x559f94b48db0 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1b2650 con 0x559f94b40720 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff533a17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1b11e0 con 0x559f94b3fa40 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff52ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff529d9b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 0x559f94b54af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff52ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33062/0 (socket says 172.21.15.38:33062) 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff52ad9d640 1 -- 172.21.15.38:0/1992895234 learned_addr learned my addr 172.21.15.38:0/1992895234 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.136+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b3fa40 0x559f94b48430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 -- 172.21.15.38:0/1992895234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b3fa40 msgr2=0x559f94b48430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b3fa40 0x559f94b48430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 -- 172.21.15.38:0/1992895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 msgr2=0x559f94b54af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 0x559f94b54af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 -- 172.21.15.38:0/1992895234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f94777170 con 0x559f94b48db0 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 0x559f94b54af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff5240cd720 tx=0x7ff5240077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3db1f6d1e8ef1f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff513fff640 1 -- 172.21.15.38:0/1992895234 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240b4070 con 0x559f94b48db0 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff513fff640 1 -- 172.21.15.38:0/1992895234 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5240a4b60 con 0x559f94b48db0 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.137+0000 7ff513fff640 1 -- 172.21.15.38:0/1992895234 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240a4930 con 0x559f94b48db0 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- 172.21.15.38:0/1992895234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 msgr2=0x559f94b44c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff5240cd720 tx=0x7ff5240077a0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- 172.21.15.38:0/1992895234 shutdown_connections 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b48db0 0x559f94b44c40 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b40720 0x559f94b54af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- 172.21.15.38:0/1992895234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b3fa40 0x559f94b48430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- 172.21.15.38:0/1992895234 >> 172.21.15.38:0/1992895234 conn(0x559f94b064f0 msgr2=0x559f94b394f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- 172.21.15.38:0/1992895234 shutdown_connections 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- 172.21.15.38:0/1992895234 wait complete. 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- start start 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 Processor -- start 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 0x559f94b43340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 0x559f94bbe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1b1f80 con 0x559f94b3fa40 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1b18b0 con 0x559f94b48db0 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c1aeaf0 con 0x559f94b40720 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.138+0000 7ff533a17740 10 client.0 _ll_register_callbacks cb 0x7ff5200042f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:08.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33078/0 (socket says 172.21.15.38:33078) 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 -- 172.21.15.38:0/1785553019 learned_addr learned my addr 172.21.15.38:0/1785553019 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 0x559f94bbe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 0x559f94b43340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 -- 172.21.15.38:0/1785553019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 msgr2=0x559f94b43340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 0x559f94b43340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 -- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 msgr2=0x559f94bbe350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 0x559f94bbe350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 -- 172.21.15.38:0/1785553019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f94bbe890 con 0x559f94b3fa40 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 0x559f94b43340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:08.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52ad9d640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 0x559f94bbe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff52a59c640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02c820 tx=0x7ff51c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d76649cb9bd4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c053060 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff51c04eb50 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5240a4b60 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff51c05a050 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff533a17740 10 client.17682 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.139+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559f94bbfa90 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5240a4b60 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff51c04e750 con 0x559f94b3fa40 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 10 client.17682 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:08.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 20 client.17682 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94bbfd70 0x559f94bc2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 4 client.17682 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559f94ab06b0 con 0x559f94bbfd70 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94bbfd70 0x559f94bc2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff511ffb640 10 client.17682 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.140+0000 7ff529d9b640 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94bbfd70 0x559f94bc2160 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9be94f9976d514 server_cookie=f9609c9e0615f19b in_seq=0 out_seq=0 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff511ffb640 1 -- 172.21.15.38:0/1785553019 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559f94ab06b0 con 0x559f94bbfd70 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff511ffb640 10 client.17682 handle_command_reply: tid=0 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 client.17682 shutdown 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 msgr2=0x559f94b42e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02c820 tx=0x7ff51c004ad0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 shutdown_connections 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559f94b48db0 0x559f94bbe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f94b40720 0x559f94b43340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559f94bbfd70 0x559f94bc2160 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff511ffb640 0 client.17682 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 --2- 172.21.15.38:0/1785553019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f94b3fa40 0x559f94b42e00 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.141+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 >> 172.21.15.38:0/1785553019 conn(0x559f94b064f0 msgr2=0x559f94b45cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 shutdown_connections 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff533a17740 1 -- 172.21.15.38:0/1785553019 wait complete. 2024-04-22T00:08:08.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff533a17740 20 client.17682 trim_cache size 0 max 16384 2024-04-22T00:08:08.146 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:08:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "84dad26c-603e-44ac-9fc0-721ed59d4fdd", 2024-04-22T00:08:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:08:08.146 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 msgr2=0x7ff5140a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 0x7ff5140a51f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff520004650 tx=0x7ff52000bef0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 msgr2=0x7ff52c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff52400cce0 tx=0x7ff52400a620 comp rx=0 tx=0).stop 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 shutdown_connections 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff5140a2d30 0x7ff5140a51f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff52c107720 0x7ff52c0744c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff52c1020d0 0x7ff52c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 --2- 172.21.15.38:0/629716261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff52c101710 0x7ff52c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 >> 172.21.15.38:0/629716261 conn(0x7ff52c0fd180 msgr2=0x7ff52c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 shutdown_connections 2024-04-22T00:08:08.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.142+0000 7ff530e37640 1 -- 172.21.15.38:0/629716261 wait complete. 2024-04-22T00:08:08.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:08:08.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:08:08.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 -- 172.21.15.38:0/3151341542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 msgr2=0x7fa8bc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 --2- 172.21.15.38:0/3151341542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc100aa0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009930 tx=0x7fa8ac02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:08.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 -- 172.21.15.38:0/3151341542 shutdown_connections 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 --2- 172.21.15.38:0/3151341542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 --2- 172.21.15.38:0/3151341542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc100aa0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 --2- 172.21.15.38:0/3151341542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc106630 0x7fa8bc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.267+0000 7fa8c418e640 1 -- 172.21.15.38:0/3151341542 >> 172.21.15.38:0/3151341542 conn(0x7fa8bc0776a0 msgr2=0x7fa8bc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.268+0000 7fa8c418e640 1 -- 172.21.15.38:0/3151341542 shutdown_connections 2024-04-22T00:08:08.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.268+0000 7fa8c418e640 1 -- 172.21.15.38:0/3151341542 wait complete. 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 Processor -- start 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 -- start start 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc106630 0x7fa8bc19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc110a90 con 0x7fa8bc100620 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc110910 con 0x7fa8bc100fe0 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c418e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc110c10 con 0x7fa8bc106630 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33086/0 (socket says 172.21.15.38:33086) 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/358804898 learned_addr learned my addr 172.21.15.38:0/358804898 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1729640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c272b640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc106630 0x7fa8bc19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/358804898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc106630 msgr2=0x7fa8bc19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc106630 0x7fa8bc19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/358804898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 msgr2=0x7fa8bc198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/358804898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc19d030 con 0x7fa8bc100620 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.269+0000 7fa8c1729640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800d950 tx=0x7fa8b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8b2ffd640 1 -- 172.21.15.38:0/358804898 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8017070 con 0x7fa8bc100620 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8b2ffd640 1 -- 172.21.15.38:0/358804898 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8009c30 con 0x7fa8bc100620 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc19d320 con 0x7fa8bc100620 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8b2ffd640 1 -- 172.21.15.38:0/358804898 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800f6b0 con 0x7fa8bc100620 2024-04-22T00:08:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.270+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a4bb0 con 0x7fa8bc100620 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.272+0000 7fa8b2ffd640 1 -- 172.21.15.38:0/358804898 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa8b800f850 con 0x7fa8bc100620 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8b2ffd640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 0x7fa8a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 Processor -- start 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c1729640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 0x7fa8a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8b2ffd640 1 -- 172.21.15.38:0/358804898 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa8b80cf300 con 0x7fa8bc100620 2024-04-22T00:08:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 -- start start 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d33ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 0x558bc4d36820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558bc4c8f560 con 0x558bc4d2af30 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558bc4c37730 con 0x558bc4d34430 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558bc4c8b520 con 0x558bc4d2b850 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c1f2a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.273+0000 7fa8c1729640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 0x7fa8a80a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009900 tx=0x7fa8ac02df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1f2a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33094/0 (socket says 172.21.15.38:33094) 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d33ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/1114603218 learned_addr learned my addr 172.21.15.38:0/1114603218 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c272b640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 0x558bc4d36820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 -- 172.21.15.38:0/1114603218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 msgr2=0x558bc4d36820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 0x558bc4d36820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 -- 172.21.15.38:0/1114603218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 msgr2=0x558bc4d2b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 -- 172.21.15.38:0/1114603218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bc4a6d7d0 con 0x558bc4d2b850 2024-04-22T00:08:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c272b640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 0x558bc4d36820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8c1729640 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d33ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036040 tx=0x7fa8ac03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db7704e5d7ec8e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8a7fff640 1 -- 172.21.15.38:0/1114603218 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0042c0 con 0x558bc4d2b850 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8a7fff640 1 -- 172.21.15.38:0/1114603218 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac050070 con 0x558bc4d2b850 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.274+0000 7fa8a7fff640 1 -- 172.21.15.38:0/1114603218 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0055b0 con 0x558bc4d2b850 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- 172.21.15.38:0/1114603218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 msgr2=0x558bc4d33ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d33ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036040 tx=0x7fa8ac03f000 comp rx=0 tx=0).stop 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- 172.21.15.38:0/1114603218 shutdown_connections 2024-04-22T00:08:08.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d34430 0x558bc4d36820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d33ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/1114603218 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2af30 0x558bc4d2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- 172.21.15.38:0/1114603218 >> 172.21.15.38:0/1114603218 conn(0x558bc4d25660 msgr2=0x558bc4d25a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- 172.21.15.38:0/1114603218 shutdown_connections 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- 172.21.15.38:0/1114603218 wait complete. 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.275+0000 7fa8c6d70740 1 -- start start 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 Processor -- start 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d31490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d31f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558bc4d371c0 con 0x558bc4d2b850 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558bc4c4ae00 con 0x558bc4d2af30 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558bc4d37340 con 0x558bc4d34430 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c6d70740 10 client.0 _ll_register_callbacks cb 0x558bc4c85eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1729640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1729640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33108/0 (socket says 172.21.15.38:33108) 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d31490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1729640 1 -- 172.21.15.38:0/3877995728 learned_addr learned my addr 172.21.15.38:0/3877995728 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c272b640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d31f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/3877995728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 msgr2=0x558bc4d31f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d31f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/3877995728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 msgr2=0x558bc4d319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 -- 172.21.15.38:0/3877995728 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bc49a33f0 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d31490 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80979f0 tx=0x7fa8b8010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a96e484c6961b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.276+0000 7fa8a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8020050 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa8b809a9e0 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac050070 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa8b80cce60 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 10 client.27041 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558bc49630f0 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac050070 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa8b80b4200 con 0x558bc4d2af30 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 10 client.27041 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:08.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 10 client.27041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 20 client.27041 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4d44720 0x558bc4d46b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 4 client.27041 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.277+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bc4c9caa0 con 0x558bc4d44720 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4d44720 0x558bc4d46b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4d44720 0x558bc4d46b50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c76cf87068b7777 server_cookie=d1716eed9691007a in_seq=0 out_seq=0 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8a57fa640 10 client.27041 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8a57fa640 1 -- 172.21.15.38:0/3877995728 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bc4c9caa0 con 0x558bc4d44720 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8a57fa640 10 client.27041 handle_command_reply: tid=0 2024-04-22T00:08:08.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.278+0000 7fa8c6d70740 1 client.27041 shutdown 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 msgr2=0x558bc4d31490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d31490 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80979f0 tx=0x7fa8b8010a90 comp rx=0 tx=0).stop 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 shutdown_connections 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d31f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4d44720 0x558bc4d46b50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8a57fa640 0 client.27041 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3877995728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d31490 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 >> 172.21.15.38:0/3877995728 conn(0x558bc4d25660 msgr2=0x558bc4d273b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 shutdown_connections 2024-04-22T00:08:08.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3877995728 wait complete. 2024-04-22T00:08:08.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.279+0000 7fa8c6d70740 20 client.27041 trim_cache size 0 max 16384 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 Processor -- start 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 -- start start 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d33bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d2bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1aea50 con 0x558bc4d2b850 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b0590 con 0x558bc4d2af30 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1af120 con 0x558bc4d34430 2024-04-22T00:08:08.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33122/0 (socket says 172.21.15.38:33122) 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 -- 172.21.15.38:0/3229774925 learned_addr learned my addr 172.21.15.38:0/3229774925 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d33bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c272b640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d2bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 -- 172.21.15.38:0/3229774925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 msgr2=0x558bc4d2bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d2bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 -- 172.21.15.38:0/3229774925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 msgr2=0x558bc4d33bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d33bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c1729640 1 -- 172.21.15.38:0/3229774925 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bc4d30440 con 0x558bc4d2b850 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.301+0000 7fa8c272b640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c1729640 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c240 tx=0x7fa8ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d701be040ac06630 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8a77fe640 1 -- 172.21.15.38:0/3229774925 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0055b0 con 0x558bc4d2b850 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8a77fe640 1 -- 172.21.15.38:0/3229774925 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac050040 con 0x558bc4d2b850 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8a77fe640 1 -- 172.21.15.38:0/3229774925 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac05e8f0 con 0x558bc4d2b850 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3229774925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 msgr2=0x558bc4d40080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c240 tx=0x7fa8ac02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3229774925 shutdown_connections 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d34430 0x558bc4d2bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.302+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d2b850 0x558bc4d40080 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/3229774925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d33bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3229774925 >> 172.21.15.38:0/3229774925 conn(0x558bc4d25660 msgr2=0x558bc4d23e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3229774925 shutdown_connections 2024-04-22T00:08:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- 172.21.15.38:0/3229774925 wait complete. 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- start start 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 Processor -- start 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d3c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d2de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1afec0 con 0x558bc4d34430 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1af7f0 con 0x558bc4d2af30 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1aca30 con 0x558bc4d2b850 2024-04-22T00:08:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c6d70740 10 client.0 _ll_register_callbacks cb 0x7fa8b809aef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c272b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c272b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33126/0 (socket says 172.21.15.38:33126) 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c272b640 1 -- 172.21.15.38:0/667553107 learned_addr learned my addr 172.21.15.38:0/667553107 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c1729640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d2de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.303+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d3c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 -- 172.21.15.38:0/667553107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 msgr2=0x558bc4d2de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d2de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 -- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 msgr2=0x558bc4d3c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d3c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 -- 172.21.15.38:0/667553107 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bc4d2ea80 con 0x558bc4d34430 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d3c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c1729640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d2de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c272b640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40148f0 tx=0x7fa8b4014db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86f3d799ca0b22d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4004b20 con 0x558bc4d34430 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fa8b4004cc0 con 0x558bc4d34430 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac050040 con 0x558bc4d34430 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa8b4027050 con 0x558bc4d34430 2024-04-22T00:08:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 10 client.17700 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558bc4daac10 con 0x558bc4d34430 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac050040 con 0x558bc4d34430 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa8b400f040 con 0x558bc4d34430 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 10 client.17700 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 10 client.17700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 20 client.17700 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.304+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4dab3f0 0x558bc4dad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:08.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 4 client.17700 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558bc4c96df0 con 0x558bc4dab3f0 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4dab3f0 0x558bc4dad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8a57fa640 10 client.17700 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c1f2a640 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4dab3f0 0x558bc4dad810 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb9a364da46170c4 server_cookie=9d4d17c0eb730daf in_seq=0 out_seq=0 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8a57fa640 1 -- 172.21.15.38:0/667553107 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x558bc4c96df0 con 0x558bc4dab3f0 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8a57fa640 10 client.17700 handle_command_reply: tid=0 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 client.17700 shutdown 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 msgr2=0x558bc4d2e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40148f0 tx=0x7fa8b4014db0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 shutdown_connections 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.305+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558bc4dab3f0 0x558bc4dad810 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bc4d34430 0x558bc4d2e340 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558bc4d2b850 0x558bc4d2de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 --2- 172.21.15.38:0/667553107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558bc4d2af30 0x558bc4d3c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8a57fa640 0 client.17700 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 >> 172.21.15.38:0/667553107 conn(0x558bc4d25660 msgr2=0x558bc4d31ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 shutdown_connections 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 1 -- 172.21.15.38:0/667553107 wait complete. 2024-04-22T00:08:08.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c6d70740 20 client.17700 trim_cache size 0 max 16384 2024-04-22T00:08:08.313 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:08.313 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (214 inodes in the stack)", 2024-04-22T00:08:08.314 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:08:08.314 INFO:teuthology.orchestra.run.smithi038.stdout: "84dad26c-603e-44ac-9fc0-721ed59d4fdd": { 2024-04-22T00:08:08.314 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:08:08.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "84dad26c-603e-44ac-9fc0-721ed59d4fdd", 2024-04-22T00:08:08.315 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:08:08.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:08:08.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:08:08.315 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 msgr2=0x7fa8a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 0x7fa8a80a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009900 tx=0x7fa8ac02df90 comp rx=0 tx=0).stop 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 msgr2=0x7fa8bc198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800d950 tx=0x7fa8b800de10 comp rx=0 tx=0).stop 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 shutdown_connections 2024-04-22T00:08:08.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa8a80a2d00 0x7fa8a80a51c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8bc106630 0x7fa8bc19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8bc100fe0 0x7fa8bc198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 --2- 172.21.15.38:0/358804898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8bc100620 0x7fa8bc198040 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 >> 172.21.15.38:0/358804898 conn(0x7fa8bc0776a0 msgr2=0x7fa8bc069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 shutdown_connections 2024-04-22T00:08:08.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:08.306+0000 7fa8c418e640 1 -- 172.21.15.38:0/358804898 wait complete. 2024-04-22T00:08:08.318 INFO:tasks.cephfs.filesystem:scrub status for tag:84dad26c-603e-44ac-9fc0-721ed59d4fdd - {'path': '/', 'tag': '84dad26c-603e-44ac-9fc0-721ed59d4fdd', 'options': 'recursive,force'} 2024-04-22T00:08:08.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:08 smithi038 ceph-mon[27972]: pgmap v184: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-04-22T00:08:08.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:08 smithi203 ceph-mon[31431]: pgmap v184: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-04-22T00:08:08.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:08 smithi111 ceph-mon[31543]: pgmap v184: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-04-22T00:08:09.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:09 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:08:09.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:09 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:09.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:09 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:09.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:09 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:08:09.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:09 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:09.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:09 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:09.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:09 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:08:09.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:09 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:09.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:09 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:10.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:10 smithi038 ceph-mon[27972]: pgmap v185: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:08:10.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:10 smithi203 ceph-mon[31431]: pgmap v185: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:08:10.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:10 smithi111 ceph-mon[31543]: pgmap v185: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 -- 172.21.15.38:0/1901503730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 msgr2=0x7f5324116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 --2- 172.21.15.38:0/1901503730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f5324116c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5320009930 tx=0x7f532002bcd0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 -- 172.21.15.38:0/1901503730 shutdown_connections 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 --2- 172.21.15.38:0/1901503730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f5324116c80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 --2- 172.21.15.38:0/1901503730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324109060 0x7f5324114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 --2- 172.21.15.38:0/1901503730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324108740 0x7f5324108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 -- 172.21.15.38:0/1901503730 >> 172.21.15.38:0/1901503730 conn(0x7f532406c810 msgr2=0x7f532406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 -- 172.21.15.38:0/1901503730 shutdown_connections 2024-04-22T00:08:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.241+0000 7f532c15c640 1 -- 172.21.15.38:0/1901503730 wait complete. 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 Processor -- start 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 -- start start 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324108740 0x7f53241aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 0x7f53241acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f532410ac50 con 0x7f5324109060 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f532410aad0 con 0x7f5324108740 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.242+0000 7f532c15c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f532410add0 con 0x7f5324114890 2024-04-22T00:08:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f53296f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 0x7f53241acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f53296f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 0x7f53241acf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33146/0 (socket says 172.21.15.38:33146) 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f53296f7640 1 -- 172.21.15.38:0/488826636 learned_addr learned my addr 172.21.15.38:0/488826636 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34310/0 (socket says 172.21.15.38:34310) 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324108740 0x7f53241aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 -- 172.21.15.38:0/488826636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324108740 msgr2=0x7f53241aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324108740 0x7f53241aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 -- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 msgr2=0x7f53241acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 0x7f53241acf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.243+0000 7f532a6f9640 1 -- 172.21.15.38:0/488826636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53241a73d0 con 0x7f5324114890 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f532a6f9640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f532002c210 tx=0x7f532002cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f531affd640 1 -- 172.21.15.38:0/488826636 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320041070 con 0x7f5324114890 2024-04-22T00:08:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f531affd640 1 -- 172.21.15.38:0/488826636 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5320004450 con 0x7f5324114890 2024-04-22T00:08:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f531affd640 1 -- 172.21.15.38:0/488826636 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320003d20 con 0x7f5324114890 2024-04-22T00:08:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241a7660 con 0x7f5324114890 2024-04-22T00:08:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.244+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241b37f0 con 0x7f5324114890 2024-04-22T00:08:12.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.247+0000 7f532ee1b740 1 Processor -- start 2024-04-22T00:08:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f531affd640 1 -- 172.21.15.38:0/488826636 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5320044070 con 0x7f5324114890 2024-04-22T00:08:12.243 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 -- start start 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f531affd640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 0x7f53100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa41cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa42210 0x55889fa44600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55889f99cb70 con 0x55889fa38b20 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55889f945470 con 0x55889fa42210 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f532ee1b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55889f998b30 con 0x55889fa39440 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.248+0000 7f531affd640 1 -- 172.21.15.38:0/488826636 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f5320003ac0 con 0x7f5324114890 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f532a6f9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa42210 0x55889fa44600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33162/0 (socket says 172.21.15.38:33162) 2024-04-22T00:08:12.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 -- 172.21.15.38:0/1771249830 learned_addr learned my addr 172.21.15.38:0/1771249830 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa41cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f5329ef8640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 0x7f53100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 -- 172.21.15.38:0/1771249830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 msgr2=0x55889fa41cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa41cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 -- 172.21.15.38:0/1771249830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa42210 msgr2=0x55889fa44600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa42210 0x55889fa44600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 -- 172.21.15.38:0/1771249830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55889f6b1210 con 0x55889fa38b20 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f5329ef8640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 0x7f53100a5460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f531400abe0 tx=0x7f531400a2d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.249+0000 7f53296f7640 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f531c009510 tx=0x7f531c009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b725e433554bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.250+0000 7f530ffff640 1 -- 172.21.15.38:0/1771249830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c003970 con 0x55889fa38b20 2024-04-22T00:08:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.250+0000 7f530ffff640 1 -- 172.21.15.38:0/1771249830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f531c011780 con 0x55889fa38b20 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.250+0000 7f530ffff640 1 -- 172.21.15.38:0/1771249830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c004430 con 0x55889fa38b20 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 -- 172.21.15.38:0/1771249830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 msgr2=0x55889fa38f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f531c009510 tx=0x7f531c009b40 comp rx=0 tx=0).stop 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 -- 172.21.15.38:0/1771249830 shutdown_connections 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa42210 0x55889fa44600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa41cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 --2- 172.21.15.38:0/1771249830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa38b20 0x55889fa38f00 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 -- 172.21.15.38:0/1771249830 >> 172.21.15.38:0/1771249830 conn(0x55889fa33440 msgr2=0x55889fa33820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 -- 172.21.15.38:0/1771249830 shutdown_connections 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.251+0000 7f532ee1b740 1 -- 172.21.15.38:0/1771249830 wait complete. 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 -- start start 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 Processor -- start 2024-04-22T00:08:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa3f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 0x55889fa3fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55889fa44de0 con 0x55889fa42210 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f53296f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa3f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58928/0 (socket says 172.21.15.38:58928) 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f5329ef8640 1 -- 172.21.15.38:0/2091299446 learned_addr learned my addr 172.21.15.38:0/2091299446 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f53296f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa3f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34316/0 (socket says 172.21.15.38:34316) 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55889f958a10 con 0x55889fa38b20 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55889fa44f60 con 0x55889fa39440 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.252+0000 7f532ee1b740 10 client.0 _ll_register_callbacks cb 0x55889f993b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f532a6f9640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 0x55889fa3fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 -- 172.21.15.38:0/2091299446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 msgr2=0x55889fa3f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa3f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 -- 172.21.15.38:0/2091299446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 msgr2=0x55889fa3fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 0x55889fa3fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55889f77d620 con 0x55889fa38b20 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f532a6f9640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 0x55889fa3fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f5329ef8640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5314005e10 tx=0x7f5314004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b9103452c99edb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f530d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314016070 con 0x55889fa38b20 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.253+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f531400fe40 con 0x55889fa38b20 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.254+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f531c011780 con 0x55889fa38b20 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.254+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5314027480 con 0x55889fa38b20 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.254+0000 7f532ee1b740 10 client.27056 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.254+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55889fa51130 con 0x55889fa38b20 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f531c011780 con 0x55889fa38b20 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f531401c6d0 con 0x55889fa38b20 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 10 client.27056 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 10 client.27056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 20 client.27056 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fa525f0 0x55889fa549e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 4 client.27056 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55889f9aa0b0 con 0x55889fa525f0 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.255+0000 7f53296f7640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fa525f0 0x55889fa549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.257+0000 7f530d7fa640 10 client.27056 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.257+0000 7f53296f7640 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fa525f0 0x55889fa549e0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=356acda8981d664c server_cookie=6a43606731d39d8 in_seq=0 out_seq=0 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.258+0000 7f530d7fa640 1 -- 172.21.15.38:0/2091299446 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55889f9aa0b0 con 0x55889fa525f0 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.258+0000 7f530d7fa640 10 client.27056 handle_command_reply: tid=0 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.258+0000 7f532ee1b740 1 client.27056 shutdown 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 msgr2=0x55889fa3f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5314005e10 tx=0x7f5314004730 comp rx=0 tx=0).stop 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 shutdown_connections 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fa525f0 0x55889fa549e0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f530d7fa640 0 client.27056 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa42210 0x55889fa3fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa39440 0x55889fa3f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2091299446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa38b20 0x55889fa3f2c0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5314005e10 tx=0x7f5314004730 comp rx=0 tx=0).stop 2024-04-22T00:08:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.259+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 >> 172.21.15.38:0/2091299446 conn(0x55889fa33440 msgr2=0x55889fa35190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.261+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 shutdown_connections 2024-04-22T00:08:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.261+0000 7f532ee1b740 1 -- 172.21.15.38:0/2091299446 wait complete. 2024-04-22T00:08:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.261+0000 7f532ee1b740 20 client.27056 trim_cache size 0 max 16384 2024-04-22T00:08:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.289+0000 7f532ee1b740 1 Processor -- start 2024-04-22T00:08:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f532ee1b740 1 -- start start 2024-04-22T00:08:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa2e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa4dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:58930/0 (socket says 172.21.15.38:58930) 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f5329ef8640 1 -- 172.21.15.38:0/263388839 learned_addr learned my addr 172.21.15.38:0/263388839 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.290+0000 7f532a6f9640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa4dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f53296f7640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa2e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53241bd7d0 con 0x55889fa525f0 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53241bf310 con 0x55889fa39440 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53241bdea0 con 0x55889fa42210 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f5329ef8640 1 -- 172.21.15.38:0/263388839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 msgr2=0x55889fa2e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f5329ef8640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa2e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f5329ef8640 1 -- 172.21.15.38:0/263388839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 msgr2=0x55889fa4dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f5329ef8640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa4dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f5329ef8640 1 -- 172.21.15.38:0/263388839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55889fa382c0 con 0x55889fa39440 2024-04-22T00:08:12.288 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.291+0000 7f532a6f9640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa4dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.292+0000 7f5329ef8640 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f531401cce0 tx=0x7f5314005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37dcc87120648043 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.292+0000 7f530f7fe640 1 -- 172.21.15.38:0/263388839 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400fc20 con 0x55889fa39440 2024-04-22T00:08:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.292+0000 7f530f7fe640 1 -- 172.21.15.38:0/263388839 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f531400fdc0 con 0x55889fa39440 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.292+0000 7f530f7fe640 1 -- 172.21.15.38:0/263388839 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314027a50 con 0x55889fa39440 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 msgr2=0x55889fa419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f531401cce0 tx=0x7f5314005d80 comp rx=0 tx=0).stop 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 shutdown_connections 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa4dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa2e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- 172.21.15.38:0/263388839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa419f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 >> 172.21.15.38:0/263388839 conn(0x55889fa33440 msgr2=0x55889fa3f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 shutdown_connections 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- 172.21.15.38:0/263388839 wait complete. 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 -- start start 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 Processor -- start 2024-04-22T00:08:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.293+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f53241bec40 con 0x55889fa525f0 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f53241be570 con 0x55889fa39440 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53241bb7b0 con 0x55889fa42210 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 10 client.0 _ll_register_callbacks cb 0x7f531400a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f53296f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33190/0 (socket says 172.21.15.38:33190) 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f53296f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34324/0 (socket says 172.21.15.38:34324) 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 -- 172.21.15.38:0/2145588531 learned_addr learned my addr 172.21.15.38:0/2145588531 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f5329ef8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 -- 172.21.15.38:0/2145588531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 msgr2=0x55889fa46030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 -- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 msgr2=0x55889fa45af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa45af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 -- 172.21.15.38:0/2145588531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55889fab7b20 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f53296f7640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532a6f9640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f53200ee420 tx=0x7f53200bcaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d90a4f33038b947 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f530d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53200395b0 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f531400fdc0 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.294+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f53200b9800 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5320045dc0 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 10 client.17715 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55889fab8d40 con 0x55889fa525f0 2024-04-22T00:08:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f531400fdc0 con 0x55889fa525f0 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5320039230 con 0x55889fa525f0 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 10 client.17715 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 10 client.17715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 20 client.17715 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fab9460 0x55889fabb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 4 client.17715 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55889f9a4400 con 0x55889fab9460 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.295+0000 7f5329ef8640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fab9460 0x55889fabb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.296+0000 7f530d7fa640 10 client.17715 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:12.296+0000 7f5329ef8640 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fab9460 0x55889fabb880 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0012f1c7b76fc0f server_cookie=7984302dc4df3c30 in_seq=0 out_seq=0 2024-04-22T00:08:12.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:12 smithi203 ceph-mon[31431]: pgmap v186: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 8.5 MiB/s wr, 2.12k op/s 2024-04-22T00:08:12.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:12 smithi111 ceph-mon[31543]: pgmap v186: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 8.5 MiB/s wr, 2.12k op/s 2024-04-22T00:08:12.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:12 smithi038 ceph-mon[27972]: pgmap v186: 129 pgs: 129 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 8.5 MiB/s wr, 2.12k op/s 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f530d7fa640 1 -- 172.21.15.38:0/2145588531 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55889f9a4400 con 0x55889fab9460 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f530d7fa640 10 client.17715 handle_command_reply: tid=0 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 client.17715 shutdown 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 msgr2=0x55889fa46570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f53200ee420 tx=0x7f53200bcaf0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 shutdown_connections 2024-04-22T00:08:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55889fab9460 0x55889fabb880 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f530d7fa640 0 client.17715 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55889fa525f0 0x55889fa46570 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55889fa42210 0x55889fa46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 --2- 172.21.15.38:0/2145588531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55889fa39440 0x55889fa45af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 >> 172.21.15.38:0/2145588531 conn(0x55889fa33440 msgr2=0x55889fa34fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 shutdown_connections 2024-04-22T00:08:13.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.322+0000 7f532ee1b740 1 -- 172.21.15.38:0/2145588531 wait complete. 2024-04-22T00:08:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532ee1b740 20 client.17715 trim_cache size 0 max 16384 2024-04-22T00:08:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:08:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:08:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:08:13.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 msgr2=0x7f53100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 0x7f53100a5460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f531400abe0 tx=0x7f531400a2d0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 msgr2=0x7f53241a6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f532002c210 tx=0x7f532002cc90 comp rx=0 tx=0).stop 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 shutdown_connections 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53100a2fa0 0x7f53100a5460 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5324114890 0x7f53241a6b10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5324109060 0x7f53241acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 --2- 172.21.15.38:0/488826636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5324108740 0x7f53241aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:13.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 >> 172.21.15.38:0/488826636 conn(0x7f532406c810 msgr2=0x7f5324072ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:13.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.323+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 shutdown_connections 2024-04-22T00:08:13.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:13.324+0000 7f532c15c640 1 -- 172.21.15.38:0/488826636 wait complete. 2024-04-22T00:08:14.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:14 smithi203 ceph-mon[31431]: pgmap v187: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-04-22T00:08:14.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:14 smithi111 ceph-mon[31543]: pgmap v187: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-04-22T00:08:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:14 smithi038 ceph-mon[27972]: pgmap v187: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 3.23k op/s 2024-04-22T00:08:16.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:16 smithi203 ceph-mon[31431]: pgmap v188: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:08:16.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:16 smithi111 ceph-mon[31543]: pgmap v188: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:08:16.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:16 smithi038 ceph-mon[27972]: pgmap v188: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.9 MiB/s wr, 2.46k op/s 2024-04-22T00:08:18.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:18 smithi203 ceph-mon[31431]: pgmap v189: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:18.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:18 smithi111 ceph-mon[31543]: pgmap v189: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:18.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:18 smithi038 ceph-mon[27972]: pgmap v189: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:20.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:20 smithi203 ceph-mon[31431]: pgmap v190: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:20.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:20 smithi111 ceph-mon[31543]: pgmap v190: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:20.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:20 smithi038 ceph-mon[27972]: pgmap v190: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:22.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:22 smithi203 ceph-mon[31431]: pgmap v191: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-22T00:08:22.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:22 smithi111 ceph-mon[31543]: pgmap v191: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-22T00:08:22.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:22 smithi038 ceph-mon[27972]: pgmap v191: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 2.43k op/s 2024-04-22T00:08:23.829 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:08:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:08:23.536+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v590>,<0x100000001fb/client.0 v568>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v568>,<0x10000000001/sv_1 v580>,<0x10000000000/qa v560>,<0x1/volumes v542>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:08:24.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: pgmap v192: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:08:24.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:24.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:24 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: pgmap v192: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:08:24.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:24.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:24 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: pgmap v192: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 14 MiB/s wr, 3.39k op/s 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:24.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:24 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:25 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:25.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:25 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:25 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:25 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:25.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:25 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:25 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:25 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:25.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:25 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:25.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:25 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.469+0000 7f1996b9e640 1 -- 172.21.15.38:0/3317266504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19901006c0 msgr2=0x7f1990104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.469+0000 7f1996b9e640 1 --2- 172.21.15.38:0/3317266504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19901006c0 0x7f1990104380 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1988009ca0 tx=0x7f198802c140 comp rx=0 tx=0).stop 2024-04-22T00:08:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.472+0000 7f1996b9e640 1 -- 172.21.15.38:0/3317266504 shutdown_connections 2024-04-22T00:08:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.472+0000 7f1996b9e640 1 --2- 172.21.15.38:0/3317266504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19901006c0 0x7f1990104380 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.472+0000 7f1996b9e640 1 --2- 172.21.15.38:0/3317266504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19900ffc40 0x7f19901000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.472+0000 7f1996b9e640 1 --2- 172.21.15.38:0/3317266504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f1990106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.472+0000 7f1996b9e640 1 -- 172.21.15.38:0/3317266504 >> 172.21.15.38:0/3317266504 conn(0x7f1990076bd0 msgr2=0x7f19900fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.477+0000 7f1996b9e640 1 -- 172.21.15.38:0/3317266504 shutdown_connections 2024-04-22T00:08:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.477+0000 7f1996b9e640 1 -- 172.21.15.38:0/3317266504 wait complete. 2024-04-22T00:08:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.478+0000 7f1996b9e640 1 Processor -- start 2024-04-22T00:08:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.478+0000 7f1996b9e640 1 -- start start 2024-04-22T00:08:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.478+0000 7f1996b9e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19900ffc40 0x7f19901a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f1996b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19901006c0 0x7f19901a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f1996b9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f1996b9e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1990118410 con 0x7f1990106880 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f1996b9e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1990118290 con 0x7f19900ffc40 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f1996b9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1990118590 con 0x7f19901006c0 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f198ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19901006c0 0x7f19901a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f199513b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f199513b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49704/0 (socket says 172.21.15.38:49704) 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f199513b640 1 -- 172.21.15.38:0/633283299 learned_addr learned my addr 172.21.15.38:0/633283299 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.479+0000 7f199493a640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19900ffc40 0x7f19901a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 -- 172.21.15.38:0/633283299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19901006c0 msgr2=0x7f19901a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19901006c0 0x7f19901a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 -- 172.21.15.38:0/633283299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19900ffc40 msgr2=0x7f19901a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19900ffc40 0x7f19901a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 -- 172.21.15.38:0/633283299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199019eee0 con 0x7f1990106880 2024-04-22T00:08:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.481+0000 7f199513b640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f1988037040 tx=0x7f1988035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.482+0000 7f198dffb640 1 -- 172.21.15.38:0/633283299 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988039040 con 0x7f1990106880 2024-04-22T00:08:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.483+0000 7f198dffb640 1 -- 172.21.15.38:0/633283299 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1988044070 con 0x7f1990106880 2024-04-22T00:08:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.483+0000 7f198dffb640 1 -- 172.21.15.38:0/633283299 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988004630 con 0x7f1990106880 2024-04-22T00:08:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.484+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199019f170 con 0x7f1990106880 2024-04-22T00:08:26.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.484+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199006b160 con 0x7f1990106880 2024-04-22T00:08:26.483 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.488+0000 7f1999783740 1 Processor -- start 2024-04-22T00:08:26.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.488+0000 7f1999783740 1 -- start start 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3411b20 0x5569e3411f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3412440 0x5569e341acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5569e3375b70 con 0x5569e3412440 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5569e331e470 con 0x5569e3411b20 2024-04-22T00:08:26.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.489+0000 7f1999783740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569e3371b30 con 0x5569e341b210 2024-04-22T00:08:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.491+0000 7f198ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.491+0000 7f198ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:51544/0 (socket says 172.21.15.38:51544) 2024-04-22T00:08:26.486 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.491+0000 7f198ffff640 1 -- 172.21.15.38:0/1603001451 learned_addr learned my addr 172.21.15.38:0/1603001451 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.492+0000 7f198ffff640 1 -- 172.21.15.38:0/1603001451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3411b20 msgr2=0x5569e3411f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.492+0000 7f198ffff640 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3411b20 0x5569e3411f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.492+0000 7f198ffff640 1 -- 172.21.15.38:0/1603001451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3412440 msgr2=0x5569e341acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.492+0000 7f198ffff640 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3412440 0x5569e341acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.492+0000 7f198ffff640 1 -- 172.21.15.38:0/1603001451 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569e308a210 con 0x5569e341b210 2024-04-22T00:08:26.488 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.493+0000 7f198ffff640 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1978009a90 tx=0x7f1978009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f8c083326e15b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.494+0000 7f1986ffd640 1 -- 172.21.15.38:0/1603001451 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978018020 con 0x5569e341b210 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.494+0000 7f1986ffd640 1 -- 172.21.15.38:0/1603001451 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1978010c20 con 0x5569e341b210 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.494+0000 7f1986ffd640 1 -- 172.21.15.38:0/1603001451 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19780136c0 con 0x5569e341b210 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.495+0000 7f1999783740 1 -- 172.21.15.38:0/1603001451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 msgr2=0x5569e341d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.495+0000 7f1999783740 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1978009a90 tx=0x7f1978009f50 comp rx=0 tx=0).stop 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.505+0000 7f1999783740 1 -- 172.21.15.38:0/1603001451 shutdown_connections 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.505+0000 7f1999783740 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e341d600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.505+0000 7f1999783740 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3412440 0x5569e341acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.500 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.505+0000 7f1999783740 1 --2- 172.21.15.38:0/1603001451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3411b20 0x5569e3411f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.501 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.505+0000 7f1999783740 1 -- 172.21.15.38:0/1603001451 >> 172.21.15.38:0/1603001451 conn(0x5569e340c440 msgr2=0x5569e340c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.508+0000 7f198dffb640 1 -- 172.21.15.38:0/633283299 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1988003970 con 0x7f1990106880 2024-04-22T00:08:26.503 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.508+0000 7f1999783740 1 -- 172.21.15.38:0/1603001451 shutdown_connections 2024-04-22T00:08:26.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.509+0000 7f198dffb640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 0x7f197c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.504 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.509+0000 7f198dffb640 1 -- 172.21.15.38:0/633283299 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f19880f17a0 con 0x7f1990106880 2024-04-22T00:08:26.506 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.511+0000 7f199493a640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 0x7f197c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.512+0000 7f1999783740 1 -- 172.21.15.38:0/1603001451 wait complete. 2024-04-22T00:08:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.512+0000 7f1999783740 1 -- start start 2024-04-22T00:08:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.512+0000 7f1999783740 1 Processor -- start 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.513+0000 7f1999783740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.513+0000 7f1999783740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3412440 0x5569e3418790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.513+0000 7f1999783740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e341b210 0x5569e3418cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.513+0000 7f199493a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f199493a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:49712/0 (socket says 172.21.15.38:49712) 2024-04-22T00:08:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f199493a640 1 -- 172.21.15.38:0/1181765040 learned_addr learned my addr 172.21.15.38:0/1181765040 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.513+0000 7f1999783740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5569e341de40 con 0x5569e3411b20 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5569e3331a10 con 0x5569e341b210 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569e341dfc0 con 0x5569e3412440 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f1999783740 10 client.0 _ll_register_callbacks cb 0x5569e336cb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.514+0000 7f199513b640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e341b210 0x5569e3418cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 0x7f197c0a5350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1980009300 tx=0x7f1980009290 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 -- 172.21.15.38:0/1181765040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3412440 msgr2=0x5569e3418790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3412440 0x5569e3418790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 -- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e341b210 msgr2=0x5569e3418cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e341b210 0x5569e3418cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569e3156620 con 0x5569e3411b20 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.515+0000 7f199493a640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1980015730 tx=0x7f1980004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cc0f21c2bed6d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.517+0000 7f196ffff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:26.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.522+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980022020 con 0x5569e3411b20 2024-04-22T00:08:26.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.522+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1978010c20 con 0x5569e3411b20 2024-04-22T00:08:26.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.522+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f19800146e0 con 0x5569e3411b20 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.528+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f19800148e0 con 0x5569e3411b20 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.528+0000 7f1999783740 10 client.17721 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.528+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5569e342a150 con 0x5569e3411b20 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1978010c20 con 0x5569e3411b20 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1980013440 con 0x5569e3411b20 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 10 client.17721 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 10 client.17721 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:26.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 20 client.17721 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e342ba30 0x5569e342ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 4 client.17721 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.529+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569e33830b0 con 0x5569e342ba30 2024-04-22T00:08:26.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.530+0000 7f198ffff640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e342ba30 0x5569e342ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.537+0000 7f198ffff640 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e342ba30 0x5569e342ded0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8355017e76587c53 server_cookie=cf99108555b67c86 in_seq=0 out_seq=0 2024-04-22T00:08:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.538+0000 7f196ffff640 10 client.17721 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.539+0000 7f196ffff640 1 -- 172.21.15.38:0/1181765040 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569e33830b0 con 0x5569e342ba30 2024-04-22T00:08:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.539+0000 7f196ffff640 10 client.17721 handle_command_reply: tid=0 2024-04-22T00:08:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.542+0000 7f1999783740 1 client.17721 shutdown 2024-04-22T00:08:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.543+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 msgr2=0x5569e3418250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.543+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1980015730 tx=0x7f1980004700 comp rx=0 tx=0).stop 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 shutdown_connections 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e341b210 0x5569e3418cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3412440 0x5569e3418790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e342ba30 0x5569e342ded0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 --2- 172.21.15.38:0/1181765040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3418250 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 >> 172.21.15.38:0/1181765040 conn(0x5569e340c440 msgr2=0x5569e340e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.542 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.546+0000 7f196ffff640 0 client.17721 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.548+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 shutdown_connections 2024-04-22T00:08:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.549+0000 7f1999783740 1 -- 172.21.15.38:0/1181765040 wait complete. 2024-04-22T00:08:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.552+0000 7f1999783740 20 client.17721 trim_cache size 0 max 16384 2024-04-22T00:08:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.591+0000 7f1999783740 1 Processor -- start 2024-04-22T00:08:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.591+0000 7f1999783740 1 -- start start 2024-04-22T00:08:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 0x5569e3426d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 0x5569e3418af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f198ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f198ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33094/0 (socket says 172.21.15.38:33094) 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f198ffff640 1 -- 172.21.15.38:0/324091436 learned_addr learned my addr 172.21.15.38:0/324091436 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f199513b640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 0x5569e3418af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19901bd980 con 0x5569e341b210 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f19901bf4c0 con 0x5569e3412440 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19901be050 con 0x5569e3411b20 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.592+0000 7f199493a640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 0x5569e3426d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 -- 172.21.15.38:0/324091436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 msgr2=0x5569e3426d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 0x5569e3426d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 -- 172.21.15.38:0/324091436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 msgr2=0x5569e3407220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 -- 172.21.15.38:0/324091436 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569e3417220 con 0x5569e341b210 2024-04-22T00:08:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199493a640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 0x5569e3426d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f198ffff640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.593+0000 7f199513b640 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 0x5569e3418af0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f19880eec60 tx=0x7f19880040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e30c8af90cd2808 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.595+0000 7f19867fc640 1 -- 172.21.15.38:0/324091436 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880bdac0 con 0x5569e341b210 2024-04-22T00:08:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.596+0000 7f19867fc640 1 -- 172.21.15.38:0/324091436 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19880c1e70 con 0x5569e341b210 2024-04-22T00:08:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.596+0000 7f19867fc640 1 -- 172.21.15.38:0/324091436 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19880ba130 con 0x5569e341b210 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 msgr2=0x5569e3418af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 0x5569e3418af0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f19880eec60 tx=0x7f19880040d0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 shutdown_connections 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e341b210 0x5569e3418af0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3407220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 --2- 172.21.15.38:0/324091436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e3411b20 0x5569e3426d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.597+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 >> 172.21.15.38:0/324091436 conn(0x5569e340c440 msgr2=0x5569e340e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.599+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 shutdown_connections 2024-04-22T00:08:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.599+0000 7f1999783740 1 -- 172.21.15.38:0/324091436 wait complete. 2024-04-22T00:08:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.600+0000 7f1999783740 1 -- start start 2024-04-22T00:08:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.600+0000 7f1999783740 1 Processor -- start 2024-04-22T00:08:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f1999783740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3423240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f1999783740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f1999783740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e3420210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f1999783740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19901bedf0 con 0x5569e3411b20 2024-04-22T00:08:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f198ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f198ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33102/0 (socket says 172.21.15.38:33102) 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f198ffff640 1 -- 172.21.15.38:0/2428658978 learned_addr learned my addr 172.21.15.38:0/2428658978 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.601+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f19901be720 con 0x5569e3412440 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f199004f500 con 0x5569e341b210 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f1999783740 10 client.0 _ll_register_callbacks cb 0x7f198000ee60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f199513b640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e3420210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 -- 172.21.15.38:0/2428658978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 msgr2=0x5569e3420210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e3420210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 -- 172.21.15.38:0/2428658978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 msgr2=0x5569e3423240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3423240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569e3420950 con 0x5569e3412440 2024-04-22T00:08:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f199513b640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e3420210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f198ffff640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1978009a90 tx=0x7f1978009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee22e3e1b0b60a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.602+0000 7f196ffff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.603+0000 7f196ffff640 1 -- 172.21.15.38:0/2428658978 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197804d060 con 0x5569e3412440 2024-04-22T00:08:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.603+0000 7f196ffff640 1 -- 172.21.15.38:0/2428658978 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f1978048b60 con 0x5569e3412440 2024-04-22T00:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.603+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19880c1e70 con 0x5569e3412440 2024-04-22T00:08:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.603+0000 7f196ffff640 1 -- 172.21.15.38:0/2428658978 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1978018040 con 0x5569e3412440 2024-04-22T00:08:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.610+0000 7f1999783740 10 client.27077 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.610+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5569e3492950 con 0x5569e3412440 2024-04-22T00:08:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f196ffff640 1 -- 172.21.15.38:0/2428658978 <== mon.2 v2:172.21.15.111:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f19780263e0 con 0x5569e3412440 2024-04-22T00:08:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 10 client.27077 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 10 client.27077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 20 client.27077 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:26.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e3492c70 0x5569e3495060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 4 client.27077 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:08:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.611+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f19880c1e70 con 0x5569e3492c70 2024-04-22T00:08:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.614+0000 7f199493a640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e3492c70 0x5569e3495060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.629+0000 7f199493a640 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e3492c70 0x5569e3495060 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dd3ae8fd18682a6 server_cookie=f9a9dad01615825a in_seq=0 out_seq=0 2024-04-22T00:08:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.629+0000 7f196ffff640 10 client.27077 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f196ffff640 1 -- 172.21.15.38:0/2428658978 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f19880c1e70 con 0x5569e3492c70 2024-04-22T00:08:26.696 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f196ffff640 10 client.27077 handle_command_reply: tid=0 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f1999783740 1 client.27077 shutdown 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 msgr2=0x5569e3423780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1978009a90 tx=0x7f1978009f50 comp rx=0 tx=0).stop 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.700+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 shutdown_connections 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569e341b210 0x5569e3420210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.697 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569e3412440 0x5569e3423780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5569e3492c70 0x5569e3495060 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 --2- 172.21.15.38:0/2428658978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5569e3411b20 0x5569e3423240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 >> 172.21.15.38:0/2428658978 conn(0x5569e340c440 msgr2=0x5569e340ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f196ffff640 0 client.27077 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 shutdown_connections 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 1 -- 172.21.15.38:0/2428658978 wait complete. 2024-04-22T00:08:26.698 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.701+0000 7f1999783740 20 client.27077 trim_cache size 0 max 16384 2024-04-22T00:08:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:08:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:08:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:08:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:08:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 msgr2=0x7f197c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 0x7f197c0a5350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1980009300 tx=0x7f1980009290 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 msgr2=0x7f199019e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f1988037040 tx=0x7f1988035d10 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 shutdown_connections 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f197c0a2e90 0x7f197c0a5350 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1990106880 0x7f199019e6d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19901006c0 0x7f19901a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 --2- 172.21.15.38:0/633283299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19900ffc40 0x7f19901a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 >> 172.21.15.38:0/633283299 conn(0x7f1990076bd0 msgr2=0x7f19901021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 shutdown_connections 2024-04-22T00:08:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:26.702+0000 7f1996b9e640 1 -- 172.21.15.38:0/633283299 wait complete. 2024-04-22T00:08:26.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:26 smithi203 ceph-mon[31431]: pgmap v193: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 9.9 MiB/s wr, 2.38k op/s 2024-04-22T00:08:26.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:26 smithi111 ceph-mon[31543]: pgmap v193: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 9.9 MiB/s wr, 2.38k op/s 2024-04-22T00:08:26.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:26 smithi038 ceph-mon[27972]: pgmap v193: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 9.9 MiB/s wr, 2.38k op/s 2024-04-22T00:08:27.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:27 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:08:27.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:27 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:08:27.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:27 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:08:28.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:28 smithi203 ceph-mon[31431]: pgmap v194: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:08:28.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:28 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:08:28.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:28 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:08:28.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:28 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:08:28.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:28 smithi111 ceph-mon[31543]: pgmap v194: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:08:28.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:28 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:08:28.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:28 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:08:28.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:28 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:08:28.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:28 smithi038 ceph-mon[27972]: pgmap v194: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 2.65k op/s 2024-04-22T00:08:28.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:28 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:08:28.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:28 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:08:28.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:28 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:08:30.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:30 smithi203 ceph-mon[31431]: pgmap v195: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 2.46k op/s 2024-04-22T00:08:30.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:30 smithi111 ceph-mon[31543]: pgmap v195: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 2.46k op/s 2024-04-22T00:08:30.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:30 smithi038 ceph-mon[27972]: pgmap v195: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 2.46k op/s 2024-04-22T00:08:31.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:31 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:31.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:31 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:31.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:31 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:32.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:32 smithi203 ceph-mon[31431]: pgmap v196: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 9.4 MiB/s wr, 2.29k op/s 2024-04-22T00:08:32.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:32 smithi111 ceph-mon[31543]: pgmap v196: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 9.4 MiB/s wr, 2.29k op/s 2024-04-22T00:08:32.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:32 smithi038 ceph-mon[27972]: pgmap v196: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 9.4 MiB/s wr, 2.29k op/s 2024-04-22T00:08:34.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:34 smithi111 ceph-mon[31543]: pgmap v197: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 3.15k op/s 2024-04-22T00:08:34.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:34 smithi038 ceph-mon[27972]: pgmap v197: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 3.15k op/s 2024-04-22T00:08:35.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:34 smithi203 ceph-mon[31431]: pgmap v197: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 3.15k op/s 2024-04-22T00:08:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:36 smithi111 ceph-mon[31543]: pgmap v198: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:08:36.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:36 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:08:36.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:36 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:08:36.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:36 smithi038 ceph-mon[27972]: pgmap v198: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:08:36.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:36 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:08:36.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:36 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:08:37.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:36 smithi203 ceph-mon[31431]: pgmap v198: 129 pgs: 129 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.5 MiB/s wr, 2.33k op/s 2024-04-22T00:08:37.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:36 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:08:37.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:36 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:08:37.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:37 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:08:37.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:37 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:08:37.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:37 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:08:37.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:37 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:08:37.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:37 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:08:37.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:37 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:08:38.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:37 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:08:38.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:37 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:08:38.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:37 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:08:38.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:08:38.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:08:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.516+0000 7fe0ba15b640 1 -- 172.21.15.38:0/2124930856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b4108d10 msgr2=0x7fe0b4109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.516+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/2124930856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b4108d10 0x7fe0b4109190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009890 tx=0x7fe0a802bb70 comp rx=0 tx=0).stop 2024-04-22T00:08:38.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.517+0000 7fe0ba15b640 1 -- 172.21.15.38:0/2124930856 shutdown_connections 2024-04-22T00:08:38.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.517+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/2124930856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b411d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.517+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/2124930856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b4108d10 0x7fe0b4109190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.517+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/2124930856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b410bf00 0x7fe0b4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.517+0000 7fe0ba15b640 1 -- 172.21.15.38:0/2124930856 >> 172.21.15.38:0/2124930856 conn(0x7fe0b406c850 msgr2=0x7fe0b406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.518+0000 7fe0ba15b640 1 -- 172.21.15.38:0/2124930856 shutdown_connections 2024-04-22T00:08:38.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.519+0000 7fe0ba15b640 1 -- 172.21.15.38:0/2124930856 wait complete. 2024-04-22T00:08:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.519+0000 7fe0ba15b640 1 Processor -- start 2024-04-22T00:08:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.519+0000 7fe0ba15b640 1 -- start start 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 0x7fe0b41a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b410bf00 0x7fe0b41a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b41ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b4120890 con 0x7fe0b411af30 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b4120710 con 0x7fe0b4108d10 2024-04-22T00:08:38.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.520+0000 7fe0ba15b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b4120a10 con 0x7fe0b410bf00 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 0x7fe0b41a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 0x7fe0b41a6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51468/0 (socket says 172.21.15.38:51468) 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b37fe640 1 -- 172.21.15.38:0/3770912408 learned_addr learned my addr 172.21.15.38:0/3770912408 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b41ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 -- 172.21.15.38:0/3770912408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b410bf00 msgr2=0x7fe0b41a7390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:08:38.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b410bf00 0x7fe0b41a7390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 -- 172.21.15.38:0/3770912408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 msgr2=0x7fe0b41a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 0x7fe0b41a6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.521+0000 7fe0b3fff640 1 -- 172.21.15.38:0/3770912408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b41abe40 con 0x7fe0b411af30 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.522+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b41ab710 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400d680 tx=0x7fe0a400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.522+0000 7fe0b0ff9640 1 -- 172.21.15.38:0/3770912408 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4017070 con 0x7fe0b411af30 2024-04-22T00:08:38.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.523+0000 7fe0b0ff9640 1 -- 172.21.15.38:0/3770912408 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a4009820 con 0x7fe0b411af30 2024-04-22T00:08:38.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.523+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b41ac130 con 0x7fe0b411af30 2024-04-22T00:08:38.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.523+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b41b39c0 con 0x7fe0b411af30 2024-04-22T00:08:38.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.524+0000 7fe0b0ff9640 1 -- 172.21.15.38:0/3770912408 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4004e00 con 0x7fe0b411af30 2024-04-22T00:08:38.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.526+0000 7fe0bce27740 1 Processor -- start 2024-04-22T00:08:38.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0b0ff9640 1 -- 172.21.15.38:0/3770912408 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe0a4010460 con 0x7fe0b411af30 2024-04-22T00:08:38.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 -- start start 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c7850 0x5583684cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684d0430 0x5583684d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55836842b560 con 0x5583684c6f30 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5583683d3730 con 0x5583684d0430 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.527+0000 7fe0bce27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558368427520 con 0x5583684c7850 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50284/0 (socket says 172.21.15.38:50284) 2024-04-22T00:08:38.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/2536433374 learned_addr learned my addr 172.21.15.38:0/2536433374 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b0ff9640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 0x7fe09c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c7850 0x5583684cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 0x7fe09c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b0ff9640 1 -- 172.21.15.38:0/3770912408 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0a40cede0 con 0x7fe0b411af30 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/2536433374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c7850 msgr2=0x5583684cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c7850 0x5583684cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/2536433374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684d0430 msgr2=0x5583684d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684d0430 0x5583684d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/2536433374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583682097d0 con 0x5583684c6f30 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005940 tx=0x7fe0a8034b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ea0e7d7d13204d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe0b3fff640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 0x7fe09c0a5310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cc2a0 tx=0x7fe0a40bb040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe099ffb640 1 -- 172.21.15.38:0/2536433374 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8044070 con 0x5583684c6f30 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe099ffb640 1 -- 172.21.15.38:0/2536433374 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8004040 con 0x5583684c6f30 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.528+0000 7fe099ffb640 1 -- 172.21.15.38:0/2536433374 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8039070 con 0x5583684c6f30 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 -- 172.21.15.38:0/2536433374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 msgr2=0x5583684c7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005940 tx=0x7fe0a8034b20 comp rx=0 tx=0).stop 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 -- 172.21.15.38:0/2536433374 shutdown_connections 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684d0430 0x5583684d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c7850 0x5583684cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 --2- 172.21.15.38:0/2536433374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684c7310 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 -- 172.21.15.38:0/2536433374 >> 172.21.15.38:0/2536433374 conn(0x5583684c1660 msgr2=0x5583684c1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 -- 172.21.15.38:0/2536433374 shutdown_connections 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.529+0000 7fe0bce27740 1 -- 172.21.15.38:0/2536433374 wait complete. 2024-04-22T00:08:38.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 -- start start 2024-04-22T00:08:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 Processor -- start 2024-04-22T00:08:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c6f30 0x5583684cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5583684d31c0 con 0x5583684d0430 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5583683e6e00 con 0x5583684c7850 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5583684d3340 con 0x5583684c6f30 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0bce27740 10 client.0 _ll_register_callbacks cb 0x558368421eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50288/0 (socket says 172.21.15.38:50288) 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/973723603 learned_addr learned my addr 172.21.15.38:0/973723603 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c6f30 0x5583684cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.530+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/973723603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c6f30 msgr2=0x5583684cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c6f30 0x5583684cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 msgr2=0x5583684cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/973723603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55836813f3f0 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8004620 tx=0x7fe0a8034c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f948179bf305451 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8044070 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0a8034850 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a8004040 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe0a8056050 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 10 client.17736 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5583680ff0f0 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8004040 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe0a802c6d0 con 0x5583684d0430 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 10 client.17736 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.531+0000 7fe0bce27740 20 client.17736 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5583684e0720 0x5583684e2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe0bce27740 4 client.17736 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558368438aa0 con 0x5583684e0720 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5583684e0720 0x5583684e2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5583684e0720 0x5583684e2b10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35eb6ba70e244de1 server_cookie=fb46170e0012a7ed in_seq=0 out_seq=0 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.532+0000 7fe092ffd640 10 client.17736 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe092ffd640 1 -- 172.21.15.38:0/973723603 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558368438aa0 con 0x5583684e0720 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe092ffd640 10 client.17736 handle_command_reply: tid=0 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 client.17736 shutdown 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 msgr2=0x5583684cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8004620 tx=0x7fe0a8034c10 comp rx=0 tx=0).stop 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 shutdown_connections 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5583684e0720 0x5583684e2b10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684d0430 0x5583684cdf40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 --2- 172.21.15.38:0/973723603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684c6f30 0x5583684cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 >> 172.21.15.38:0/973723603 conn(0x5583684c1660 msgr2=0x5583684c33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe092ffd640 0 client.17736 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.533+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 shutdown_connections 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.534+0000 7fe0bce27740 1 -- 172.21.15.38:0/973723603 wait complete. 2024-04-22T00:08:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.534+0000 7fe0bce27740 20 client.17736 trim_cache size 0 max 16384 2024-04-22T00:08:38.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.554+0000 7fe0bce27740 1 Processor -- start 2024-04-22T00:08:38.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.554+0000 7fe0bce27740 1 -- start start 2024-04-22T00:08:38.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684c0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a000fda0 con 0x5583684c6f30 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a00118e0 con 0x5583684c7850 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0bce27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0010470 con 0x5583684d0430 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50302/0 (socket says 172.21.15.38:50302) 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3139760853 learned_addr learned my addr 172.21.15.38:0/3139760853 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3139760853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 msgr2=0x5583684e5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3139760853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 msgr2=0x5583684c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684c0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3139760853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583684cba10 con 0x5583684c6f30 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802cad0 tx=0x7fe0a8005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616becb0e35268b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.555+0000 7fe0997fa640 1 -- 172.21.15.38:0/3139760853 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8044030 con 0x5583684c6f30 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0997fa640 1 -- 172.21.15.38:0/3139760853 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8045040 con 0x5583684c6f30 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0997fa640 1 -- 172.21.15.38:0/3139760853 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8039070 con 0x5583684c6f30 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 -- 172.21.15.38:0/3139760853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 msgr2=0x5583684cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe0a802cad0 tx=0x7fe0a8005ab0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 -- 172.21.15.38:0/3139760853 shutdown_connections 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3139760853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684cfbf0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.556+0000 7fe0bce27740 1 -- 172.21.15.38:0/3139760853 >> 172.21.15.38:0/3139760853 conn(0x5583684c1660 msgr2=0x5583684bfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- 172.21.15.38:0/3139760853 shutdown_connections 2024-04-22T00:08:38.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- 172.21.15.38:0/3139760853 wait complete. 2024-04-22T00:08:38.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- start start 2024-04-22T00:08:38.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 Processor -- start 2024-04-22T00:08:38.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0011210 con 0x5583684c6f30 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0010b40 con 0x5583684c7850 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a000dd80 con 0x5583684d0430 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0bce27740 10 client.0 _ll_register_callbacks cb 0x7fe0a8042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0b2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50304/0 (socket says 172.21.15.38:50304) 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3033933598 learned_addr learned my addr 172.21.15.38:0/3033933598 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.557+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3033933598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 msgr2=0x5583684e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 msgr2=0x5583684e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 -- 172.21.15.38:0/3033933598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583684e3b80 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0b2ffd640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005df0 tx=0x7fe0a8034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9682d2ae73de165 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8039070 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fe0a802c510 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a8045040 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe0a8044040 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0bce27740 10 client.17742 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5583685468e0 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8045040 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.558+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe0a8004eb0 con 0x5583684c6f30 2024-04-22T00:08:38.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 10 client.17742 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 20 client.17742 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558368546d90 0x5583685491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 4 client.17742 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558368432df0 con 0x558368546d90 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558368546d90 0x5583685491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0b37fe640 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558368546d90 0x5583685491b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da79a21d6521659e server_cookie=62b84c59d8598be0 in_seq=0 out_seq=0 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe092ffd640 10 client.17742 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe092ffd640 1 -- 172.21.15.38:0/3033933598 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558368432df0 con 0x558368546d90 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe092ffd640 10 client.17742 handle_command_reply: tid=0 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 client.17742 shutdown 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 msgr2=0x5583684e5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005df0 tx=0x7fe0a8034650 comp rx=0 tx=0).stop 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 shutdown_connections 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5583684d0430 0x5583684e3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5583684c7850 0x5583684e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558368546d90 0x5583685491b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.559+0000 7fe0bce27740 1 --2- 172.21.15.38:0/3033933598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5583684c6f30 0x5583684e5a80 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe092ffd640 0 client.17742 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 >> 172.21.15.38:0/3033933598 conn(0x5583684c1660 msgr2=0x5583684dbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 shutdown_connections 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0bce27740 1 -- 172.21.15.38:0/3033933598 wait complete. 2024-04-22T00:08:38.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0bce27740 20 client.17742 trim_cache size 0 max 16384 2024-04-22T00:08:38.563 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:08:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:08:38.563 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 msgr2=0x7fe09c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 0x7fe09c0a5310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cc2a0 tx=0x7fe0a40bb040 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 msgr2=0x7fe0b41ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b41ab710 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400d680 tx=0x7fe0a400db40 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 shutdown_connections 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe09c0a2e50 0x7fe09c0a5310 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0b411af30 0x7fe0b41ab710 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0b410bf00 0x7fe0b41a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 --2- 172.21.15.38:0/3770912408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0b4108d10 0x7fe0b41a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 >> 172.21.15.38:0/3770912408 conn(0x7fe0b406c850 msgr2=0x7fe0b4109940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 shutdown_connections 2024-04-22T00:08:38.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.560+0000 7fe0ba15b640 1 -- 172.21.15.38:0/3770912408 wait complete. 2024-04-22T00:08:38.566 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:08:38.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:08:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 -- 172.21.15.38:0/4212290376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c10e6a0 msgr2=0x7f153c11cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 --2- 172.21.15.38:0/4212290376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c10e6a0 0x7f153c11cf90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1534009ca0 tx=0x7f153402c140 comp rx=0 tx=0).stop 2024-04-22T00:08:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 -- 172.21.15.38:0/4212290376 shutdown_connections 2024-04-22T00:08:38.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 --2- 172.21.15.38:0/4212290376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c10e6a0 0x7f153c11cf90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 --2- 172.21.15.38:0/4212290376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c110310 0x7f153c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 --2- 172.21.15.38:0/4212290376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c10fea0 0x7f153c10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.711+0000 7f1541312640 1 -- 172.21.15.38:0/4212290376 >> 172.21.15.38:0/4212290376 conn(0x7f153c06c860 msgr2=0x7f153c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:38.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:38 smithi038 ceph-mon[27972]: pgmap v199: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:38.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.715+0000 7f1541312640 1 -- 172.21.15.38:0/4212290376 shutdown_connections 2024-04-22T00:08:38.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.715+0000 7f1541312640 1 -- 172.21.15.38:0/4212290376 wait complete. 2024-04-22T00:08:38.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.716+0000 7f1541312640 1 Processor -- start 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 -- start start 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c10fea0 0x7f153c11c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c115a90 0x7f153c115f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f153c1209a0 con 0x7f153c11ca10 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f153c120820 con 0x7f153c115a90 2024-04-22T00:08:38.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.720+0000 7f1541312640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f153c120b20 con 0x7f153c10fea0 2024-04-22T00:08:38.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.722+0000 7f153a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.722+0000 7f153a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50328/0 (socket says 172.21.15.38:50328) 2024-04-22T00:08:38.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.722+0000 7f153a7fc640 1 -- 172.21.15.38:0/3011712297 learned_addr learned my addr 172.21.15.38:0/3011712297 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.724+0000 7f153a7fc640 1 -- 172.21.15.38:0/3011712297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c10fea0 msgr2=0x7f153c11c4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.724+0000 7f153a7fc640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c10fea0 0x7f153c11c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.724+0000 7f153a7fc640 1 -- 172.21.15.38:0/3011712297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c115a90 msgr2=0x7f153c115f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.724+0000 7f153a7fc640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c115a90 0x7f153c115f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.724+0000 7f153a7fc640 1 -- 172.21.15.38:0/3011712297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153c11a1e0 con 0x7f153c11ca10 2024-04-22T00:08:38.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.725+0000 7f153a7fc640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f153000b450 tx=0x7f153000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.726+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530015020 con 0x7f153c11ca10 2024-04-22T00:08:38.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.726+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153c11a4d0 con 0x7f153c11ca10 2024-04-22T00:08:38.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.726+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153c11aa20 con 0x7f153c11ca10 2024-04-22T00:08:38.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.728+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1530011d70 con 0x7f153c11ca10 2024-04-22T00:08:38.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.728+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153000f340 con 0x7f153c11ca10 2024-04-22T00:08:38.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.730+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153c06daf0 con 0x7f153c11ca10 2024-04-22T00:08:38.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.733+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1530018020 con 0x7f153c11ca10 2024-04-22T00:08:38.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.734+0000 7f1523fff640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 0x7f15240a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:38.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.734+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f15300cf1a0 con 0x7f153c11ca10 2024-04-22T00:08:38.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.737+0000 7f153affd640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 0x7f15240a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:38.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.740+0000 7f153affd640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 0x7f15240a53d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f152c009a60 tx=0x7f152c005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:38.744+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f153001f020 con 0x7f153c11ca10 2024-04-22T00:08:38.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:38 smithi111 ceph-mon[31543]: pgmap v199: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:38.933 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:38 smithi203 ceph-mon[31431]: pgmap v199: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 11 MiB/s wr, 2.73k op/s 2024-04-22T00:08:39.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.067+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f153c116a80 con 0x7f153c11ca10 2024-04-22T00:08:39.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.068+0000 7f1523fff640 1 -- 172.21.15.38:0/3011712297 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7f153000d040 con 0x7f153c11ca10 2024-04-22T00:08:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:08:39.064 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:08:39.064 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:08:39.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 msgr2=0x7f15240a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 0x7f15240a53d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f152c009a60 tx=0x7f152c005f90 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 msgr2=0x7f153c115550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f153000b450 tx=0x7f153000b910 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 shutdown_connections 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f15240a2f10 0x7f15240a53d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f153c115a90 0x7f153c115f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f153c11ca10 0x7f153c115550 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 --2- 172.21.15.38:0/3011712297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f153c10fea0 0x7f153c11c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 >> 172.21.15.38:0/3011712297 conn(0x7f153c06c860 msgr2=0x7f153c070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 shutdown_connections 2024-04-22T00:08:39.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.070+0000 7f1541312640 1 -- 172.21.15.38:0/3011712297 wait complete. 2024-04-22T00:08:39.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:08:39.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- 172.21.15.38:0/2454663763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc107720 msgr2=0x7fcccc107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 --2- 172.21.15.38:0/2454663763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc107720 0x7fcccc107b00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009930 tx=0x7fccbc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:39.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- 172.21.15.38:0/2454663763 shutdown_connections 2024-04-22T00:08:39.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 --2- 172.21.15.38:0/2454663763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc1020d0 0x7fcccc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 --2- 172.21.15.38:0/2454663763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 0x7fcccc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 --2- 172.21.15.38:0/2454663763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc107720 0x7fcccc107b00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- 172.21.15.38:0/2454663763 >> 172.21.15.38:0/2454663763 conn(0x7fcccc0fd180 msgr2=0x7fcccc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- 172.21.15.38:0/2454663763 shutdown_connections 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- 172.21.15.38:0/2454663763 wait complete. 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 Processor -- start 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.191+0000 7fccd18e6640 1 -- start start 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 0x7fcccc198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc107720 0x7fcccc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc111b30 con 0x7fcccc1020d0 2024-04-22T00:08:39.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1119b0 con 0x7fcccc101710 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc111cb0 con 0x7fcccc107720 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 0x7fcccc198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50350/0 (socket says 172.21.15.38:50350) 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc107720 0x7fcccc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 -- 172.21.15.38:0/403743793 learned_addr learned my addr 172.21.15.38:0/403743793 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 -- 172.21.15.38:0/403743793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc107720 msgr2=0x7fcccc19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc107720 0x7fcccc19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 -- 172.21.15.38:0/403743793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 msgr2=0x7fcccc198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 0x7fcccc198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 -- 172.21.15.38:0/403743793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc19d0b0 con 0x7fcccc1020d0 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccc3fff640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fccb400c890 tx=0x7fccb400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd08e4640 1 -- 172.21.15.38:0/403743793 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4004220 con 0x7fcccc1020d0 2024-04-22T00:08:39.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd08e4640 1 -- 172.21.15.38:0/403743793 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb40043c0 con 0x7fcccc1020d0 2024-04-22T00:08:39.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd08e4640 1 -- 172.21.15.38:0/403743793 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb40049e0 con 0x7fcccc1020d0 2024-04-22T00:08:39.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc19d3a0 con 0x7fcccc1020d0 2024-04-22T00:08:39.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.192+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc1a4c30 con 0x7fcccc1020d0 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.195+0000 7fccd08e4640 1 -- 172.21.15.38:0/403743793 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fccb40079b0 con 0x7fcccc1020d0 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.195+0000 7fccd45af740 1 Processor -- start 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd08e4640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 0x7fccb80a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 -- start start 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd08e4640 1 -- 172.21.15.38:0/403743793 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fccb40cf720 con 0x7fcccc1020d0 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 0x7fccb80a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x55818897b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55818897b960 0x55818897bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5581888d3530 con 0x558188969410 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55818887b830 con 0x558188969dc0 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccd45af740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5581888cf4f0 con 0x55818897b960 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50354/0 (socket says 172.21.15.38:50354) 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 -- 172.21.15.38:0/2390818410 learned_addr learned my addr 172.21.15.38:0/2390818410 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55818897b960 0x55818897bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x55818897b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 -- 172.21.15.38:0/2390818410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55818897b960 msgr2=0x55818897bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55818897b960 0x55818897bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 -- 172.21.15.38:0/2390818410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 msgr2=0x55818897b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x55818897b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccaffd640 1 -- 172.21.15.38:0/2390818410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55818889f2c0 con 0x558188969410 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x55818897b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.196+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 0x7fccb80a52c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009f70 tx=0x7fccc4009650 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.197+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009900 tx=0x7fccbc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2fd5c22037b6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.197+0000 7fccc1ffb640 1 -- 172.21.15.38:0/2390818410 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc042070 con 0x558188969410 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.197+0000 7fccc1ffb640 1 -- 172.21.15.38:0/2390818410 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccbc02cbc0 con 0x558188969410 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.197+0000 7fccc1ffb640 1 -- 172.21.15.38:0/2390818410 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0051e0 con 0x558188969410 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 -- 172.21.15.38:0/2390818410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 msgr2=0x5581889697f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009900 tx=0x7fccbc02c650 comp rx=0 tx=0).stop 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 -- 172.21.15.38:0/2390818410 shutdown_connections 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55818897b960 0x55818897bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x55818897b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 --2- 172.21.15.38:0/2390818410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558188969410 0x5581889697f0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 -- 172.21.15.38:0/2390818410 >> 172.21.15.38:0/2390818410 conn(0x55818896bd40 msgr2=0x55818896c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 -- 172.21.15.38:0/2390818410 shutdown_connections 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.198+0000 7fccd45af740 1 -- 172.21.15.38:0/2390818410 wait complete. 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 -- start start 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 Processor -- start 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 0x5581889802a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x5581889807e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558188983ad0 con 0x55818897b960 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55818888ef10 con 0x558188969410 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558188983c50 con 0x558188969dc0 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccd45af740 10 client.0 _ll_register_callbacks cb 0x5581885e6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 0x5581889802a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51556/0 (socket says 172.21.15.38:51556) 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fcccaffd640 1 -- 172.21.15.38:0/3096271620 learned_addr learned my addr 172.21.15.38:0/3096271620 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x5581889807e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.199+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 0x5581889802a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34544/0 (socket says 172.21.15.38:34544) 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 -- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 msgr2=0x5581889802a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 0x5581889802a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 -- 172.21.15.38:0/3096271620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 msgr2=0x5581889807e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x5581889807e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 -- 172.21.15.38:0/3096271620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558188860c80 con 0x558188969410 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fcccaffd640 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004290 tx=0x7fccbc005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5632bd13e16d42cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:39.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc042070 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fccbc004f80 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccbc02cbc0 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fccbc056050 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccd45af740 10 client.27101 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.200+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5581885e73f0 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccbc02cbc0 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fccbc003c90 con 0x558188969410 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 10 client.27101 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 10 client.27101 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 20 client.27101 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558188990ae0 0x558188992ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 4 client.27101 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581888e0a70 con 0x558188990ae0 2024-04-22T00:08:39.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.201+0000 7fccc3fff640 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558188990ae0 0x558188992ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.202+0000 7fccaaffd640 10 client.27101 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.202+0000 7fccc3fff640 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558188990ae0 0x558188992ed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac8bce546b01a01b server_cookie=5a8c29226591eef3 in_seq=0 out_seq=0 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccaaffd640 1 -- 172.21.15.38:0/3096271620 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581888e0a70 con 0x558188990ae0 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccaaffd640 10 client.27101 handle_command_reply: tid=0 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 client.27101 shutdown 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 msgr2=0x558188982f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004290 tx=0x7fccbc005c00 comp rx=0 tx=0).stop 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 shutdown_connections 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558188990ae0 0x558188992ed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x5581889807e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969dc0 0x5581889802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccaaffd640 0 client.27101 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 --2- 172.21.15.38:0/3096271620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969410 0x558188982f10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 >> 172.21.15.38:0/3096271620 conn(0x55818896bd40 msgr2=0x55818896b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 shutdown_connections 2024-04-22T00:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 1 -- 172.21.15.38:0/3096271620 wait complete. 2024-04-22T00:08:39.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.203+0000 7fccd45af740 20 client.27101 trim_cache size 0 max 16384 2024-04-22T00:08:39.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.221+0000 7fccd45af740 1 Processor -- start 2024-04-22T00:08:39.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.221+0000 7fccd45af740 1 -- start start 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.221+0000 7fccd45af740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x55818898c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.221+0000 7fccd45af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x5581889829d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccd45af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccd45af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1ac830 con 0x55818897b960 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccd45af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1ae370 con 0x558188969dc0 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccd45af740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1acf00 con 0x558188969410 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50374/0 (socket says 172.21.15.38:50374) 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 -- 172.21.15.38:0/3268957258 learned_addr learned my addr 172.21.15.38:0/3268957258 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccaffd640 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x55818898c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccc3fff640 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x5581889829d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 -- 172.21.15.38:0/3268957258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 msgr2=0x55818898c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x55818898c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 -- 172.21.15.38:0/3268957258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 msgr2=0x5581889829d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x5581889829d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 -- 172.21.15.38:0/3268957258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581886b17d0 con 0x55818897b960 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fccc4018840 tx=0x7fccc4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb371150cbca41a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.222+0000 7fccc17fa640 1 -- 172.21.15.38:0/3268957258 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017d20 con 0x55818897b960 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccc17fa640 1 -- 172.21.15.38:0/3268957258 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc40228d0 con 0x55818897b960 2024-04-22T00:08:39.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccc17fa640 1 -- 172.21.15.38:0/3268957258 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc40217b0 con 0x55818897b960 2024-04-22T00:08:39.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 -- 172.21.15.38:0/3268957258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 msgr2=0x558188982f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fccc4018840 tx=0x7fccc4019000 comp rx=0 tx=0).stop 2024-04-22T00:08:39.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 -- 172.21.15.38:0/3268957258 shutdown_connections 2024-04-22T00:08:39.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188982f10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x5581889829d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 --2- 172.21.15.38:0/3268957258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x55818898c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 -- 172.21.15.38:0/3268957258 >> 172.21.15.38:0/3268957258 conn(0x55818896bd40 msgr2=0x558188968850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 -- 172.21.15.38:0/3268957258 shutdown_connections 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.223+0000 7fccd45af740 1 -- 172.21.15.38:0/3268957258 wait complete. 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 -- start start 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 Processor -- start 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x558188989430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188983ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1adca0 con 0x55818897b960 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1ad5d0 con 0x558188969dc0 2024-04-22T00:08:39.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1aa810 con 0x558188969410 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccd45af740 10 client.0 _ll_register_callbacks cb 0x7fccbc0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fcccaffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34572/0 (socket says 172.21.15.38:34572) 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fcccaffd640 1 -- 172.21.15.38:0/2422447000 learned_addr learned my addr 172.21.15.38:0/2422447000 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x558188989430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.224+0000 7fcccb7fe640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188983ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 -- 172.21.15.38:0/2422447000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 msgr2=0x558188989430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x558188989430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 -- 172.21.15.38:0/2422447000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 msgr2=0x558188983ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188983ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 -- 172.21.15.38:0/2422447000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558188984460 con 0x558188969410 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x558188989430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fcccaffd640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004c00 tx=0x7fccbc0345b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a6e61f0600826d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc042070 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccc40228d0 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fccbc034a30 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fccbc063050 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccd45af740 10 client.26959 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5581889f7590 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.225+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc40228d0 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fccbc0050d0 con 0x558188969410 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 10 client.26959 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 10 client.26959 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 20 client.26959 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5581889f7c80 0x5581889fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 4 client.26959 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581888dadc0 con 0x5581889f7c80 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5581889f7c80 0x5581889fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccaaffd640 10 client.26959 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.226+0000 7fccc3fff640 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5581889f7c80 0x5581889fa0a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=23ce0b43c9537409 server_cookie=eed01d55872f0250 in_seq=0 out_seq=0 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccaaffd640 1 -- 172.21.15.38:0/2422447000 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581888dadc0 con 0x5581889f7c80 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccaaffd640 10 client.26959 handle_command_reply: tid=0 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 client.26959 shutdown 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 msgr2=0x558188988ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004c00 tx=0x7fccbc0345b0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 shutdown_connections 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x5581889f7c80 0x5581889fa0a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55818897b960 0x558188983ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558188969dc0 0x558188989430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccaaffd640 0 client.26959 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:08:39.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 --2- 172.21.15.38:0/2422447000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558188969410 0x558188988ef0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 >> 172.21.15.38:0/2422447000 conn(0x55818896bd40 msgr2=0x55818897a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 shutdown_connections 2024-04-22T00:08:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 1 -- 172.21.15.38:0/2422447000 wait complete. 2024-04-22T00:08:39.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.227+0000 7fccd45af740 20 client.26959 trim_cache size 0 max 16384 2024-04-22T00:08:39.229 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 msgr2=0x7fccb80a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 0x7fccb80a52c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009f70 tx=0x7fccc4009650 comp rx=0 tx=0).stop 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 msgr2=0x7fcccc198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fccb400c890 tx=0x7fccb400cd50 comp rx=0 tx=0).stop 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 shutdown_connections 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fccb80a2e00 0x7fccb80a52c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcccc107720 0x7fcccc19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcccc1020d0 0x7fcccc198650 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 --2- 172.21.15.38:0/403743793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcccc101710 0x7fcccc198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 >> 172.21.15.38:0/403743793 conn(0x7fcccc0fd180 msgr2=0x7fcccc0fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 shutdown_connections 2024-04-22T00:08:39.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.228+0000 7fccd18e6640 1 -- 172.21.15.38:0/403743793 wait complete. 2024-04-22T00:08:39.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:08:39.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 -- 172.21.15.38:0/2407674747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500107730 msgr2=0x7f7500107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 --2- 172.21.15.38:0/2407674747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500107730 0x7f7500107b10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009930 tx=0x7f74f002bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:39.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 -- 172.21.15.38:0/2407674747 shutdown_connections 2024-04-22T00:08:39.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 --2- 172.21.15.38:0/2407674747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 0x7f750010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 --2- 172.21.15.38:0/2407674747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7500069dc0 0x7f750006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 --2- 172.21.15.38:0/2407674747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500107730 0x7f7500107b10 secure :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009930 tx=0x7f74f002bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:39.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 -- 172.21.15.38:0/2407674747 >> 172.21.15.38:0/2407674747 conn(0x7f75000776a0 msgr2=0x7f7500077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.367+0000 7f7504965640 1 -- 172.21.15.38:0/2407674747 shutdown_connections 2024-04-22T00:08:39.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 -- 172.21.15.38:0/2407674747 wait complete. 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 Processor -- start 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 -- start start 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 0x7f75001987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 0x7f750019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7500111b90 con 0x7f7500069dc0 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7500111a10 con 0x7f750006a780 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.368+0000 7f7504965640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7500111d10 con 0x7f750019cb20 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50400/0 (socket says 172.21.15.38:50400) 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 -- 172.21.15.38:0/1366060670 learned_addr learned my addr 172.21.15.38:0/1366060670 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 0x7f750019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74f7fff640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 0x7f75001987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 -- 172.21.15.38:0/1366060670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 msgr2=0x7f750019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 0x7f750019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 -- 172.21.15.38:0/1366060670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 msgr2=0x7f75001987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 0x7f75001987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 -- 172.21.15.38:0/1366060670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750019d610 con 0x7f7500069dc0 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 0x7f750019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:39.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009900 tx=0x7f74f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74f77fe640 1 -- 172.21.15.38:0/1366060670 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0043020 con 0x7f7500069dc0 2024-04-22T00:08:39.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74f77fe640 1 -- 172.21.15.38:0/1366060670 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f00054e0 con 0x7f7500069dc0 2024-04-22T00:08:39.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750019d8a0 con 0x7f7500069dc0 2024-04-22T00:08:39.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f74f77fe640 1 -- 172.21.15.38:0/1366060670 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f00039f0 con 0x7f7500069dc0 2024-04-22T00:08:39.368 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.369+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75001a5070 con 0x7f7500069dc0 2024-04-22T00:08:39.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74f77fe640 1 -- 172.21.15.38:0/1366060670 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f74f0046020 con 0x7f7500069dc0 2024-04-22T00:08:39.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 Processor -- start 2024-04-22T00:08:39.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 -- start start 2024-04-22T00:08:39.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74f77fe640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 0x7f74ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc330fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc333910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc28c530 con 0x55d5dc328020 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc234830 con 0x55d5dc331520 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f750762c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc2884f0 con 0x55d5dc328940 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74f77fe640 1 -- 172.21.15.38:0/1366060670 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f74f00f1400 con 0x7f7500069dc0 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 0x7f74ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.372+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50416/0 (socket says 172.21.15.38:50416) 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc330fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 -- 172.21.15.38:0/1221074203 learned_addr learned my addr 172.21.15.38:0/1221074203 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc333910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 -- 172.21.15.38:0/1221074203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 msgr2=0x55d5dc330fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc330fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 -- 172.21.15.38:0/1221074203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 msgr2=0x55d5dc333910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc333910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 -- 172.21.15.38:0/1221074203 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5dc2582c0 con 0x55d5dc328020 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f74f00bd720 tx=0x7f74f00b9af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbe23489108e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 0x7f74ec0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005fd0 tx=0x7f74f8012b30 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74f4ff9640 1 -- 172.21.15.38:0/1221074203 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f002fb60 con 0x55d5dc328020 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74f4ff9640 1 -- 172.21.15.38:0/1221074203 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0005690 con 0x55d5dc328020 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.373+0000 7f74f4ff9640 1 -- 172.21.15.38:0/1221074203 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0037a00 con 0x55d5dc328020 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- 172.21.15.38:0/1221074203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 msgr2=0x55d5dc328400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f74f00bd720 tx=0x7f74f00b9af0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- 172.21.15.38:0/1221074203 shutdown_connections 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc333910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc330fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- 172.21.15.38:0/1221074203 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc328400 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- 172.21.15.38:0/1221074203 >> 172.21.15.38:0/1221074203 conn(0x55d5dc322750 msgr2=0x55d5dc322b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- 172.21.15.38:0/1221074203 shutdown_connections 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- 172.21.15.38:0/1221074203 wait complete. 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- start start 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 Processor -- start 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc330f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc32e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc28c530 con 0x55d5dc328940 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc247f10 con 0x55d5dc331520 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.374+0000 7f750762c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d5dc3340d0 con 0x55d5dc328020 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f750762c740 10 client.0 _ll_register_callbacks cb 0x55d5dbf9fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50432/0 (socket says 172.21.15.38:50432) 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 -- 172.21.15.38:0/151714923 learned_addr learned my addr 172.21.15.38:0/151714923 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74fed9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc32e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74fe59c640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc330f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 -- 172.21.15.38:0/151714923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 msgr2=0x55d5dc330f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc330f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 -- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 msgr2=0x55d5dc32e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc32e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 -- 172.21.15.38:0/151714923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5dc219c80 con 0x55d5dc328940 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74f7fff640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f74e8012700 tx=0x7f74e8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9899597c9557591 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8004260 con 0x55d5dc328940 2024-04-22T00:08:39.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f74e8004400 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d5dc299a70 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f74e80119a0 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f750762c740 10 client.17778 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.375+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d5dbfa03f0 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d5dc299a70 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f74e800f440 con 0x55d5dc328940 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 10 client.17778 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 10 client.17778 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 20 client.17778 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3412f0 0x55d5dc343720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 4 client.17778 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5dc293dc0 con 0x55d5dc3412f0 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f74fe59c640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3412f0 0x55d5dc343720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f74e5ffb640 10 client.17778 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.376+0000 7f74fe59c640 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3412f0 0x55d5dc343720 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c53995b9fa46d08f server_cookie=91ee3bee178a7e07 in_seq=0 out_seq=0 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f74e5ffb640 1 -- 172.21.15.38:0/151714923 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5dc293dc0 con 0x55d5dc3412f0 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f74e5ffb640 10 client.17778 handle_command_reply: tid=0 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 client.17778 shutdown 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 msgr2=0x55d5dc32e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f74e8012700 tx=0x7f74e8012bc0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 shutdown_connections 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc32e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3412f0 0x55d5dc343720 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc32e2a0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f74e5ffb640 0 client.17778 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 --2- 172.21.15.38:0/151714923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc330f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 >> 172.21.15.38:0/151714923 conn(0x55d5dc322750 msgr2=0x55d5dc3244a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 shutdown_connections 2024-04-22T00:08:39.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 1 -- 172.21.15.38:0/151714923 wait complete. 2024-04-22T00:08:39.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.377+0000 7f750762c740 20 client.17778 trim_cache size 0 max 16384 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 Processor -- start 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 -- start start 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc33cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc330f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75001aefd0 con 0x55d5dc328940 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75001b0b10 con 0x55d5dc331520 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f750762c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75001af6a0 con 0x55d5dc328020 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50440/0 (socket says 172.21.15.38:50440) 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f74f7fff640 1 -- 172.21.15.38:0/1582853908 learned_addr learned my addr 172.21.15.38:0/1582853908 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.393+0000 7f74fe59c640 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc33cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74fed9d640 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc330f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 -- 172.21.15.38:0/1582853908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 msgr2=0x55d5dc33cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc33cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 -- 172.21.15.38:0/1582853908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 msgr2=0x55d5dc330f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc330f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 -- 172.21.15.38:0/1582853908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5dc328770 con 0x55d5dc328940 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74f7fff640 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f74e800fa50 tx=0x7f74e8009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e3a86cbefecc90 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74e7fff640 1 -- 172.21.15.38:0/1582853908 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e802d070 con 0x55d5dc328940 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74e7fff640 1 -- 172.21.15.38:0/1582853908 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e8005850 con 0x55d5dc328940 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.394+0000 7f74e7fff640 1 -- 172.21.15.38:0/1582853908 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8005040 con 0x55d5dc328940 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- 172.21.15.38:0/1582853908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 msgr2=0x55d5dc3309d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f74e800fa50 tx=0x7f74e8009d20 comp rx=0 tx=0).stop 2024-04-22T00:08:39.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- 172.21.15.38:0/1582853908 shutdown_connections 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc330f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328940 0x55d5dc3309d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- 172.21.15.38:0/1582853908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328020 0x55d5dc33cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- 172.21.15.38:0/1582853908 >> 172.21.15.38:0/1582853908 conn(0x55d5dc322750 msgr2=0x55d5dc321850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- 172.21.15.38:0/1582853908 shutdown_connections 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- 172.21.15.38:0/1582853908 wait complete. 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- start start 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 Processor -- start 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc339740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc3361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75001b0440 con 0x55d5dc328020 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75001afd70 con 0x55d5dc331520 2024-04-22T00:08:39.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75001acfb0 con 0x55d5dc328940 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f750762c740 10 client.0 _ll_register_callbacks cb 0x7f74e80096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.395+0000 7f74fe59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50456/0 (socket says 172.21.15.38:50456) 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 -- 172.21.15.38:0/134959035 learned_addr learned my addr 172.21.15.38:0/134959035 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fed9d640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc3361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74f7fff640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc339740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 -- 172.21.15.38:0/134959035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 msgr2=0x55d5dc339740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc339740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 -- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 msgr2=0x55d5dc3361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc3361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 -- 172.21.15.38:0/134959035 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5dc32ec20 con 0x55d5dc328020 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74f7fff640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc339740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74fe59c640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f74f00bd440 tx=0x7f74f00b9b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d6a6bcbd5ff532 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.401 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f002f980 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f74f00bba90 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74e8005850 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f74f00bde90 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f750762c740 10 client.17784 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d5dc06a7d0 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.396+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e8005850 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f74f00eeb30 con 0x55d5dc328020 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 10 client.17784 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 10 client.17784 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 20 client.17784 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3a8560 0x55d5dc3aa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.402 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 4 client.17784 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5dc2116e0 con 0x55d5dc3a8560 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74f7fff640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3a8560 0x55d5dc3aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74f7fff640 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3a8560 0x55d5dc3aa980 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b48ff939de297f2b server_cookie=9b1da65002af6dc7 in_seq=0 out_seq=0 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74e5ffb640 10 client.17784 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74e5ffb640 1 -- 172.21.15.38:0/134959035 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5dc2116e0 con 0x55d5dc3a8560 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f74e5ffb640 10 client.17784 handle_command_reply: tid=0 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.397+0000 7f750762c740 1 client.17784 shutdown 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 msgr2=0x55d5dc339200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f74f00bd440 tx=0x7f74f00b9b60 comp rx=0 tx=0).stop 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 shutdown_connections 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d5dc331520 0x55d5dc3361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5dc328940 0x55d5dc339740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55d5dc3a8560 0x55d5dc3aa980 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 --2- 172.21.15.38:0/134959035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d5dc328020 0x55d5dc339200 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f74e5ffb640 0 client.17784 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 >> 172.21.15.38:0/134959035 conn(0x55d5dc322750 msgr2=0x55d5dc323a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 shutdown_connections 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 1 -- 172.21.15.38:0/134959035 wait complete. 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f750762c740 20 client.17784 trim_cache size 0 max 16384 2024-04-22T00:08:39.404 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 msgr2=0x7f74ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 0x7f74ec0a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005fd0 tx=0x7f74f8012b30 comp rx=0 tx=0).stop 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 msgr2=0x7f7500198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009900 tx=0x7f74f00042f0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 shutdown_connections 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f74ec0a2d00 0x7f74ec0a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750019cb20 0x7f750019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f750006a780 0x7f75001987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 --2- 172.21.15.38:0/1366060670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7500069dc0 0x7f7500198260 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 >> 172.21.15.38:0/1366060670 conn(0x7f75000776a0 msgr2=0x7f75000fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 shutdown_connections 2024-04-22T00:08:39.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.398+0000 7f7504965640 1 -- 172.21.15.38:0/1366060670 wait complete. 2024-04-22T00:08:39.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:08:39.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 -- 172.21.15.38:0/2009091742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 msgr2=0x7fd28806a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 --2- 172.21.15.38:0/2009091742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd28806a1c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009900 tx=0x7fd27c02bcd0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 -- 172.21.15.38:0/2009091742 shutdown_connections 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 --2- 172.21.15.38:0/2009091742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd28806a700 0x7fd28810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 --2- 172.21.15.38:0/2009091742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd28806a1c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 --2- 172.21.15.38:0/2009091742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd288107820 0x7fd288107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 -- 172.21.15.38:0/2009091742 >> 172.21.15.38:0/2009091742 conn(0x7fd288077620 msgr2=0x7fd288077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 -- 172.21.15.38:0/2009091742 shutdown_connections 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.519+0000 7fd28e2df640 1 -- 172.21.15.38:0/2009091742 wait complete. 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 Processor -- start 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 -- start start 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd28806a700 0x7fd288198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd288107820 0x7fd28819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd288111ba0 con 0x7fd288069d60 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd288111a20 con 0x7fd28806a700 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28e2df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd288111d20 con 0x7fd288107820 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50470/0 (socket says 172.21.15.38:50470) 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 -- 172.21.15.38:0/596324543 learned_addr learned my addr 172.21.15.38:0/596324543 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd2877fe640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd28806a700 0x7fd288198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd28c87c640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd288107820 0x7fd28819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 -- 172.21.15.38:0/596324543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd288107820 msgr2=0x7fd28819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd288107820 0x7fd28819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 -- 172.21.15.38:0/596324543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd28806a700 msgr2=0x7fd288198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd28806a700 0x7fd288198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 -- 172.21.15.38:0/596324543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28819cff0 con 0x7fd288069d60 2024-04-22T00:08:39.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.520+0000 7fd287fff640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd27800d950 tx=0x7fd27800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.521+0000 7fd2857fa640 1 -- 172.21.15.38:0/596324543 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278017070 con 0x7fd288069d60 2024-04-22T00:08:39.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.521+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28819d2e0 con 0x7fd288069d60 2024-04-22T00:08:39.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.521+0000 7fd2857fa640 1 -- 172.21.15.38:0/596324543 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd278009c30 con 0x7fd288069d60 2024-04-22T00:08:39.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.521+0000 7fd2857fa640 1 -- 172.21.15.38:0/596324543 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27800f6b0 con 0x7fd288069d60 2024-04-22T00:08:39.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.521+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2881a4b70 con 0x7fd288069d60 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 Processor -- start 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 -- start start 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1b3b0 0x560a32a283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560a3297f530 con 0x560a32a1b3b0 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a32927830 con 0x560a32a1aa00 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd290fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560a3297b4f0 con 0x560a32a28930 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd2857fa640 1 -- 172.21.15.38:0/596324543 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd2780109d0 con 0x7fd288069d60 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd2877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34662/0 (socket says 172.21.15.38:34662) 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.524+0000 7fd2877fe640 1 -- 172.21.15.38:0/1379996467 learned_addr learned my addr 172.21.15.38:0/1379996467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd28c87c640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd287fff640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1b3b0 0x560a32a283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 -- 172.21.15.38:0/1379996467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 msgr2=0x560a32a1ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a1ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 -- 172.21.15.38:0/1379996467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1b3b0 msgr2=0x560a32a283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1b3b0 0x560a32a283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 -- 172.21.15.38:0/1379996467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a3294b2c0 con 0x560a32a28930 2024-04-22T00:08:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd28c87c640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a1ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009900 tx=0x7fd27c02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f391a587efeb1d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd2857fa640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 0x7fd2700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd28c87c640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 0x7fd2700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.525+0000 7fd26e7fc640 1 -- 172.21.15.38:0/1379996467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c043070 con 0x560a32a28930 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd2857fa640 1 -- 172.21.15.38:0/596324543 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd2780cf2f0 con 0x7fd288069d60 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd26e7fc640 1 -- 172.21.15.38:0/1379996467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd27c02c950 con 0x560a32a28930 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd26e7fc640 1 -- 172.21.15.38:0/1379996467 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c004c20 con 0x560a32a28930 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 -- 172.21.15.38:0/1379996467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 msgr2=0x560a32a2ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009900 tx=0x7fd27c02c670 comp rx=0 tx=0).stop 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd28c87c640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 0x7fd2700a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd280005fd0 tx=0x7fd280009600 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 -- 172.21.15.38:0/1379996467 shutdown_connections 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a2ad20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1b3b0 0x560a32a283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1379996467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a1ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 -- 172.21.15.38:0/1379996467 >> 172.21.15.38:0/1379996467 conn(0x560a32a15750 msgr2=0x560a32a15b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 -- 172.21.15.38:0/1379996467 shutdown_connections 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.526+0000 7fd290fa1740 1 -- 172.21.15.38:0/1379996467 wait complete. 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 -- start start 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 Processor -- start 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1b3b0 0x560a32a255e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a25b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560a32a2b690 con 0x560a32a1aa00 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560a3293af10 con 0x560a32a1b3b0 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560a32a2b810 con 0x560a32a28930 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd290fa1740 10 client.0 _ll_register_callbacks cb 0x560a32692be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50486/0 (socket says 172.21.15.38:50486) 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 -- 172.21.15.38:0/1067445463 learned_addr learned my addr 172.21.15.38:0/1067445463 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd28c87c640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a25b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1b3b0 0x560a32a255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 -- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 msgr2=0x560a32a25b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a25b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 -- 172.21.15.38:0/1067445463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1b3b0 msgr2=0x560a32a255e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1b3b0 0x560a32a255e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.527+0000 7fd287fff640 1 -- 172.21.15.38:0/1067445463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a3290cc80 con 0x560a32a1aa00 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd287fff640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd278009610 tx=0x7fd2780d2a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56649c91f535576b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278020070 con 0x560a32a1aa00 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd27809a9d0 con 0x560a32a1aa00 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd27c02c950 con 0x560a32a1aa00 2024-04-22T00:08:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd27809aee0 con 0x560a32a1aa00 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 10 client.17796 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560a326933f0 con 0x560a32a1aa00 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd27c02c950 con 0x560a32a1aa00 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd278084ac0 con 0x560a32a1aa00 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 10 client.17796 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 10 client.17796 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.528+0000 7fd290fa1740 20 client.17796 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a388e0 0x560a32a3acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd290fa1740 4 client.17796 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a3298ca70 con 0x560a32a388e0 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a388e0 0x560a32a3acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd2677fe640 10 client.17796 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.529+0000 7fd2877fe640 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a388e0 0x560a32a3acd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f85a5d1007c1e782 server_cookie=cbef81ad829aa654 in_seq=0 out_seq=0 2024-04-22T00:08:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd2677fe640 1 -- 172.21.15.38:0/1067445463 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a3298ca70 con 0x560a32a388e0 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd2677fe640 10 client.17796 handle_command_reply: tid=0 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 client.17796 shutdown 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 msgr2=0x560a32a28250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd278009610 tx=0x7fd2780d2a30 comp rx=0 tx=0).stop 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 shutdown_connections 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a388e0 0x560a32a3acd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a28930 0x560a32a25b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd2677fe640 0 client.17796 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1b3b0 0x560a32a255e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 --2- 172.21.15.38:0/1067445463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a28250 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 >> 172.21.15.38:0/1067445463 conn(0x560a32a15750 msgr2=0x560a32a174a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 shutdown_connections 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 1 -- 172.21.15.38:0/1067445463 wait complete. 2024-04-22T00:08:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.530+0000 7fd290fa1740 20 client.17796 trim_cache size 0 max 16384 2024-04-22T00:08:39.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 Processor -- start 2024-04-22T00:08:39.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 -- start start 2024-04-22T00:08:39.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a345f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a20aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881ac770 con 0x560a32a28930 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881ae2b0 con 0x560a32a1aa00 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd290fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881ace40 con 0x560a32a1b3b0 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50496/0 (socket says 172.21.15.38:50496) 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 -- 172.21.15.38:0/3188005575 learned_addr learned my addr 172.21.15.38:0/3188005575 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd287fff640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd2877fe640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a20aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 -- 172.21.15.38:0/3188005575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 msgr2=0x560a32a20aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a20aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 -- 172.21.15.38:0/3188005575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 msgr2=0x560a32a345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a345f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd28c87c640 1 -- 172.21.15.38:0/3188005575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a32a1b150 con 0x560a32a28930 2024-04-22T00:08:39.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd287fff640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a345f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.545+0000 7fd2877fe640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a20aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.546+0000 7fd28c87c640 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd2800118b0 tx=0x7fd280016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5174da9957a51706 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.546+0000 7fd26dffb640 1 -- 172.21.15.38:0/3188005575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280011ea0 con 0x560a32a28930 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.546+0000 7fd26dffb640 1 -- 172.21.15.38:0/3188005575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd28001d910 con 0x560a32a28930 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.546+0000 7fd26dffb640 1 -- 172.21.15.38:0/3188005575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280020a90 con 0x560a32a28930 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- 172.21.15.38:0/3188005575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 msgr2=0x560a32a40570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd2800118b0 tx=0x7fd280016000 comp rx=0 tx=0).stop 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- 172.21.15.38:0/3188005575 shutdown_connections 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a28930 0x560a32a40570 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a20aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3188005575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a1aa00 0x560a32a345f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- 172.21.15.38:0/3188005575 >> 172.21.15.38:0/3188005575 conn(0x560a32a15750 msgr2=0x560a32a20390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- 172.21.15.38:0/3188005575 shutdown_connections 2024-04-22T00:08:39.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- 172.21.15.38:0/3188005575 wait complete. 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- start start 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 Processor -- start 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a2c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a28930 0x560a32a2c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881adbe0 con 0x560a32a1aa00 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881ad510 con 0x560a32a28930 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2881aa750 con 0x560a32a1b3b0 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd290fa1740 10 client.0 _ll_register_callbacks cb 0x7fd27800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd287fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50512/0 (socket says 172.21.15.38:50512) 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd287fff640 1 -- 172.21.15.38:0/3552232600 learned_addr learned my addr 172.21.15.38:0/3552232600 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.547+0000 7fd2877fe640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a2c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd28c87c640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a28930 0x560a32a2c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 -- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 msgr2=0x560a32a2c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a2c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 -- 172.21.15.38:0/3552232600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a28930 msgr2=0x560a32a2c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a28930 0x560a32a2c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 -- 172.21.15.38:0/3552232600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a3275d7d0 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd287fff640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd2780a3c10 tx=0x7fd278000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283fe2d6538bc071 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd2677fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278020040 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fd2780b4410 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd28001d910 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd27809b350 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd290fa1740 10 client.17802 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560a32a9ebd0 con 0x560a32a1aa00 2024-04-22T00:08:39.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.548+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd28001d910 con 0x560a32a1aa00 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd27809a9d0 con 0x560a32a1aa00 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 10 client.17802 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 10 client.17802 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 20 client.17802 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a9f100 0x560a32aa14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 4 client.17802 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a32a25eb0 con 0x560a32a9f100 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2877fe640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a9f100 0x560a32aa14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2677fe640 10 client.17802 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2877fe640 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a9f100 0x560a32aa14f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=998377fc26694d80 server_cookie=c9daa35e3acfb2b6 in_seq=0 out_seq=0 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2677fe640 1 -- 172.21.15.38:0/3552232600 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a32a25eb0 con 0x560a32a9f100 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd2677fe640 10 client.17802 handle_command_reply: tid=0 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.549+0000 7fd290fa1740 1 client.17802 shutdown 2024-04-22T00:08:39.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 msgr2=0x560a32a30e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd2780a3c10 tx=0x7fd278000bc0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 shutdown_connections 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x560a32a9f100 0x560a32aa14f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a32a28930 0x560a32a2c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a32a1b3b0 0x560a32a2c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 --2- 172.21.15.38:0/3552232600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560a32a1aa00 0x560a32a30e70 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd2677fe640 0 client.17802 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 >> 172.21.15.38:0/3552232600 conn(0x560a32a15750 msgr2=0x560a32a20c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 shutdown_connections 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 1 -- 172.21.15.38:0/3552232600 wait complete. 2024-04-22T00:08:39.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd290fa1740 20 client.17802 trim_cache size 0 max 16384 2024-04-22T00:08:39.553 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:39.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 msgr2=0x7fd2700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 0x7fd2700a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd280005fd0 tx=0x7fd280009600 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 msgr2=0x7fd288198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd27800d950 tx=0x7fd27800de10 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 shutdown_connections 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd2700a2d00 0x7fd2700a51c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd288107820 0x7fd28819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd28806a700 0x7fd288198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 --2- 172.21.15.38:0/596324543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd288069d60 0x7fd288198000 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 >> 172.21.15.38:0/596324543 conn(0x7fd288077620 msgr2=0x7fd2880ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.550+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 shutdown_connections 2024-04-22T00:08:39.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.551+0000 7fd28e2df640 1 -- 172.21.15.38:0/596324543 wait complete. 2024-04-22T00:08:39.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:08:39.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:39 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3011712297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:39.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.749+0000 7f4b9c884640 1 -- 172.21.15.38:0/2668387017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98105ad0 msgr2=0x7f4b98105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.749+0000 7f4b9c884640 1 --2- 172.21.15.38:0/2668387017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98105ad0 0x7f4b98105f60 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009ca0 tx=0x7f4b9002c230 comp rx=0 tx=0).stop 2024-04-22T00:08:39.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.753+0000 7f4b9c884640 1 -- 172.21.15.38:0/2668387017 shutdown_connections 2024-04-22T00:08:39.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.753+0000 7f4b9c884640 1 --2- 172.21.15.38:0/2668387017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b98105ad0 0x7f4b98105f60 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.753+0000 7f4b9c884640 1 --2- 172.21.15.38:0/2668387017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b981010d0 0x7f4b98105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.753+0000 7f4b9c884640 1 --2- 172.21.15.38:0/2668387017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981007b0 0x7f4b98100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.753+0000 7f4b9c884640 1 -- 172.21.15.38:0/2668387017 >> 172.21.15.38:0/2668387017 conn(0x7f4b98077740 msgr2=0x7f4b98077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.755+0000 7f4b9c884640 1 -- 172.21.15.38:0/2668387017 shutdown_connections 2024-04-22T00:08:39.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.756+0000 7f4b9c884640 1 -- 172.21.15.38:0/2668387017 wait complete. 2024-04-22T00:08:39.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.756+0000 7f4b9c884640 1 Processor -- start 2024-04-22T00:08:39.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 -- start start 2024-04-22T00:08:39.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981010d0 0x7f4b9810ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 0x7f4b9810f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981075b0 con 0x7f4b981007b0 2024-04-22T00:08:39.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98107430 con 0x7f4b981010d0 2024-04-22T00:08:39.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.757+0000 7f4b9c884640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98107730 con 0x7f4b98105ad0 2024-04-22T00:08:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.759+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.759+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50526/0 (socket says 172.21.15.38:50526) 2024-04-22T00:08:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.759+0000 7f4b9659c640 1 -- 172.21.15.38:0/3123542112 learned_addr learned my addr 172.21.15.38:0/3123542112 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.760+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 0x7f4b9810f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.761+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981010d0 0x7f4b9810ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.763+0000 7f4b96d9d640 1 -- 172.21.15.38:0/3123542112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981010d0 msgr2=0x7f4b9810ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.763+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981010d0 0x7f4b9810ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.763+0000 7f4b96d9d640 1 -- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 msgr2=0x7f4b98114c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.763+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.763+0000 7f4b96d9d640 1 -- 172.21.15.38:0/3123542112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9810fa90 con 0x7f4b98105ad0 2024-04-22T00:08:39.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.764+0000 7f4b9659c640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:39.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.766+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 0x7f4b9810f1d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009bf0 tx=0x7f4b90035d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.768+0000 7f4b877fe640 1 -- 172.21.15.38:0/3123542112 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90039040 con 0x7f4b98105ad0 2024-04-22T00:08:39.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.768+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9811d7a0 con 0x7f4b98105ad0 2024-04-22T00:08:39.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.768+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9811dcf0 con 0x7f4b98105ad0 2024-04-22T00:08:39.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.768+0000 7f4b877fe640 1 -- 172.21.15.38:0/3123542112 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b90003e50 con 0x7f4b98105ad0 2024-04-22T00:08:39.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.768+0000 7f4b877fe640 1 -- 172.21.15.38:0/3123542112 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90032e90 con 0x7f4b98105ad0 2024-04-22T00:08:39.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.779+0000 7f4b877fe640 1 -- 172.21.15.38:0/3123542112 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4b900042a0 con 0x7f4b98105ad0 2024-04-22T00:08:39.776 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.781+0000 7f4b877fe640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 0x7f4b800a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.781+0000 7f4b877fe640 1 -- 172.21.15.38:0/3123542112 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4b900f20d0 con 0x7f4b98105ad0 2024-04-22T00:08:39.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.782+0000 7f4b9659c640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 0x7f4b800a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.786+0000 7f4b9f46b740 1 Processor -- start 2024-04-22T00:08:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.786+0000 7f4b9659c640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 0x7f4b800a5340 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4b88005fd0 tx=0x7f4b8800b320 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 -- start start 2024-04-22T00:08:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 0x561b2e908400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 0x561b2e913910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561b2e86c530 con 0x561b2e911520 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561b2e814830 con 0x561b2e908940 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.788+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b2e8684f0 con 0x561b2e908020 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 0x561b2e908400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 0x561b2e908400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34710/0 (socket says 172.21.15.38:34710) 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b9659c640 1 -- 172.21.15.38:0/1789422900 learned_addr learned my addr 172.21.15.38:0/1789422900 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 0x561b2e913910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 -- 172.21.15.38:0/1789422900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 msgr2=0x561b2e908400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 0x561b2e908400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 -- 172.21.15.38:0/1789422900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 msgr2=0x561b2e910fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.789+0000 7f4b96d9d640 1 -- 172.21.15.38:0/1789422900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2e8382c0 con 0x561b2e911520 2024-04-22T00:08:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.790+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 0x561b2e913910 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4b900ef5f0 tx=0x7f4b900bdf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b889e1ffe9ee11 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.790+0000 7f4b84ff9640 1 -- 172.21.15.38:0/1789422900 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b900beac0 con 0x561b2e911520 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.790+0000 7f4b84ff9640 1 -- 172.21.15.38:0/1789422900 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b900be3c0 con 0x561b2e911520 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.791+0000 7f4b9f46b740 1 -- 172.21.15.38:0/1789422900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 msgr2=0x561b2e913910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.791+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 0x561b2e913910 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4b900ef5f0 tx=0x7f4b900bdf00 comp rx=0 tx=0).stop 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.792+0000 7f4b9f46b740 1 -- 172.21.15.38:0/1789422900 shutdown_connections 2024-04-22T00:08:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.792+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e911520 0x561b2e913910 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.792+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.792+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/1789422900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e908020 0x561b2e908400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.792+0000 7f4b9f46b740 1 -- 172.21.15.38:0/1789422900 >> 172.21.15.38:0/1789422900 conn(0x561b2e902750 msgr2=0x561b2e902b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.796+0000 7f4b9f46b740 1 -- 172.21.15.38:0/1789422900 shutdown_connections 2024-04-22T00:08:39.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.797+0000 7f4b9f46b740 1 -- 172.21.15.38:0/1789422900 wait complete. 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.797+0000 7f4b9f46b740 1 -- start start 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.797+0000 7f4b9f46b740 1 Processor -- start 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908020 0x561b2e910d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908940 0x561b2e90e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561b2e9140d0 con 0x561b2e908940 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561b2e827f10 con 0x561b2e908020 2024-04-22T00:08:39.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b2e914250 con 0x561b2e911520 2024-04-22T00:08:39.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b9f46b740 10 client.0 _ll_register_callbacks cb 0x561b2e57fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b96d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b96d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34718/0 (socket says 172.21.15.38:34718) 2024-04-22T00:08:39.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.798+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2977400348 learned_addr learned my addr 172.21.15.38:0/2977400348 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b9659c640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908020 0x561b2e910d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908940 0x561b2e90e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2977400348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908020 msgr2=0x561b2e910d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908020 0x561b2e910d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2977400348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908940 msgr2=0x561b2e90e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908940 0x561b2e90e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.799+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2977400348 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2e7f9c80 con 0x561b2e911520 2024-04-22T00:08:39.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.800+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b900bc670 tx=0x7f4b900be850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d40850448b7dab48 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.800+0000 7f4b7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:39.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b900c2770 con 0x561b2e911520 2024-04-22T00:08:39.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b900be3c0 con 0x561b2e911520 2024-04-22T00:08:39.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4b900c6db0 con 0x561b2e911520 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4b900c7030 con 0x561b2e911520 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b9f46b740 10 client.27001 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b2e5803f0 con 0x561b2e911520 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.801+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b900be3c0 con 0x561b2e911520 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4b9003c030 con 0x561b2e911520 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 10 client.27001 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 10 client.27001 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 20 client.27001 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9320e0 0x561b2e9344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 4 client.27001 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:39.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b2e879a70 con 0x561b2e9320e0 2024-04-22T00:08:39.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.802+0000 7f4b9659c640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9320e0 0x561b2e9344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.803+0000 7f4b9659c640 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9320e0 0x561b2e9344d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a80a8f037a3edb12 server_cookie=bf585b8a866568a5 in_seq=0 out_seq=0 2024-04-22T00:08:39.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.803+0000 7f4b7dffb640 10 client.27001 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:39.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.804+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2977400348 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b2e879a70 con 0x561b2e9320e0 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.804+0000 7f4b7dffb640 10 client.27001 handle_command_reply: tid=0 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.804+0000 7f4b9f46b740 1 client.27001 shutdown 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 msgr2=0x561b2e90e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b900bc670 tx=0x7f4b900be850 comp rx=0 tx=0).stop 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 shutdown_connections 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9320e0 0x561b2e9344d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e90e650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908940 0x561b2e90e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2977400348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908020 0x561b2e910d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 >> 172.21.15.38:0/2977400348 conn(0x561b2e902750 msgr2=0x561b2e9044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b7dffb640 0 client.27001 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 shutdown_connections 2024-04-22T00:08:39.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2977400348 wait complete. 2024-04-22T00:08:39.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.805+0000 7f4b9f46b740 20 client.27001 trim_cache size 0 max 16384 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.833+0000 7f4b9f46b740 1 Processor -- start 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 -- start start 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e910d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50554/0 (socket says 172.21.15.38:50554) 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9659c640 1 -- 172.21.15.38:0/3742921011 learned_addr learned my addr 172.21.15.38:0/3742921011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7800fde0 con 0x561b2e908020 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78011920 con 0x561b2e908940 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b780104b0 con 0x561b2e911520 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e910d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.834+0000 7f4b95d9b640 1 -- 172.21.15.38:0/3742921011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 msgr2=0x561b2e910d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e910d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b95d9b640 1 -- 172.21.15.38:0/3742921011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 msgr2=0x561b2e91cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b95d9b640 1 -- 172.21.15.38:0/3742921011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2e908770 con 0x561b2e908940 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b9659c640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x561b2e90d970 tx=0x7f4b8c00bae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819d39576bcc68a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b7ffff640 1 -- 172.21.15.38:0/3742921011 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c018020 con 0x561b2e908940 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b7ffff640 1 -- 172.21.15.38:0/3742921011 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b8c00fbc0 con 0x561b2e908940 2024-04-22T00:08:39.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.835+0000 7f4b7ffff640 1 -- 172.21.15.38:0/3742921011 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c014cf0 con 0x561b2e908940 2024-04-22T00:08:39.833 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:39 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3011712297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:39.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 msgr2=0x561b2e910840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910840 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x561b2e90d970 tx=0x7f4b8c00bae0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 shutdown_connections 2024-04-22T00:08:39.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e910d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e910840 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/3742921011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e91cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 >> 172.21.15.38:0/3742921011 conn(0x561b2e902750 msgr2=0x561b2e9096f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 shutdown_connections 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- 172.21.15.38:0/3742921011 wait complete. 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.836+0000 7f4b9f46b740 1 -- start start 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 Processor -- start 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e9196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e914ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78011250 con 0x561b2e908020 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78010b80 con 0x561b2e908940 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7800ddc0 con 0x561b2e911520 2024-04-22T00:08:39.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9f46b740 10 client.0 _ll_register_callbacks cb 0x7f4b900babf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e9196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34728/0 (socket says 172.21.15.38:34728) 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2433388190 learned_addr learned my addr 172.21.15.38:0/2433388190 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b9659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e9196a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50568/0 (socket says 172.21.15.38:50568) 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b95d9b640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e914ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2433388190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 msgr2=0x561b2e914ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e914ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2433388190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 msgr2=0x561b2e9196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e9196a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.837+0000 7f4b96d9d640 1 -- 172.21.15.38:0/2433388190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2e64a7d0 con 0x561b2e911520 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b96d9d640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4b90032ca0 tx=0x7f4b900bab80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dbcec9345ac3d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b900bdd20 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b8c00fbc0 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f4b900bd930 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4b900bdb30 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 10 client.27007 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b2e64a7d0 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b8c00fbc0 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4b90004590 con 0x561b2e911520 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 10 client.27007 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 10 client.27007 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 20 client.27007 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9984a0 0x561b2e99a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 4 client.27007 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:39.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.838+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b2e873dc0 con 0x561b2e9984a0 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9659c640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9984a0 0x561b2e99a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b7dffb640 10 client.27007 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9659c640 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9984a0 0x561b2e99a890 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5c815538ec34000 server_cookie=3ca1fa66e9f5d844 in_seq=0 out_seq=0 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b7dffb640 1 -- 172.21.15.38:0/2433388190 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b2e873dc0 con 0x561b2e9984a0 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b7dffb640 10 client.27007 handle_command_reply: tid=0 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 client.27007 shutdown 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 msgr2=0x561b2e914ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4b90032ca0 tx=0x7f4b900bab80 comp rx=0 tx=0).stop 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 shutdown_connections 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x561b2e9984a0 0x561b2e99a890 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b2e911520 0x561b2e914ff0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b7dffb640 0 client.27007 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b2e908940 0x561b2e914ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 --2- 172.21.15.38:0/2433388190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561b2e908020 0x561b2e9196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.839+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 >> 172.21.15.38:0/2433388190 conn(0x561b2e902750 msgr2=0x561b2e903af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 shutdown_connections 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9f46b740 1 -- 172.21.15.38:0/2433388190 wait complete. 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9f46b740 20 client.27007 trim_cache size 0 max 16384 2024-04-22T00:08:39.843 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 msgr2=0x7f4b800a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 0x7f4b800a5340 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4b88005fd0 tx=0x7f4b8800b320 comp rx=0 tx=0).stop 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 msgr2=0x7f4b9810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 0x7f4b9810f1d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009bf0 tx=0x7f4b90035d80 comp rx=0 tx=0).stop 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 shutdown_connections 2024-04-22T00:08:39.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4b800a2e80 0x7f4b800a5340 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b98105ad0 0x7f4b9810f1d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b981010d0 0x7f4b9810ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 --2- 172.21.15.38:0/3123542112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b981007b0 0x7f4b98114c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 >> 172.21.15.38:0/3123542112 conn(0x7f4b98077740 msgr2=0x7f4b98103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 shutdown_connections 2024-04-22T00:08:39.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.840+0000 7f4b9c884640 1 -- 172.21.15.38:0/3123542112 wait complete. 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 -- 172.21.15.38:0/1490107799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0100670 msgr2=0x7fb2e0100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/1490107799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0100670 0x7fb2e0100af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009860 tx=0x7fb2d002bb70 comp rx=0 tx=0).stop 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 -- 172.21.15.38:0/1490107799 shutdown_connections 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/1490107799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0101030 0x7fb2e010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/1490107799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0100670 0x7fb2e0100af0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/1490107799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0106630 0x7fb2e0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 -- 172.21.15.38:0/1490107799 >> 172.21.15.38:0/1490107799 conn(0x7fb2e00777c0 msgr2=0x7fb2e0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 -- 172.21.15.38:0/1490107799 shutdown_connections 2024-04-22T00:08:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.848+0000 7fb2e7a27640 1 -- 172.21.15.38:0/1490107799 wait complete. 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.849+0000 7fb2e7a27640 1 Processor -- start 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.849+0000 7fb2e7a27640 1 -- start start 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0100670 0x7fb2e0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 0x7fb2e01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0110a40 con 0x7fb2e0106630 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01108c0 con 0x7fb2e0101030 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e7a27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e0110bc0 con 0x7fb2e0100670 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50584/0 (socket says 172.21.15.38:50584) 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2795832251 learned_addr learned my addr 172.21.15.38:0/2795832251 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0100670 0x7fb2e0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 0x7fb2e01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2795832251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0100670 msgr2=0x7fb2e0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0100670 0x7fb2e0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2795832251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 msgr2=0x7fb2e01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 0x7fb2e01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2795832251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e019cf10 con 0x7fb2e0106630 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 0x7fb2e01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009fd0 tx=0x7fb2d800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.850+0000 7fb2d67fc640 1 -- 172.21.15.38:0/2795832251 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8004b20 con 0x7fb2e0106630 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.851+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e019d200 con 0x7fb2e0106630 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.851+0000 7fb2d67fc640 1 -- 172.21.15.38:0/2795832251 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d8003e50 con 0x7fb2e0106630 2024-04-22T00:08:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.851+0000 7fb2d67fc640 1 -- 172.21.15.38:0/2795832251 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d800b630 con 0x7fb2e0106630 2024-04-22T00:08:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.851+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e01a4a90 con 0x7fb2e0106630 2024-04-22T00:08:39.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:08:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2d67fc640 1 -- 172.21.15.38:0/2795832251 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb2d801d020 con 0x7fb2e0106630 2024-04-22T00:08:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 Processor -- start 2024-04-22T00:08:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 -- start start 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b48f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2d67fc640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 0x7fb2cc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b54600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5602e7aacb70 con 0x5602e7b49440 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602e7a55470 con 0x5602e7b52210 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.853+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5602e7aa8b30 con 0x5602e7b48b20 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 0x7fb2cc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50592/0 (socket says 172.21.15.38:50592) 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2240241585 learned_addr learned my addr 172.21.15.38:0/2240241585 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2d67fc640 1 -- 172.21.15.38:0/2795832251 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb2d80cf740 con 0x7fb2e0106630 2024-04-22T00:08:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b48f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b54600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2240241585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 msgr2=0x5602e7b48f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b48f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2240241585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 msgr2=0x5602e7b54600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b54600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2240241585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e77c1210 con 0x5602e7b49440 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b54600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009ca0 tx=0x7fb2dc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a908b2a3cff00856 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 0x7fb2cc0a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002c180 tx=0x7fb2d002e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2cb7fe640 1 -- 172.21.15.38:0/2240241585 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc01e070 con 0x5602e7b49440 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2cb7fe640 1 -- 172.21.15.38:0/2240241585 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2dc019b10 con 0x5602e7b49440 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.854+0000 7fb2cb7fe640 1 -- 172.21.15.38:0/2240241585 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc00ea70 con 0x5602e7b49440 2024-04-22T00:08:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2240241585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 msgr2=0x5602e7b51cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009ca0 tx=0x7fb2dc011810 comp rx=0 tx=0).stop 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2240241585 shutdown_connections 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b54600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b51cd0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2240241585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b48f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2240241585 >> 172.21.15.38:0/2240241585 conn(0x5602e7b43440 msgr2=0x5602e7b43820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2240241585 shutdown_connections 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2240241585 wait complete. 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.855+0000 7fb2ea60d740 1 -- start start 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 Processor -- start 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 0x5602e7b4f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b4fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5602e7b54e40 con 0x5602e7b48b20 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602e7a68a10 con 0x5602e7b49440 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5602e7b54fc0 con 0x5602e7b52210 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2ea60d740 10 client.0 _ll_register_callbacks cb 0x5602e7aa3b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50600/0 (socket says 172.21.15.38:50600) 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2828189533 learned_addr learned my addr 172.21.15.38:0/2828189533 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b4fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 0x5602e7b4f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2828189533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 msgr2=0x5602e7b4fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b4fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 msgr2=0x5602e7b4f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 0x5602e7b4f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 -- 172.21.15.38:0/2828189533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e788d620 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 0x5602e7b4f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009c70 tx=0x7fb2dc0198b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83509484317408a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.856+0000 7fb2c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc027070 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb2dc004220 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2dc019b10 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb2dc032050 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 10 client.17832 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5602e7b60ff0 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2dc019b10 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb2dc00e870 con 0x5602e7b48b20 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 10 client.17832 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 10 client.17832 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 20 client.17832 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7b62400 0x5602e7b647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 4 client.17832 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602e7aba0b0 con 0x5602e7b62400 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.857+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7b62400 0x5602e7b647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.858+0000 7fb2c8ff9640 10 client.17832 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.858+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7b62400 0x5602e7b647f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ea2c89c07b7285a server_cookie=61e26c0aa8be472f in_seq=0 out_seq=0 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/2828189533 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602e7aba0b0 con 0x5602e7b62400 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2c8ff9640 10 client.17832 handle_command_reply: tid=0 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 client.17832 shutdown 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 msgr2=0x5602e7b4f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009c70 tx=0x7fb2dc0198b0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 shutdown_connections 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b4fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b49440 0x5602e7b4f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7b62400 0x5602e7b647f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2828189533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b48b20 0x5602e7b4f220 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2c8ff9640 0 client.17832 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 >> 172.21.15.38:0/2828189533 conn(0x5602e7b43440 msgr2=0x5602e7b45190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 shutdown_connections 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2828189533 wait complete. 2024-04-22T00:08:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.859+0000 7fb2ea60d740 20 client.17832 trim_cache size 0 max 16384 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 Processor -- start 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 -- start start 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b51950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b69f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01ae950 con 0x5602e7b49440 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01b0490 con 0x5602e7b52210 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01af020 con 0x5602e7b48b20 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e4fc2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e4fc2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50612/0 (socket says 172.21.15.38:50612) 2024-04-22T00:08:39.871 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e4fc2640 1 -- 172.21.15.38:0/2622937402 learned_addr learned my addr 172.21.15.38:0/2622937402 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b69f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b51950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2622937402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 msgr2=0x5602e7b51950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b51950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.874+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2622937402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 msgr2=0x5602e7b5dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.875+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.875+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/2622937402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e7b3d7a0 con 0x5602e7b52210 2024-04-22T00:08:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.876+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.877+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b69f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80ccb50 tx=0x7fb2d8004210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19414777b4716c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.877+0000 7fb2caffd640 1 -- 172.21.15.38:0/2622937402 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8084e60 con 0x5602e7b52210 2024-04-22T00:08:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.877+0000 7fb2caffd640 1 -- 172.21.15.38:0/2622937402 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d800ce30 con 0x5602e7b52210 2024-04-22T00:08:39.873 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.877+0000 7fb2caffd640 1 -- 172.21.15.38:0/2622937402 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d801a040 con 0x5602e7b52210 2024-04-22T00:08:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.878+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2622937402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 msgr2=0x5602e7b69f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.874 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.878+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b69f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80ccb50 tx=0x7fb2d8004210 comp rx=0 tx=0).stop 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2622937402 shutdown_connections 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b52210 0x5602e7b69f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b5dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/2622937402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b48b20 0x5602e7b51950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2622937402 >> 172.21.15.38:0/2622937402 conn(0x5602e7b43440 msgr2=0x5602e7b4daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2622937402 shutdown_connections 2024-04-22T00:08:39.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.880+0000 7fb2ea60d740 1 -- 172.21.15.38:0/2622937402 wait complete. 2024-04-22T00:08:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.881+0000 7fb2ea60d740 1 -- start start 2024-04-22T00:08:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.881+0000 7fb2ea60d740 1 Processor -- start 2024-04-22T00:08:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.881+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.881+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.881+0000 7fb2ea60d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b56610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01afdc0 con 0x5602e7b49440 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01af6f0 con 0x5602e7b48b20 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2ea60d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e01ac930 con 0x5602e7b52210 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e57c3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51756/0 (socket says 172.21.15.38:51756) 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e57c3640 1 -- 172.21.15.38:0/910436109 learned_addr learned my addr 172.21.15.38:0/910436109 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b56610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2ea60d740 10 client.0 _ll_register_callbacks cb 0x7fb2dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 msgr2=0x5602e7b55b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/910436109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 msgr2=0x5602e7b560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e5fc4640 1 -- 172.21.15.38:0/910436109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e7bd0000 con 0x5602e7b52210 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e4fc2640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.882+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2e5fc4640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b56610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2d809bae0 tx=0x7fb2d80a1590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d7ab28feed9062b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8084e60 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7fb2d8086110 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2d800ce30 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.883+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb2d80b9050 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 10 client.27025 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5602e7bd1230 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d8086310 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb2d800b040 con 0x5602e7b52210 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 10 client.27025 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 10 client.27025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 20 client.27025 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7bd18a0 0x5602e7bd3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 4 client.27025 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb2d800ce30 con 0x5602e7bd18a0 2024-04-22T00:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.884+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7bd18a0 0x5602e7bd3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.886+0000 7fb2e57c3640 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7bd18a0 0x5602e7bd3c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760ca45097be721a server_cookie=80c799efdd7a5d5e in_seq=0 out_seq=0 2024-04-22T00:08:39.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:39.886+0000 7fb2c8ff9640 10 client.27025 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:39.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 -- 172.21.15.38:0/2971987930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e41066b0 msgr2=0x7f62e4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2971987930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e41066b0 0x7f62e4106a90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009930 tx=0x7f62e002bc20 comp rx=0 tx=0).stop 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 -- 172.21.15.38:0/2971987930 shutdown_connections 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2971987930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100fd0 0x7f62e410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2971987930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e4100610 0x7f62e4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2971987930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e41066b0 0x7f62e4106a90 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 -- 172.21.15.38:0/2971987930 >> 172.21.15.38:0/2971987930 conn(0x7f62e4077740 msgr2=0x7f62e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 -- 172.21.15.38:0/2971987930 shutdown_connections 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.994+0000 7f62ebf77640 1 -- 172.21.15.38:0/2971987930 wait complete. 2024-04-22T00:08:39.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 Processor -- start 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 -- start start 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100610 0x7f62e4198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e41066b0 0x7f62e41a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4110a30 con 0x7f62e4100fd0 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41108b0 con 0x7f62e41066b0 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ebf77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4110bb0 con 0x7f62e4100610 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62e9512640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62e9512640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50648/0 (socket says 172.21.15.38:50648) 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62e9d13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100610 0x7f62e4198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62e9512640 1 -- 172.21.15.38:0/2401235913 learned_addr learned my addr 172.21.15.38:0/2401235913 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.995+0000 7f62ea514640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e41066b0 0x7f62e41a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:39.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 -- 172.21.15.38:0/2401235913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100610 msgr2=0x7f62e4198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100610 0x7f62e4198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 -- 172.21.15.38:0/2401235913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e41066b0 msgr2=0x7f62e41a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e41066b0 0x7f62e41a3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 -- 172.21.15.38:0/2401235913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e41a4330 con 0x7f62e4100fd0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62e9512640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f62d400d950 tx=0x7f62d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62daffd640 1 -- 172.21.15.38:0/2401235913 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4017030 con 0x7f62e4100fd0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e41a4560 con 0x7f62e4100fd0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62daffd640 1 -- 172.21.15.38:0/2401235913 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d4004020 con 0x7f62e4100fd0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62daffd640 1 -- 172.21.15.38:0/2401235913 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40047a0 con 0x7f62e4100fd0 2024-04-22T00:08:39.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.996+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e41a4a80 con 0x7f62e4100fd0 2024-04-22T00:08:40.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.998+0000 7f62daffd640 1 -- 172.21.15.38:0/2401235913 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f62d40041d0 con 0x7f62e4100fd0 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.998+0000 7f62ee7fd740 1 Processor -- start 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.998+0000 7f62ee7fd740 1 -- start start 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 0x55a24ac18130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac27d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a24ab7c530 con 0x55a24ac18670 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a24ab24830 con 0x55a24ac17d50 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a24ab784f0 con 0x55a24ac25930 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ea514640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 0x55a24ac18130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62daffd640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 0x7f62d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50650/0 (socket says 172.21.15.38:50650) 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ea514640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 0x55a24ac18130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51782/0 (socket says 172.21.15.38:51782) 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9512640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac27d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 -- 172.21.15.38:0/2190641517 learned_addr learned my addr 172.21.15.38:0/2190641517 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62ea514640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 0x7f62d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 -- 172.21.15.38:0/2190641517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 msgr2=0x55a24ac27d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac27d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 -- 172.21.15.38:0/2190641517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 msgr2=0x55a24ac18130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 0x55a24ac18130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 -- 172.21.15.38:0/2190641517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a24ab482c0 con 0x55a24ac18670 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62daffd640 1 -- 172.21.15.38:0/2401235913 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f62d40cf0b0 con 0x7f62e4100fd0 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:39.999+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009900 tx=0x7f62e0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f523adc02434f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ea514640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 0x7f62d00a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009fd0 tx=0x7f62dc004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:40.003 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62cffff640 1 -- 172.21.15.38:0/2190641517 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e002ca70 con 0x55a24ac18670 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62cffff640 1 -- 172.21.15.38:0/2190641517 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e002cc10 con 0x55a24ac18670 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62cffff640 1 -- 172.21.15.38:0/2190641517 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0005130 con 0x55a24ac18670 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2190641517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 msgr2=0x55a24ac253f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009900 tx=0x7f62e0005dd0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2190641517 shutdown_connections 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac27d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac253f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2190641517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac17d50 0x55a24ac18130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2190641517 >> 172.21.15.38:0/2190641517 conn(0x55a24ac12750 msgr2=0x55a24ac12b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2190641517 shutdown_connections 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.000+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2190641517 wait complete. 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 -- start start 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 Processor -- start 2024-04-22T00:08:40.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac25230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 0x55a24ac22b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a24ac28690 con 0x55a24ac18670 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a24ab37f10 con 0x55a24ac25930 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a24ac28810 con 0x55a24ac17d50 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ee7fd740 10 client.0 _ll_register_callbacks cb 0x55a24a88fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50652/0 (socket says 172.21.15.38:50652) 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 -- 172.21.15.38:0/2147022790 learned_addr learned my addr 172.21.15.38:0/2147022790 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac25230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62ea514640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 0x55a24ac22b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 -- 172.21.15.38:0/2147022790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 msgr2=0x55a24ac25230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac25230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.001+0000 7f62e9512640 1 -- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 msgr2=0x55a24ac22b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62e9512640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 0x55a24ac22b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62e9512640 1 -- 172.21.15.38:0/2147022790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a24ab09c80 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ea514640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 0x55a24ac22b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62e9512640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f62d409b220 tx=0x7f62d409aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6338cc08beff93a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d409b990 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f62d40a3a70 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62e002cc10 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f62d40b4050 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ee7fd740 10 client.17850 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a24a8903f0 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e002cc10 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f62d40bb0b0 con 0x55a24ac18670 2024-04-22T00:08:40.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ee7fd740 10 client.17850 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.002+0000 7f62ee7fd740 10 client.17850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62ee7fd740 20 client.17850 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24ac35920 0x55a24ac37d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62ee7fd740 4 client.17850 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a24ab89a70 con 0x55a24ac35920 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24ac35920 0x55a24ac37d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62cd7fa640 10 client.17850 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.003+0000 7f62e9d13640 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24ac35920 0x55a24ac37d30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ddde84c700b77a server_cookie=f712adf07a6c2897 in_seq=0 out_seq=0 2024-04-22T00:08:40.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:39 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3011712297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:08:40.113 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.117+0000 7fb2c8ff9640 1 -- 172.21.15.38:0/910436109 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fb2d800ce30 con 0x5602e7bd18a0 2024-04-22T00:08:40.114 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.117+0000 7fb2c8ff9640 10 client.27025 handle_command_reply: tid=0 2024-04-22T00:08:40.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.117+0000 7f62cd7fa640 1 -- 172.21.15.38:0/2147022790 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a24ab89a70 con 0x55a24ac35920 2024-04-22T00:08:40.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.117+0000 7f62cd7fa640 10 client.17850 handle_command_reply: tid=0 2024-04-22T00:08:40.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.117+0000 7f62ee7fd740 1 client.17850 shutdown 2024-04-22T00:08:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.118+0000 7fb2ea60d740 1 client.27025 shutdown 2024-04-22T00:08:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.118+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 msgr2=0x5602e7b56610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.115 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.118+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b56610 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2d809bae0 tx=0x7fb2d80a1590 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 msgr2=0x55a24ac225c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f62d409b220 tx=0x7f62d409aef0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 shutdown_connections 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac25930 0x55a24ac22b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24ac35920 0x55a24ac37d30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac18670 0x55a24ac225c0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/2147022790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac25230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 >> 172.21.15.38:0/2147022790 conn(0x55a24ac12750 msgr2=0x55a24ac144a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62cd7fa640 0 client.17850 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 shutdown_connections 2024-04-22T00:08:40.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 1 -- 172.21.15.38:0/2147022790 wait complete. 2024-04-22T00:08:40.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.118+0000 7f62ee7fd740 20 client.17850 trim_cache size 0 max 16384 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 shutdown_connections 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5602e7b52210 0x5602e7b56610 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5602e7bd18a0 0x5602e7bd3c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602e7b49440 0x5602e7b560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 --2- 172.21.15.38:0/910436109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5602e7b48b20 0x5602e7b55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.117 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2c8ff9640 0 client.27025 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 >> 172.21.15.38:0/910436109 conn(0x5602e7b43440 msgr2=0x5602e7b4f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 shutdown_connections 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 1 -- 172.21.15.38:0/910436109 wait complete. 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.119+0000 7fb2ea60d740 20 client.27025 trim_cache size 0 max 16384 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:08:40.118 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:08:40.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:08:40.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.120+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 msgr2=0x7fb2cc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.120+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 0x7fb2cc0a5260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002c180 tx=0x7fb2d002e040 comp rx=0 tx=0).stop 2024-04-22T00:08:40.119 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.120+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 msgr2=0x7fb2e019c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.120+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009fd0 tx=0x7fb2d800c800 comp rx=0 tx=0).stop 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 shutdown_connections 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb2cc0a2da0 0x7fb2cc0a5260 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2e0106630 0x7fb2e019c7e0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2e0101030 0x7fb2e01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 --2- 172.21.15.38:0/2795832251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e0100670 0x7fb2e0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 >> 172.21.15.38:0/2795832251 conn(0x7fb2e00777c0 msgr2=0x7fb2e00ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 shutdown_connections 2024-04-22T00:08:40.120 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:40.121+0000 7fb2e7a27640 1 -- 172.21.15.38:0/2795832251 wait complete. 2024-04-22T00:08:40.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.150+0000 7f62ee7fd740 1 Processor -- start 2024-04-22T00:08:40.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 -- start start 2024-04-22T00:08:40.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac24cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac25930 0x55a24ac25230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41ac830 con 0x55a24ac25930 2024-04-22T00:08:40.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41ae370 con 0x55a24ac18670 2024-04-22T00:08:40.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.151+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41acf00 con 0x55a24ac17d50 2024-04-22T00:08:40.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34816/0 (socket says 172.21.15.38:34816) 2024-04-22T00:08:40.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 -- 172.21.15.38:0/949136114 learned_addr learned my addr 172.21.15.38:0/949136114 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62ea514640 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac25930 0x55a24ac25230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 -- 172.21.15.38:0/949136114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 msgr2=0x55a24ac24cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac24cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 -- 172.21.15.38:0/949136114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac25930 msgr2=0x55a24ac25230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac25930 0x55a24ac25230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 -- 172.21.15.38:0/949136114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a24a95a7d0 con 0x55a24ac17d50 2024-04-22T00:08:40.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.157+0000 7f62e9d13640 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f62e0005ec0 tx=0x7f62e002bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4e2053f520a081 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:40.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.158+0000 7f62cf7fe640 1 -- 172.21.15.38:0/949136114 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0004050 con 0x55a24ac17d50 2024-04-22T00:08:40.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.158+0000 7f62cf7fe640 1 -- 172.21.15.38:0/949136114 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e00041f0 con 0x55a24ac17d50 2024-04-22T00:08:40.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.159+0000 7f62ee7fd740 1 -- 172.21.15.38:0/949136114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 msgr2=0x55a24ac31590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.159+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f62e0005ec0 tx=0x7f62e002bc20 comp rx=0 tx=0).stop 2024-04-22T00:08:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.163+0000 7f62ee7fd740 1 -- 172.21.15.38:0/949136114 shutdown_connections 2024-04-22T00:08:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.163+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac25930 0x55a24ac25230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.163+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac24cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.163+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/949136114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac17d50 0x55a24ac31590 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.163+0000 7f62ee7fd740 1 -- 172.21.15.38:0/949136114 >> 172.21.15.38:0/949136114 conn(0x55a24ac12750 msgr2=0x55a24ac1dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.164+0000 7f62ee7fd740 1 -- 172.21.15.38:0/949136114 shutdown_connections 2024-04-22T00:08:40.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.164+0000 7f62ee7fd740 1 -- 172.21.15.38:0/949136114 wait complete. 2024-04-22T00:08:40.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.164+0000 7f62ee7fd740 1 -- start start 2024-04-22T00:08:40.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.165+0000 7f62ee7fd740 1 Processor -- start 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac17d50 0x55a24ac1a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac1a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41adca0 con 0x55a24ac17d50 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41ad5d0 con 0x55a24ac18670 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41aa810 con 0x55a24ac25930 2024-04-22T00:08:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.169+0000 7f62ee7fd740 10 client.0 _ll_register_callbacks cb 0x7f62d4004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.172+0000 7f62ea514640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.172+0000 7f62ea514640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34822/0 (socket says 172.21.15.38:34822) 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.172+0000 7f62ea514640 1 -- 172.21.15.38:0/4158976890 learned_addr learned my addr 172.21.15.38:0/4158976890 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.173+0000 7f62ea514640 1 -- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 msgr2=0x55a24ac1a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.173+0000 7f62ea514640 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac1a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.173+0000 7f62ea514640 1 -- 172.21.15.38:0/4158976890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac17d50 msgr2=0x55a24ac1a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.173+0000 7f62ea514640 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac17d50 0x55a24ac1a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.173+0000 7f62ea514640 1 -- 172.21.15.38:0/4158976890 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a24ac2cb10 con 0x55a24ac25930 2024-04-22T00:08:40.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.177+0000 7f62ea514640 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62dc005fd0 tx=0x7f62dc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4085824cfb3097 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:40.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.178+0000 7f62cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:08:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.184+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc004b20 con 0x55a24ac25930 2024-04-22T00:08:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.184+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f62dc004cc0 con 0x55a24ac25930 2024-04-22T00:08:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.185+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a24ac22e90 con 0x55a24ac25930 2024-04-22T00:08:40.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.185+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f62dc005020 con 0x55a24ac25930 2024-04-22T00:08:40.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.187+0000 7f62ee7fd740 10 client.27043 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:40.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.187+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a24ac2de60 con 0x55a24ac25930 2024-04-22T00:08:40.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a24ac22e90 con 0x55a24ac25930 2024-04-22T00:08:40.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f62dc02a070 con 0x55a24ac25930 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 10 client.27043 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 10 client.27043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 20 client.27043 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24acb1780 0x55a24acb3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 4 client.27043 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.192+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a24ab83dc0 con 0x55a24acb1780 2024-04-22T00:08:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.193+0000 7f62e9d13640 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24acb1780 0x55a24acb3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:40.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.197+0000 7f62e9d13640 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24acb1780 0x55a24acb3bd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc72d43ebbd065b4 server_cookie=d2a441aff11b756e in_seq=0 out_seq=0 2024-04-22T00:08:40.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.199+0000 7f62cd7fa640 10 client.27043 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:40.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.199+0000 7f62cd7fa640 1 -- 172.21.15.38:0/4158976890 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a24ab83dc0 con 0x55a24acb1780 2024-04-22T00:08:40.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.199+0000 7f62cd7fa640 10 client.27043 handle_command_reply: tid=0 2024-04-22T00:08:40.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.199+0000 7f62ee7fd740 1 client.27043 shutdown 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.200+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 msgr2=0x55a24ac1adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.200+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62dc005fd0 tx=0x7f62dc004a40 comp rx=0 tx=0).stop 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 shutdown_connections 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a24ac25930 0x55a24ac1adb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a24ac18670 0x55a24ac1a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a24acb1780 0x55a24acb3bd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 --2- 172.21.15.38:0/4158976890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a24ac17d50 0x55a24ac1a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 >> 172.21.15.38:0/4158976890 conn(0x55a24ac12750 msgr2=0x55a24ac13750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.201+0000 7f62cd7fa640 0 client.27043 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:40.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.203+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 shutdown_connections 2024-04-22T00:08:40.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.207+0000 7f62ee7fd740 1 -- 172.21.15.38:0/4158976890 wait complete. 2024-04-22T00:08:40.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.207+0000 7f62ee7fd740 20 client.27043 trim_cache size 0 max 16384 2024-04-22T00:08:40.205 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 msgr2=0x7f62d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 0x7f62d00a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009fd0 tx=0x7f62dc004ab0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 msgr2=0x7f62e41992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f62d400d950 tx=0x7f62d400de10 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 shutdown_connections 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f62d00a2d00 0x7f62d00a51c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e41066b0 0x7f62e41a3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62e4100fd0 0x7f62e41992b0 secure :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f62d400d950 tx=0x7f62d400de10 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 --2- 172.21.15.38:0/2401235913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62e4100610 0x7f62e4198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:40.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.210+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 >> 172.21.15.38:0/2401235913 conn(0x7f62e4077740 msgr2=0x7f62e40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:40.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.212+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 shutdown_connections 2024-04-22T00:08:40.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:40.214+0000 7f62ebf77640 1 -- 172.21.15.38:0/2401235913 wait complete. 2024-04-22T00:08:40.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:40 smithi111 ceph-mon[31543]: pgmap v200: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-04-22T00:08:40.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:40 smithi038 ceph-mon[27972]: pgmap v200: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-04-22T00:08:41.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:40 smithi203 ceph-mon[31431]: pgmap v200: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-04-22T00:08:41.234 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:08:41.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:08:41.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:08:41.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.423+0000 7f0e62a8f640 1 -- 172.21.15.38:0/2040799624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c106630 msgr2=0x7f0e5c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.423+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/2040799624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c106630 0x7f0e5c106a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009cd0 tx=0x7f0e5402c0e0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.425+0000 7f0e62a8f640 1 -- 172.21.15.38:0/2040799624 shutdown_connections 2024-04-22T00:08:41.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.425+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/2040799624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.425+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/2040799624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c100620 0x7f0e5c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.426+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/2040799624 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c106630 0x7f0e5c106a10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.426+0000 7f0e62a8f640 1 -- 172.21.15.38:0/2040799624 >> 172.21.15.38:0/2040799624 conn(0x7f0e5c0769c0 msgr2=0x7f0e5c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.430+0000 7f0e62a8f640 1 -- 172.21.15.38:0/2040799624 shutdown_connections 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.431+0000 7f0e62a8f640 1 -- 172.21.15.38:0/2040799624 wait complete. 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.432+0000 7f0e62a8f640 1 Processor -- start 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.432+0000 7f0e62a8f640 1 -- start start 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100620 0x7f0e5c073000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 0x7f0e5c06c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c10fa00 con 0x7f0e5c100620 2024-04-22T00:08:41.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c10f880 con 0x7f0e5c106630 2024-04-22T00:08:41.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.433+0000 7f0e62a8f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c10fb80 con 0x7f0e5c100fe0 2024-04-22T00:08:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.435+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.435+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34828/0 (socket says 172.21.15.38:34828) 2024-04-22T00:08:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.435+0000 7f0e5bfff640 1 -- 172.21.15.38:0/993428738 learned_addr learned my addr 172.21.15.38:0/993428738 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.435+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 0x7f0e5c06c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.435+0000 7f0e6082b640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100620 0x7f0e5c073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 -- 172.21.15.38:0/993428738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 msgr2=0x7f0e5c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c06c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 -- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100620 msgr2=0x7f0e5c073000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100620 0x7f0e5c073000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 -- 172.21.15.38:0/993428738 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c06cc30 con 0x7f0e5c106630 2024-04-22T00:08:41.433 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.436+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 0x7f0e5c06c5c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000b410 tx=0x7f0e5000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.438+0000 7f0e59ffb640 1 -- 172.21.15.38:0/993428738 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50015020 con 0x7f0e5c106630 2024-04-22T00:08:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.438+0000 7f0e59ffb640 1 -- 172.21.15.38:0/993428738 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e50011d10 con 0x7f0e5c106630 2024-04-22T00:08:41.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.438+0000 7f0e59ffb640 1 -- 172.21.15.38:0/993428738 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5000f2e0 con 0x7f0e5c106630 2024-04-22T00:08:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.440+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c070b20 con 0x7f0e5c106630 2024-04-22T00:08:41.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.440+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c071020 con 0x7f0e5c106630 2024-04-22T00:08:41.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.447+0000 7f0e59ffb640 1 -- 172.21.15.38:0/993428738 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0e50018020 con 0x7f0e5c106630 2024-04-22T00:08:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.452+0000 7f0e6567e740 1 Processor -- start 2024-04-22T00:08:41.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 -- start start 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383e09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562238330170 con 0x5622383de5c0 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622382d87c0 con 0x5622383d5f30 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e6567e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56223832c130 con 0x5622383d5250 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e59ffb640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 0x7f0e440a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.453+0000 7f0e59ffb640 1 -- 172.21.15.38:0/993428738 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e500cee10 con 0x7f0e5c106630 2024-04-22T00:08:41.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.454+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 0x7f0e440a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.457+0000 7f0e6102c640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 0x7f0e440a5350 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0e500cc330 tx=0x7f0e500046c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.458+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.459+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34842/0 (socket says 172.21.15.38:34842) 2024-04-22T00:08:41.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.459+0000 7f0e6082b640 1 -- 172.21.15.38:0/3175775314 learned_addr learned my addr 172.21.15.38:0/3175775314 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.461+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.461+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383e09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.464+0000 7f0e6082b640 1 -- 172.21.15.38:0/3175775314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 msgr2=0x5622383d4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.464+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383d4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.464+0000 7f0e6082b640 1 -- 172.21.15.38:0/3175775314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 msgr2=0x5622383e09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.464+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383e09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.464+0000 7f0e6082b640 1 -- 172.21.15.38:0/3175775314 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622382e9e60 con 0x5622383d5250 2024-04-22T00:08:41.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.468+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402c670 tx=0x7f0e540324d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d1e0d972ba9612 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.472+0000 7f0e42ffd640 1 -- 172.21.15.38:0/3175775314 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54043020 con 0x5622383d5250 2024-04-22T00:08:41.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.472+0000 7f0e42ffd640 1 -- 172.21.15.38:0/3175775314 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e54032e70 con 0x5622383d5250 2024-04-22T00:08:41.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.473+0000 7f0e6567e740 1 -- 172.21.15.38:0/3175775314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 msgr2=0x5622383d56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.473+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402c670 tx=0x7f0e540324d0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.480+0000 7f0e6567e740 1 -- 172.21.15.38:0/3175775314 shutdown_connections 2024-04-22T00:08:41.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.480+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383e09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.480+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383d56d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.480+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3175775314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383d4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.480+0000 7f0e6567e740 1 -- 172.21.15.38:0/3175775314 >> 172.21.15.38:0/3175775314 conn(0x5622383934f0 msgr2=0x5622383cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.481+0000 7f0e6567e740 1 -- 172.21.15.38:0/3175775314 shutdown_connections 2024-04-22T00:08:41.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.485+0000 7f0e6567e740 1 -- 172.21.15.38:0/3175775314 wait complete. 2024-04-22T00:08:41.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.485+0000 7f0e6567e740 1 -- start start 2024-04-22T00:08:41.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.485+0000 7f0e6567e740 1 Processor -- start 2024-04-22T00:08:41.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5f30 0x5622383dbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383de5c0 0x5622383dc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5622383c7150 con 0x5622383d5f30 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622382ebd40 con 0x5622383d5250 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5622383c72d0 con 0x5622383de5c0 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.487+0000 7f0e6567e740 10 client.0 _ll_register_callbacks cb 0x562238327820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.488+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.488+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51854/0 (socket says 172.21.15.38:51854) 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.488+0000 7f0e5bfff640 1 -- 172.21.15.38:0/1761733059 learned_addr learned my addr 172.21.15.38:0/1761733059 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.488+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383de5c0 0x5622383dc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.488+0000 7f0e5bfff640 1 -- 172.21.15.38:0/1761733059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383de5c0 msgr2=0x5622383dc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.489+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383de5c0 0x5622383dc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.489+0000 7f0e5bfff640 1 -- 172.21.15.38:0/1761733059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5f30 msgr2=0x5622383dbb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:08:41.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.489+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5f30 0x5622383dbb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.489+0000 7f0e5bfff640 1 -- 172.21.15.38:0/1761733059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56223810c7e0 con 0x5622383d5250 2024-04-22T00:08:41.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.490+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c018d90 tx=0x7f0e4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a5e1a93c6cdc77b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.490+0000 7f0e3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.490+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c013e40 con 0x5622383d5250 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.491+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562238304450 con 0x5622383d5250 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.491+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e4c022380 con 0x5622383d5250 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.491+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0e4c02a050 con 0x5622383d5250 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.491+0000 7f0e6567e740 10 client.27194 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:41.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.491+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56223816cf70 con 0x5622383d5250 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562238304450 con 0x5622383d5250 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0e4c0155d0 con 0x5622383d5250 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 10 client.27194 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 10 client.27194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 20 client.27194 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622383f7310 0x5622383f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 4 client.27194 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:41.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.492+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56223833d6b0 con 0x5622383f7310 2024-04-22T00:08:41.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.495+0000 7f0e6082b640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622383f7310 0x5622383f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.497+0000 7f0e6082b640 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622383f7310 0x5622383f97b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7a215e5864d902 server_cookie=a0edc2f8af823798 in_seq=0 out_seq=0 2024-04-22T00:08:41.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.497+0000 7f0e3bfff640 10 client.27194 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.499+0000 7f0e3bfff640 1 -- 172.21.15.38:0/1761733059 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56223833d6b0 con 0x5622383f7310 2024-04-22T00:08:41.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.499+0000 7f0e3bfff640 10 client.27194 handle_command_reply: tid=0 2024-04-22T00:08:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.499+0000 7f0e6567e740 1 client.27194 shutdown 2024-04-22T00:08:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 msgr2=0x5622383db640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c018d90 tx=0x7f0e4c0046c0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 shutdown_connections 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383de5c0 0x5622383dc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622383f7310 0x5622383f97b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5f30 0x5622383dbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 --2- 172.21.15.38:0/1761733059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5250 0x5622383db640 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 >> 172.21.15.38:0/1761733059 conn(0x5622383934f0 msgr2=0x5622383d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.501+0000 7f0e3bfff640 0 client.27194 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.505+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 shutdown_connections 2024-04-22T00:08:41.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.506+0000 7f0e6567e740 1 -- 172.21.15.38:0/1761733059 wait complete. 2024-04-22T00:08:41.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.509+0000 7f0e6567e740 20 client.27194 trim_cache size 0 max 16384 2024-04-22T00:08:41.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 Processor -- start 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 -- start start 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5250 0x5622383dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383de5c0 0x5622383ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1a42c0 con 0x5622383d5250 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1c7ef0 con 0x5622383de5c0 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.544+0000 7f0e6567e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1a4990 con 0x5622383d5f30 2024-04-22T00:08:41.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:34870/0 (socket says 172.21.15.38:34870) 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 -- 172.21.15.38:0/3007594910 learned_addr learned my addr 172.21.15.38:0/3007594910 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5250 0x5622383dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 -- 172.21.15.38:0/3007594910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383de5c0 msgr2=0x5622383ff5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383de5c0 0x5622383ff5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 -- 172.21.15.38:0/3007594910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5250 msgr2=0x5622383dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5250 0x5622383dbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 -- 172.21.15.38:0/3007594910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562238004170 con 0x5622383d5f30 2024-04-22T00:08:41.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.545+0000 7f0e6082b640 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e54003eb0 tx=0x7f0e540324d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3cd816582baf39 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.546+0000 7f0e427fc640 1 -- 172.21.15.38:0/3007594910 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54052070 con 0x5622383d5f30 2024-04-22T00:08:41.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.547+0000 7f0e427fc640 1 -- 172.21.15.38:0/3007594910 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e540047d0 con 0x5622383d5f30 2024-04-22T00:08:41.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.547+0000 7f0e427fc640 1 -- 172.21.15.38:0/3007594910 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54043040 con 0x5622383d5f30 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.547+0000 7f0e6567e740 1 -- 172.21.15.38:0/3007594910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 msgr2=0x5622383ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.547+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0e54003eb0 tx=0x7f0e540324d0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 -- 172.21.15.38:0/3007594910 shutdown_connections 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383de5c0 0x5622383ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5f30 0x5622383ff080 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 --2- 172.21.15.38:0/3007594910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383d5250 0x5622383dbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 -- 172.21.15.38:0/3007594910 >> 172.21.15.38:0/3007594910 conn(0x5622383934f0 msgr2=0x5622383c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 -- 172.21.15.38:0/3007594910 shutdown_connections 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 -- 172.21.15.38:0/3007594910 wait complete. 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 -- start start 2024-04-22T00:08:41.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.548+0000 7f0e6567e740 1 Processor -- start 2024-04-22T00:08:41.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1a5730 con 0x5622383de5c0 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c1a5060 con 0x5622383d5f30 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c0689b0 con 0x5622383d5250 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.549+0000 7f0e6567e740 10 client.0 _ll_register_callbacks cb 0x7f0e4c009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51868/0 (socket says 172.21.15.38:51868) 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 -- 172.21.15.38:0/2120262681 learned_addr learned my addr 172.21.15.38:0/2120262681 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 -- 172.21.15.38:0/2120262681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 msgr2=0x5622383cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 -- 172.21.15.38:0/2120262681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 msgr2=0x5622383c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383c7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 -- 172.21.15.38:0/2120262681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622383c7cb0 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.550+0000 7f0e6082b640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402f020 tx=0x7f0e54005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3671d361fc4df6a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54052040 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e540047d0 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f0e54032870 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0e540042d0 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e6567e740 10 client.27197 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5622383ccfa0 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.551+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e540047d0 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0e54004470 con 0x5622383d5f30 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 10 client.27197 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:41.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 10 client.27197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 20 client.27197 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56223845d8f0 0x56223845fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 4 client.27197 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.552+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562238337a00 con 0x56223845d8f0 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.554+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56223845d8f0 0x56223845fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.554+0000 7f0e3bfff640 10 client.27197 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.554+0000 7f0e5bfff640 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56223845d8f0 0x56223845fce0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dff96d0d610e89d server_cookie=e930d5046522a222 in_seq=0 out_seq=0 2024-04-22T00:08:41.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e3bfff640 1 -- 172.21.15.38:0/2120262681 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562238337a00 con 0x56223845d8f0 2024-04-22T00:08:41.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e3bfff640 10 client.27197 handle_command_reply: tid=0 2024-04-22T00:08:41.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 client.27197 shutdown 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 msgr2=0x5622383cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402f020 tx=0x7f0e54005cf0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 shutdown_connections 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56223845d8f0 0x56223845fce0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622383de5c0 0x5622383c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622383d5f30 0x5622383cde90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 --2- 172.21.15.38:0/2120262681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622383d5250 0x5622383cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.556+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 >> 172.21.15.38:0/2120262681 conn(0x5622383934f0 msgr2=0x5622383d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.557+0000 7f0e3bfff640 0 client.27197 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.557+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 shutdown_connections 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.557+0000 7f0e6567e740 1 -- 172.21.15.38:0/2120262681 wait complete. 2024-04-22T00:08:41.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.557+0000 7f0e6567e740 20 client.27197 trim_cache size 0 max 16384 2024-04-22T00:08:41.554 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:41.554 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:08:41.554 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "5dc84a98-613d-4b45-b556-45a2fde8cc15", 2024-04-22T00:08:41.554 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:08:41.554 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 msgr2=0x7f0e440a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 0x7f0e440a5350 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0e500cc330 tx=0x7f0e500046c0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 msgr2=0x7f0e5c06c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 0x7f0e5c06c5c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000b410 tx=0x7f0e5000b8d0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 shutdown_connections 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0e440a2e90 0x7f0e440a5350 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.558+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e5c106630 0x7f0e5c06c5c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.559+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e5c100fe0 0x7f0e5c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.559+0000 7f0e62a8f640 1 --2- 172.21.15.38:0/993428738 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e5c100620 0x7f0e5c073000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.559+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 >> 172.21.15.38:0/993428738 conn(0x7f0e5c0769c0 msgr2=0x7f0e5c0fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.559+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 shutdown_connections 2024-04-22T00:08:41.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.559+0000 7f0e62a8f640 1 -- 172.21.15.38:0/993428738 wait complete. 2024-04-22T00:08:41.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:08:41.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:08:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.746+0000 7f673effd640 1 -- 172.21.15.38:0/3022832509 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673002cc80 con 0x7f6740106630 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 -- 172.21.15.38:0/3022832509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740106630 msgr2=0x7f6740106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 --2- 172.21.15.38:0/3022832509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740106630 0x7f6740106a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6730009930 tx=0x7f673002bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 -- 172.21.15.38:0/3022832509 shutdown_connections 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 --2- 172.21.15.38:0/3022832509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 0x7f674010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 --2- 172.21.15.38:0/3022832509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740100620 0x7f6740100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 --2- 172.21.15.38:0/3022832509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740106630 0x7f6740106a10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 -- 172.21.15.38:0/3022832509 >> 172.21.15.38:0/3022832509 conn(0x7f6740077770 msgr2=0x7f6740077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.747+0000 7f6745f5b640 1 -- 172.21.15.38:0/3022832509 shutdown_connections 2024-04-22T00:08:41.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.748+0000 7f6745f5b640 1 -- 172.21.15.38:0/3022832509 wait complete. 2024-04-22T00:08:41.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 Processor -- start 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 -- start start 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740100620 0x7f6740072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 0x7f6740073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f674006b730 con 0x7f6740106630 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f674006b5b0 con 0x7f6740100fe0 2024-04-22T00:08:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.749+0000 7f6745f5b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f674006b8b0 con 0x7f6740100620 2024-04-22T00:08:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.750+0000 7f673f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740100620 0x7f6740072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.750+0000 7f673ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.750+0000 7f673ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50722/0 (socket says 172.21.15.38:50722) 2024-04-22T00:08:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.750+0000 7f673ffff640 1 -- 172.21.15.38:0/4269450026 learned_addr learned my addr 172.21.15.38:0/4269450026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.750+0000 7f6737fff640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 0x7f6740073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.751+0000 7f673ffff640 1 -- 172.21.15.38:0/4269450026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740100620 msgr2=0x7f6740072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.751+0000 7f673ffff640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740100620 0x7f6740072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.751+0000 7f673ffff640 1 -- 172.21.15.38:0/4269450026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 msgr2=0x7f6740073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.751+0000 7f673ffff640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 0x7f6740073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.751+0000 7f673ffff640 1 -- 172.21.15.38:0/4269450026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674006d460 con 0x7f6740106630 2024-04-22T00:08:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.752+0000 7f673ffff640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f673800eb10 tx=0x7f673800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.752+0000 7f673d7fa640 1 -- 172.21.15.38:0/4269450026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738004b20 con 0x7f6740106630 2024-04-22T00:08:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.753+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674006d750 con 0x7f6740106630 2024-04-22T00:08:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.753+0000 7f673d7fa640 1 -- 172.21.15.38:0/4269450026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6738004cc0 con 0x7f6740106630 2024-04-22T00:08:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.753+0000 7f673d7fa640 1 -- 172.21.15.38:0/4269450026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738010790 con 0x7f6740106630 2024-04-22T00:08:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.753+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67401b3910 con 0x7f6740106630 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 Processor -- start 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f673d7fa640 1 -- 172.21.15.38:0/4269450026 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f6738005000 con 0x7f6740106630 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 -- start start 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc089ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc08c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a5fbfe5560 con 0x55a5fc081850 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5fbf8d730 con 0x55a5fc080f30 2024-04-22T00:08:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.756+0000 7f6748c26740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a5fbfe1520 con 0x55a5fc08a430 2024-04-22T00:08:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51890/0 (socket says 172.21.15.38:51890) 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 -- 172.21.15.38:0/3056521989 learned_addr learned my addr 172.21.15.38:0/3056521989 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f673f7fe640 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc089ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc08c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 -- 172.21.15.38:0/3056521989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 msgr2=0x55a5fc08c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc08c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 -- 172.21.15.38:0/3056521989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 msgr2=0x55a5fc089ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc089ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 -- 172.21.15.38:0/3056521989 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5fbdc37d0 con 0x55a5fc080f30 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f673d7fa640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 0x7f672c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.757+0000 7f6737fff640 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f672800b3c0 tx=0x7f6728005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9d7c3cf0194423e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.758+0000 7f6735ffb640 1 -- 172.21.15.38:0/3056521989 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728005220 con 0x55a5fc080f30 2024-04-22T00:08:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.758+0000 7f673d7fa640 1 -- 172.21.15.38:0/4269450026 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f67380cfe50 con 0x7f6740106630 2024-04-22T00:08:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.758+0000 7f673f7fe640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 0x7f672c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.758+0000 7f6735ffb640 1 -- 172.21.15.38:0/3056521989 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728004c80 con 0x55a5fc080f30 2024-04-22T00:08:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.758+0000 7f6735ffb640 1 -- 172.21.15.38:0/3056521989 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672801b810 con 0x55a5fc080f30 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 -- 172.21.15.38:0/3056521989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 msgr2=0x55a5fc081310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f673f7fe640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 0x7f672c0a5300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f673002c240 tx=0x7f673002df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f672800b3c0 tx=0x7f6728005f90 comp rx=0 tx=0).stop 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 -- 172.21.15.38:0/3056521989 shutdown_connections 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc08c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc089ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 --2- 172.21.15.38:0/3056521989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc081310 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 -- 172.21.15.38:0/3056521989 >> 172.21.15.38:0/3056521989 conn(0x55a5fc07b660 msgr2=0x55a5fc07ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 -- 172.21.15.38:0/3056521989 shutdown_connections 2024-04-22T00:08:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.759+0000 7f6748c26740 1 -- 172.21.15.38:0/3056521989 wait complete. 2024-04-22T00:08:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 -- start start 2024-04-22T00:08:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 Processor -- start 2024-04-22T00:08:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc080f30 0x55a5fc087480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc0879c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a5fc08d160 con 0x55a5fc081850 2024-04-22T00:08:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a5fbfa0e00 con 0x55a5fc08a430 2024-04-22T00:08:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a5fc08d2e0 con 0x55a5fc080f30 2024-04-22T00:08:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f6748c26740 10 client.0 _ll_register_callbacks cb 0x55a5fbfdbeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f673f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc080f30 0x55a5fc087480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.760+0000 7f673ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51902/0 (socket says 172.21.15.38:51902) 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 -- 172.21.15.38:0/3781540199 learned_addr learned my addr 172.21.15.38:0/3781540199 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f6737fff640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc0879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 -- 172.21.15.38:0/3781540199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc080f30 msgr2=0x55a5fc087480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc080f30 0x55a5fc087480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 -- 172.21.15.38:0/3781540199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 msgr2=0x55a5fc0879c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc0879c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 -- 172.21.15.38:0/3781540199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5fbcf93f0 con 0x55a5fc08a430 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f673ffff640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67380989a0 tx=0x7f673809bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2568f35b8518044 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.761+0000 7f671effd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.762+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67380cd690 con 0x55a5fc08a430 2024-04-22T00:08:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.762+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f67380bc070 con 0x55a5fc08a430 2024-04-22T00:08:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.762+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728004c80 con 0x55a5fc08a430 2024-04-22T00:08:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f67380a47b0 con 0x55a5fc08a430 2024-04-22T00:08:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f6748c26740 10 client.27206 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a5fbcb90f0 con 0x55a5fc08a430 2024-04-22T00:08:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728004c80 con 0x55a5fc08a430 2024-04-22T00:08:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f673809b420 con 0x55a5fc08a430 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f6748c26740 10 client.27206 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.763+0000 7f6748c26740 10 client.27206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.764+0000 7f6748c26740 20 client.27206 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.764+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc0bc800 0x55a5fc0bec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.764+0000 7f6748c26740 4 client.27206 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.764+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5fbff2aa0 con 0x55a5fc0bc800 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.764+0000 7f673f7fe640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc0bc800 0x55a5fc0bec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.766+0000 7f671effd640 10 client.27206 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.766+0000 7f673f7fe640 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc0bc800 0x55a5fc0bec50 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ca2857f514c2a90 server_cookie=f3e08bf0e68892f5 in_seq=0 out_seq=0 2024-04-22T00:08:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.766+0000 7f671effd640 1 -- 172.21.15.38:0/3781540199 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5fbff2aa0 con 0x55a5fc0bc800 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.766+0000 7f671effd640 10 client.27206 handle_command_reply: tid=0 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.766+0000 7f6748c26740 1 client.27206 shutdown 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 msgr2=0x55a5fc087f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67380989a0 tx=0x7f673809bfd0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 shutdown_connections 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc08a430 0x55a5fc087f00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc0bc800 0x55a5fc0bec50 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc0879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 --2- 172.21.15.38:0/3781540199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc080f30 0x55a5fc087480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 >> 172.21.15.38:0/3781540199 conn(0x55a5fc07b660 msgr2=0x55a5fc07d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f671effd640 0 client.27206 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 shutdown_connections 2024-04-22T00:08:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 1 -- 172.21.15.38:0/3781540199 wait complete. 2024-04-22T00:08:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.767+0000 7f6748c26740 20 client.27206 trim_cache size 0 max 16384 2024-04-22T00:08:41.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.800+0000 7f6748c26740 1 Processor -- start 2024-04-22T00:08:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.801+0000 7f6748c26740 1 -- start start 2024-04-22T00:08:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.801+0000 7f6748c26740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc080f30 0x55a5fc096050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.801+0000 7f6748c26740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.801+0000 7f6748c26740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc07d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6748c26740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6740119070 con 0x55a5fc080f30 2024-04-22T00:08:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6748c26740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f674011abb0 con 0x55a5fc081850 2024-04-22T00:08:41.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6748c26740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6740119740 con 0x55a5fc08a430 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6737fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6737fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:51916/0 (socket says 172.21.15.38:51916) 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f6737fff640 1 -- 172.21.15.38:0/840957175 learned_addr learned my addr 172.21.15.38:0/840957175 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.802+0000 7f673ffff640 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc07d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.803+0000 7f6737fff640 1 -- 172.21.15.38:0/840957175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 msgr2=0x55a5fc07d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.803+0000 7f6737fff640 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc07d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.803+0000 7f6737fff640 1 -- 172.21.15.38:0/840957175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc080f30 msgr2=0x55a5fc096050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.803+0000 7f6737fff640 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc080f30 0x55a5fc096050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.803+0000 7f6737fff640 1 -- 172.21.15.38:0/840957175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5fc0822a0 con 0x55a5fc081850 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.805+0000 7f6737fff640 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6728018690 tx=0x7f672800bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b029732221ee975 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.806+0000 7f67357fa640 1 -- 172.21.15.38:0/840957175 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728029070 con 0x55a5fc081850 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.806+0000 7f67357fa640 1 -- 172.21.15.38:0/840957175 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728004620 con 0x55a5fc081850 2024-04-22T00:08:41.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.806+0000 7f67357fa640 1 -- 172.21.15.38:0/840957175 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672801f040 con 0x55a5fc081850 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.808+0000 7f6748c26740 1 -- 172.21.15.38:0/840957175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 msgr2=0x55a5fc087d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.808+0000 7f6748c26740 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6728018690 tx=0x7f672800bc20 comp rx=0 tx=0).stop 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.810+0000 7f6748c26740 1 -- 172.21.15.38:0/840957175 shutdown_connections 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc07d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc081850 0x55a5fc087d20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 --2- 172.21.15.38:0/840957175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc080f30 0x55a5fc096050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 -- 172.21.15.38:0/840957175 >> 172.21.15.38:0/840957175 conn(0x55a5fc07b660 msgr2=0x55a5fc07ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 -- 172.21.15.38:0/840957175 shutdown_connections 2024-04-22T00:08:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.811+0000 7f6748c26740 1 -- 172.21.15.38:0/840957175 wait complete. 2024-04-22T00:08:41.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.812+0000 7f6748c26740 1 -- start start 2024-04-22T00:08:41.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.812+0000 7f6748c26740 1 Processor -- start 2024-04-22T00:08:41.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.812+0000 7f6748c26740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc092750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc083cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc084230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f674011a4e0 con 0x55a5fc081850 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f6740119e10 con 0x55a5fc080f30 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6740067cd0 con 0x55a5fc08a430 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6748c26740 10 client.0 _ll_register_callbacks cb 0x7f6738098270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc083cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6737fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc083cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50756/0 (socket says 172.21.15.38:50756) 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f6737fff640 1 -- 172.21.15.38:0/885825452 learned_addr learned my addr 172.21.15.38:0/885825452 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:41.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc092750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 -- 172.21.15.38:0/885825452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 msgr2=0x55a5fc084230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc084230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 -- 172.21.15.38:0/885825452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 msgr2=0x55a5fc083cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc083cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.813+0000 7f673f7fe640 1 -- 172.21.15.38:0/885825452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5fc084870 con 0x55a5fc080f30 2024-04-22T00:08:41.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.815+0000 7f673f7fe640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc092750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55a5fc080a00 tx=0x7f673002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=580f78dddf53eb19 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:41.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.815+0000 7f671effd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:41.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730005240 con 0x55a5fc080f30 2024-04-22T00:08:41.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7f67300053e0 con 0x55a5fc080f30 2024-04-22T00:08:41.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6728004620 con 0x55a5fc080f30 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f67300055e0 con 0x55a5fc080f30 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f6748c26740 10 client.27212 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a5fc122c30 con 0x55a5fc080f30 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.816+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6728004620 con 0x55a5fc080f30 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f673004a040 con 0x55a5fc080f30 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 10 client.27212 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 20 client.27212 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc123430 0x55a5fc125850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 4 client.27212 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a5fbfecdf0 con 0x55a5fc123430 2024-04-22T00:08:41.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.817+0000 7f6737fff640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc123430 0x55a5fc125850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:41.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f671effd640 10 client.27212 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6737fff640 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc123430 0x55a5fc125850 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca621c96b529f08 server_cookie=2230cca05ad1ac9e in_seq=0 out_seq=0 2024-04-22T00:08:41.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f671effd640 1 -- 172.21.15.38:0/885825452 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a5fbfecdf0 con 0x55a5fc123430 2024-04-22T00:08:41.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f671effd640 10 client.27212 handle_command_reply: tid=0 2024-04-22T00:08:41.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 client.27212 shutdown 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 msgr2=0x55a5fc092750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc092750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55a5fc080a00 tx=0x7f673002bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 shutdown_connections 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a5fc08a430 0x55a5fc084230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55a5fc123430 0x55a5fc125850 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f671effd640 0 client.27212 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a5fc081850 0x55a5fc083cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 --2- 172.21.15.38:0/885825452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5fc080f30 0x55a5fc092750 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.818+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 >> 172.21.15.38:0/885825452 conn(0x55a5fc07b660 msgr2=0x55a5fc07d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.819+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 shutdown_connections 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.819+0000 7f6748c26740 1 -- 172.21.15.38:0/885825452 wait complete. 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.819+0000 7f6748c26740 20 client.27212 trim_cache size 0 max 16384 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (118 inodes in the stack)", 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "5dc84a98-613d-4b45-b556-45a2fde8cc15": { 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "5dc84a98-613d-4b45-b556-45a2fde8cc15", 2024-04-22T00:08:41.817 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:08:41.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:08:41.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:08:41.818 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 msgr2=0x7f672c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 0x7f672c0a5300 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f673002c240 tx=0x7f673002df90 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 msgr2=0x7f674006cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f673800eb10 tx=0x7f673800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 shutdown_connections 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f672c0a2e40 0x7f672c0a5300 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6740106630 0x7f674006cc90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6740100fe0 0x7f6740073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 --2- 172.21.15.38:0/4269450026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6740100620 0x7f6740072c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 >> 172.21.15.38:0/4269450026 conn(0x7f6740077770 msgr2=0x7f674010b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:41.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.820+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 shutdown_connections 2024-04-22T00:08:41.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:08:41.821+0000 7f6745f5b640 1 -- 172.21.15.38:0/4269450026 wait complete. 2024-04-22T00:08:41.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:41 smithi111 ceph-mon[31543]: pgmap v201: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.39k op/s 2024-04-22T00:08:41.839 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:41 smithi038 ceph-mon[27972]: pgmap v201: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.39k op/s 2024-04-22T00:08:41.841 INFO:tasks.cephfs.filesystem:scrub status for tag:5dc84a98-613d-4b45-b556-45a2fde8cc15 - {'path': '/', 'tag': '5dc84a98-613d-4b45-b556-45a2fde8cc15', 'options': 'recursive,force'} 2024-04-22T00:08:42.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:41 smithi203 ceph-mon[31431]: pgmap v201: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.39k op/s 2024-04-22T00:08:42.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:42 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:08:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:42 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:42 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:42.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:42 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:08:42.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:42 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:42.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:42 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:43.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:42 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:08:43.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:42 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:43.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:42 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:43.829 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:43 smithi111 ceph-mon[31543]: pgmap v202: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 13 MiB/s wr, 3.17k op/s 2024-04-22T00:08:43.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:43 smithi038 ceph-mon[27972]: pgmap v202: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 13 MiB/s wr, 3.17k op/s 2024-04-22T00:08:44.032 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:43 smithi203 ceph-mon[31431]: pgmap v202: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 13 MiB/s wr, 3.17k op/s 2024-04-22T00:08:46.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:46 smithi203 ceph-mon[31431]: pgmap v203: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.7 MiB/s wr, 2.38k op/s 2024-04-22T00:08:46.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:46 smithi111 ceph-mon[31543]: pgmap v203: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.7 MiB/s wr, 2.38k op/s 2024-04-22T00:08:46.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:46 smithi038 ceph-mon[27972]: pgmap v203: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 9.7 MiB/s wr, 2.38k op/s 2024-04-22T00:08:48.282 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:48 smithi203 ceph-mon[31431]: pgmap v204: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 12 MiB/s wr, 2.84k op/s 2024-04-22T00:08:48.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:48 smithi111 ceph-mon[31543]: pgmap v204: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 12 MiB/s wr, 2.84k op/s 2024-04-22T00:08:48.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:48 smithi038 ceph-mon[27972]: pgmap v204: 129 pgs: 129 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 12 MiB/s wr, 2.84k op/s 2024-04-22T00:08:50.079 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:50 smithi111 ceph-mon[31543]: pgmap v205: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:50.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:50 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:50.080 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:08:49 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:08:49.793+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v646>,<0x100000001fb/client.0 v624>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v622>,<0x10000000001/sv_1 v634>,<0x10000000000/qa v612>,<0x1/volumes v591>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:08:50.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:50 smithi038 ceph-mon[27972]: pgmap v205: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:50.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:50 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:50.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:50 smithi203 ceph-mon[31431]: pgmap v205: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 2.86k op/s 2024-04-22T00:08:50.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:50 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:51.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:51 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:51.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:51 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:51.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:51 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:52.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:52 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:08:52.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:52 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:52.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:52 smithi111 ceph-mon[31543]: pgmap v206: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.40k op/s 2024-04-22T00:08:52.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:52 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:08:52.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:52 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:52.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:52 smithi038 ceph-mon[27972]: pgmap v206: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.40k op/s 2024-04-22T00:08:52.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:52 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:08:52.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:52 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:52.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:52 smithi203 ceph-mon[31431]: pgmap v206: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 2.40k op/s 2024-04-22T00:08:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 -- 172.21.15.38:0/3269992498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8100d50 msgr2=0x7ff3d81011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/3269992498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8100d50 0x7ff3d81011d0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009930 tx=0x7ff3cc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:08:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 -- 172.21.15.38:0/3269992498 shutdown_connections 2024-04-22T00:08:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/3269992498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8101710 0x7ff3d810d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/3269992498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8100d50 0x7ff3d81011d0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/3269992498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8106840 0x7ff3d8106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 -- 172.21.15.38:0/3269992498 >> 172.21.15.38:0/3269992498 conn(0x7ff3d8076bd0 msgr2=0x7ff3d80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 -- 172.21.15.38:0/3269992498 shutdown_connections 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.240+0000 7ff3dce0e640 1 -- 172.21.15.38:0/3269992498 wait complete. 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 Processor -- start 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 -- start start 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 0x7ff3d810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8106840 0x7ff3d810ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8111dc0 con 0x7ff3d8101710 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8111c40 con 0x7ff3d8100d50 2024-04-22T00:08:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3dce0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8111f40 con 0x7ff3d8106840 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46928/0 (socket says 172.21.15.38:46928) 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1125784658 learned_addr learned my addr 172.21.15.38:0/1125784658 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 0x7ff3d810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.241+0000 7ff3d6d9d640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8106840 0x7ff3d810ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1125784658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8106840 msgr2=0x7ff3d810ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8106840 0x7ff3d810ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1125784658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 msgr2=0x7ff3d810e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 0x7ff3d810e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1125784658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d81a66c0 con 0x7ff3d8101710 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 0x7ff3d810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009900 tx=0x7ff3cc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3c77fe640 1 -- 172.21.15.38:0/1125784658 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc043020 con 0x7ff3d8101710 2024-04-22T00:08:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d81a6950 con 0x7ff3d8101710 2024-04-22T00:08:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3c77fe640 1 -- 172.21.15.38:0/1125784658 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3cc0054e0 con 0x7ff3d8101710 2024-04-22T00:08:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d81a6ef0 con 0x7ff3d8101710 2024-04-22T00:08:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.242+0000 7ff3c77fe640 1 -- 172.21.15.38:0/1125784658 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc003bb0 con 0x7ff3d8101710 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.246+0000 7ff3df9f8740 1 Processor -- start 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 -- start start 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 0x560403986710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403991e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5604038e1b70 con 0x560403986330 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56040388a470 con 0x560403986c50 2024-04-22T00:08:53.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604038ddb30 con 0x56040398fa20 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 0x560403986710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 0x560403986710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46936/0 (socket says 172.21.15.38:46936) 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3d6d9d640 1 -- 172.21.15.38:0/1023546929 learned_addr learned my addr 172.21.15.38:0/1023546929 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.247+0000 7ff3c77fe640 1 -- 172.21.15.38:0/1125784658 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff3cc046020 con 0x7ff3d8101710 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 -- 172.21.15.38:0/1023546929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 msgr2=0x560403991e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403991e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 -- 172.21.15.38:0/1023546929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 msgr2=0x560403986710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 0x560403986710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 -- 172.21.15.38:0/1023546929 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604035f6210 con 0x560403986c50 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398f4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009f50 tx=0x7ff3c80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d72901692d96d230 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3c4ff9640 1 -- 172.21.15.38:0/1023546929 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8004020 con 0x560403986c50 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3c4ff9640 1 -- 172.21.15.38:0/1023546929 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c80041c0 con 0x560403986c50 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3c4ff9640 1 -- 172.21.15.38:0/1023546929 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800f870 con 0x560403986c50 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.248+0000 7ff3c77fe640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 0x7ff3c00a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3c77fe640 1 -- 172.21.15.38:0/1125784658 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3cc003950 con 0x7ff3d8101710 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3d6d9d640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 0x7ff3c00a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1023546929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 msgr2=0x56040398f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398f4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009f50 tx=0x7ff3c80118d0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1023546929 shutdown_connections 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403991e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398f4e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1023546929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986330 0x560403986710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1023546929 >> 172.21.15.38:0/1023546929 conn(0x560403980c50 msgr2=0x560403981030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.249+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1023546929 shutdown_connections 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.250+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1023546929 wait complete. 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.250+0000 7ff3d6d9d640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 0x7ff3c00a54e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0012640 tx=0x7ff3d0009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.250+0000 7ff3df9f8740 1 -- start start 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.250+0000 7ff3df9f8740 1 Processor -- start 2024-04-22T00:08:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 0x56040398d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560403992650 con 0x56040398fa20 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56040389da10 con 0x560403986c50 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604039927d0 con 0x560403986330 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3df9f8740 10 client.0 _ll_register_callbacks cb 0x5604038d8b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 0x56040398d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 0x56040398d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46944/0 (socket says 172.21.15.38:46944) 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d6d9d640 1 -- 172.21.15.38:0/1101076239 learned_addr learned my addr 172.21.15.38:0/1101076239 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1101076239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 msgr2=0x56040398ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1101076239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 msgr2=0x56040398d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 0x56040398d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d5d9b640 1 -- 172.21.15.38:0/1101076239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604036c2620 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.251+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398cf90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0ee990 tx=0x7ff3cc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d488759b444e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc0d6060 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c80041c0 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3cc0eeb00 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3cc004c80 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3df9f8740 10 client.27224 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56040399e8b0 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.252+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c80041c0 con 0x560403986c50 2024-04-22T00:08:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff3cc0bd9c0 con 0x560403986c50 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 10 client.27224 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 10 client.27224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 20 client.27224 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56040399ff40 0x5604039a2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 4 client.27224 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604038ef0b0 con 0x56040399ff40 2024-04-22T00:08:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.253+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56040399ff40 0x5604039a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.254+0000 7ff3bdffb640 10 client.27224 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.254+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56040399ff40 0x5604039a2360 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=474974e8320fd2c8 server_cookie=85c649dda320074a in_seq=0 out_seq=0 2024-04-22T00:08:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.254+0000 7ff3bdffb640 1 -- 172.21.15.38:0/1101076239 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604038ef0b0 con 0x56040399ff40 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.254+0000 7ff3bdffb640 10 client.27224 handle_command_reply: tid=0 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 client.27224 shutdown 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 msgr2=0x56040398cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398cf90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0ee990 tx=0x7ff3cc009260 comp rx=0 tx=0).stop 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 shutdown_connections 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56040399ff40 0x5604039a2360 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56040398fa20 0x56040398d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986c50 0x56040398cf90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1101076239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 >> 172.21.15.38:0/1101076239 conn(0x560403980c50 msgr2=0x5604039829a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3bdffb640 0 client.27224 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 shutdown_connections 2024-04-22T00:08:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1101076239 wait complete. 2024-04-22T00:08:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.255+0000 7ff3df9f8740 20 client.27224 trim_cache size 0 max 16384 2024-04-22T00:08:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.277+0000 7ff3df9f8740 1 Processor -- start 2024-04-22T00:08:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.277+0000 7ff3df9f8740 1 -- start start 2024-04-22T00:08:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x56040399b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 0x560403977340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81ac890 con 0x560403986c50 2024-04-22T00:08:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81ae3d0 con 0x56040398fa20 2024-04-22T00:08:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81acf60 con 0x560403986330 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3d659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 0x560403977340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3d6d9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 0x560403977340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:41654/0 (socket says 172.21.15.38:41654) 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3d5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x56040399b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.278+0000 7ff3d6d9d640 1 -- 172.21.15.38:0/1187265973 learned_addr learned my addr 172.21.15.38:0/1187265973 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 -- 172.21.15.38:0/1187265973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 msgr2=0x560403977340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 0x560403977340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 -- 172.21.15.38:0/1187265973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 msgr2=0x56040399b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x56040399b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 -- 172.21.15.38:0/1187265973 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560403988630 con 0x560403986330 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3d659c640 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398f180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8011dd0 tx=0x7ff3c80041f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a40a7da849580004 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3bffff640 1 -- 172.21.15.38:0/1187265973 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c801d040 con 0x560403986330 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3bffff640 1 -- 172.21.15.38:0/1187265973 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c802f070 con 0x560403986330 2024-04-22T00:08:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.279+0000 7ff3bffff640 1 -- 172.21.15.38:0/1187265973 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8003a30 con 0x560403986330 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1187265973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 msgr2=0x56040398f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398f180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8011dd0 tx=0x7ff3c80041f0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1187265973 shutdown_connections 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56040398fa20 0x560403977340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x56040399b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/1187265973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560403986330 0x56040398f180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1187265973 >> 172.21.15.38:0/1187265973 conn(0x560403980c50 msgr2=0x560403976c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1187265973 shutdown_connections 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- 172.21.15.38:0/1187265973 wait complete. 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 -- start start 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.280+0000 7ff3df9f8740 1 Processor -- start 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x560403997c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403992b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81add00 con 0x560403986c50 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81ad630 con 0x560403986330 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81aa870 con 0x56040398fa20 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:41660/0 (socket says 172.21.15.38:41660) 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 -- 172.21.15.38:0/2752726169 learned_addr learned my addr 172.21.15.38:0/2752726169 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3df9f8740 10 client.0 _ll_register_callbacks cb 0x7ff3cc009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d6d9d640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403992b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 -- 172.21.15.38:0/2752726169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 msgr2=0x560403992b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403992b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 -- 172.21.15.38:0/2752726169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 msgr2=0x560403997c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x560403997c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.281+0000 7ff3d659c640 1 -- 172.21.15.38:0/2752726169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604039932f0 con 0x560403986330 2024-04-22T00:08:53.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3d659c640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8011910 tx=0x7ff3c8003fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd29bab8889f320e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c803a060 con 0x560403986330 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7ff3c80042e0 con 0x560403986330 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c802f070 con 0x560403986330 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3c801d040 con 0x560403986330 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3df9f8740 10 client.27230 fetch_fsmap learned FSMap version 35 2024-04-22T00:08:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.282+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560403a05f50 con 0x560403986330 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c8005520 con 0x560403986330 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff3c802a430 con 0x560403986330 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 10 client.27230 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 20 client.27230 populate_metadata read hostname 'smithi038' 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560403a06470 0x560403a08860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 4 client.27230 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:08:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.284+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff3c802f070 con 0x560403a06470 2024-04-22T00:08:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.285+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560403a06470 0x560403a08860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:08:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.285+0000 7ff3d5d9b640 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560403a06470 0x560403a08860 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e4f8419f3bfeca server_cookie=33a37944144f91ec in_seq=0 out_seq=0 2024-04-22T00:08:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.286+0000 7ff3bdffb640 10 client.27230 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:53.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:53 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:53.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:53 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:53.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:53 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3bdffb640 1 -- 172.21.15.38:0/2752726169 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff3c802f070 con 0x560403a06470 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3bdffb640 10 client.27230 handle_command_reply: tid=0 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3df9f8740 1 client.27230 shutdown 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 msgr2=0x5604039976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8011910 tx=0x7ff3c8003fb0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.811+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 shutdown_connections 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56040398fa20 0x560403992b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x560403a06470 0x560403a08860 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560403986c50 0x560403997c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 --2- 172.21.15.38:0/2752726169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560403986330 0x5604039976f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3bdffb640 0 client.27230 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 >> 172.21.15.38:0/2752726169 conn(0x560403980c50 msgr2=0x5604039774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 shutdown_connections 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 1 -- 172.21.15.38:0/2752726169 wait complete. 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.812+0000 7ff3df9f8740 20 client.27230 trim_cache size 0 max 16384 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:08:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:08:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 msgr2=0x7ff3c00a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 0x7ff3c00a54e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0012640 tx=0x7ff3d0009500 comp rx=0 tx=0).stop 2024-04-22T00:08:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 msgr2=0x7ff3d810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009900 tx=0x7ff3cc004360 comp rx=0 tx=0).stop 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 shutdown_connections 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff3c00a3020 0x7ff3c00a54e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3d8106840 0x7ff3d810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3d8101710 0x7ff3d810e550 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 --2- 172.21.15.38:0/1125784658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3d8100d50 0x7ff3d810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 >> 172.21.15.38:0/1125784658 conn(0x7ff3d8076bd0 msgr2=0x7ff3d81036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 shutdown_connections 2024-04-22T00:08:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:08:53.813+0000 7ff3dce0e640 1 -- 172.21.15.38:0/1125784658 wait complete. 2024-04-22T00:08:54.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:54 smithi111 ceph-mon[31543]: pgmap v207: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 15 MiB/s wr, 3.62k op/s 2024-04-22T00:08:54.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:54 smithi038 ceph-mon[27972]: pgmap v207: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 15 MiB/s wr, 3.62k op/s 2024-04-22T00:08:54.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:54 smithi203 ceph-mon[31431]: pgmap v207: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 15 MiB/s wr, 3.62k op/s 2024-04-22T00:08:56.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:56 smithi111 ceph-mon[31543]: pgmap v208: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-04-22T00:08:56.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:56 smithi038 ceph-mon[27972]: pgmap v208: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-04-22T00:08:56.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:56 smithi203 ceph-mon[31431]: pgmap v208: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-04-22T00:08:58.329 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:08:58 smithi111 ceph-mon[31543]: pgmap v209: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:08:58.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:08:58 smithi038 ceph-mon[27972]: pgmap v209: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:08:58.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:08:58 smithi203 ceph-mon[31431]: pgmap v209: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 12 MiB/s wr, 3.05k op/s 2024-04-22T00:09:00.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:00 smithi111 ceph-mon[31543]: pgmap v210: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-04-22T00:09:00.460 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:00 smithi038 ceph-mon[27972]: pgmap v210: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-04-22T00:09:00.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:00 smithi203 ceph-mon[31431]: pgmap v210: 129 pgs: 129 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-04-22T00:09:01.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:01 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:01.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:01 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:01.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:01 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:02.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:02 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:02.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:02 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:02.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:02 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:02.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:02 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:02.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:02 smithi038 ceph-mon[27972]: pgmap v211: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 2.64k op/s 2024-04-22T00:09:02.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:02 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:02.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:02 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:02.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:02 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:02.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:02 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:02.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:02 smithi203 ceph-mon[31431]: pgmap v211: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 2.64k op/s 2024-04-22T00:09:02.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:02 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:02.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:02 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:02.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:02 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:02.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:02 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:02.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:02 smithi111 ceph-mon[31543]: pgmap v211: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 2.64k op/s 2024-04-22T00:09:04.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:04 smithi038 ceph-mon[27972]: pgmap v212: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-04-22T00:09:04.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:04 smithi203 ceph-mon[31431]: pgmap v212: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-04-22T00:09:04.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:04 smithi111 ceph-mon[31543]: pgmap v212: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 14 MiB/s wr, 3.48k op/s 2024-04-22T00:09:06.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:06 smithi038 ceph-mon[27972]: pgmap v213: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-04-22T00:09:06.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:06 smithi203 ceph-mon[31431]: pgmap v213: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-04-22T00:09:06.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:06 smithi111 ceph-mon[31543]: pgmap v213: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 10 MiB/s wr, 2.55k op/s 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 -- 172.21.15.38:0/1507314512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c101750 msgr2=0x7efc0c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 --2- 172.21.15.38:0/1507314512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c101750 0x7efc0c101bd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7efbf4009930 tx=0x7efbf402bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 -- 172.21.15.38:0/1507314512 shutdown_connections 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 --2- 172.21.15.38:0/1507314512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 0x7efc0c10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 --2- 172.21.15.38:0/1507314512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c101750 0x7efc0c101bd0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 --2- 172.21.15.38:0/1507314512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c107760 0x7efc0c107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 -- 172.21.15.38:0/1507314512 >> 172.21.15.38:0/1507314512 conn(0x7efc0c078bd0 msgr2=0x7efc0c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 -- 172.21.15.38:0/1507314512 shutdown_connections 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.910+0000 7efc1173e640 1 -- 172.21.15.38:0/1507314512 wait complete. 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 Processor -- start 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 -- start start 2024-04-22T00:09:06.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 0x7efc0c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 0x7efc0c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c111b30 con 0x7efc0c107760 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1119b0 con 0x7efc0c102110 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc1173e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c111cb0 con 0x7efc0c101750 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 0x7efc0c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41944/0 (socket says 172.21.15.38:41944) 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3455456627 learned_addr learned my addr 172.21.15.38:0/3455456627 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0affd640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 0x7efc0c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50560/0 (socket says 172.21.15.38:50560) 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc03fff640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 0x7efc0c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3455456627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 msgr2=0x7efc0c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 0x7efc0c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3455456627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 msgr2=0x7efc0c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 0x7efc0c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.911+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3455456627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0c19cf60 con 0x7efc0c107760 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7efc0400ea60 tx=0x7efc0400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc08ff9640 1 -- 172.21.15.38:0/3455456627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0400ce40 con 0x7efc0c107760 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc08ff9640 1 -- 172.21.15.38:0/3455456627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc04005020 con 0x7efc0c107760 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0c19d250 con 0x7efc0c107760 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc08ff9640 1 -- 172.21.15.38:0/3455456627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04010690 con 0x7efc0c107760 2024-04-22T00:09:06.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.912+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0c1a4ae0 con 0x7efc0c107760 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.914+0000 7efc08ff9640 1 -- 172.21.15.38:0/3455456627 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efc04005520 con 0x7efc0c107760 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.914+0000 7efc14407740 1 Processor -- start 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.914+0000 7efc14407740 1 -- start start 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc08ff9640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 0x7efbf80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e7a2114b70 con 0x55e7a21ba210 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e7a20bd470 con 0x55e7a21b1440 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc14407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e7a2110b30 con 0x55e7a21b0b20 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc08ff9640 1 -- 172.21.15.38:0/3455456627 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efc040cfc00 con 0x7efc0c107760 2024-04-22T00:09:06.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc03fff640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 0x7efbf80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41946/0 (socket says 172.21.15.38:41946) 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3513062282 learned_addr learned my addr 172.21.15.38:0/3513062282 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0affd640 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc03fff640 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3513062282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 msgr2=0x55e7a21b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3513062282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 msgr2=0x55e7a21b9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21b9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3513062282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7a1e29210 con 0x55e7a21ba210 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7efc040987b0 tx=0x7efc0409b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c2c67d97dfedac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc03fff640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 0x7efbf80a52e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efbf4005ab0 tx=0x7efbf402c980 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc017fa640 1 -- 172.21.15.38:0/3513062282 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0409bde0 con 0x55e7a21ba210 2024-04-22T00:09:06.920 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc017fa640 1 -- 172.21.15.38:0/3513062282 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc040a0190 con 0x55e7a21ba210 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.915+0000 7efc017fa640 1 -- 172.21.15.38:0/3513062282 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040a45c0 con 0x55e7a21ba210 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- 172.21.15.38:0/3513062282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 msgr2=0x55e7a21bc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7efc040987b0 tx=0x7efc0409b800 comp rx=0 tx=0).stop 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- 172.21.15.38:0/3513062282 shutdown_connections 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21bc600 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21b9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 --2- 172.21.15.38:0/3513062282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- 172.21.15.38:0/3513062282 >> 172.21.15.38:0/3513062282 conn(0x55e7a21ab440 msgr2=0x55e7a21ab820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- 172.21.15.38:0/3513062282 shutdown_connections 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- 172.21.15.38:0/3513062282 wait complete. 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.916+0000 7efc14407740 1 -- start start 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 Processor -- start 2024-04-22T00:09:06.921 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b1440 0x55e7a21b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e7a21bce40 con 0x55e7a21ba210 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e7a20d0a10 con 0x55e7a21b0b20 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e7a21bcfc0 con 0x55e7a21b1440 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc14407740 10 client.0 _ll_register_callbacks cb 0x55e7a210bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc03fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b1440 0x55e7a21b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41960/0 (socket says 172.21.15.38:41960) 2024-04-22T00:09:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 -- 172.21.15.38:0/1988087664 learned_addr learned my addr 172.21.15.38:0/1988087664 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0affd640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 -- 172.21.15.38:0/1988087664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b1440 msgr2=0x55e7a21b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b1440 0x55e7a21b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 -- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 msgr2=0x55e7a21b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 -- 172.21.15.38:0/1988087664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7a1ef5620 con 0x55e7a21ba210 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0affd640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7efc0409bd70 tx=0x7efc04022d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3841fb0c868ab41e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efbf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.917+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040b9070 con 0x55e7a21ba210 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efc040a45c0 con 0x55e7a21ba210 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc040a0190 con 0x55e7a21ba210 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efc040a4a50 con 0x55e7a21ba210 2024-04-22T00:09:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 10 client.17904 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7a21c9100 con 0x55e7a21ba210 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc040a0190 con 0x55e7a21ba210 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efc0409aca0 con 0x55e7a21ba210 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 10 client.17904 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 10 client.17904 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 20 client.17904 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a21ca450 0x55e7a21cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 4 client.17904 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7a21220b0 con 0x55e7a21ca450 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.918+0000 7efc0affd640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a21ca450 0x55e7a21cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.919+0000 7efc0affd640 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a21ca450 0x55e7a21cc840 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d0c4879aebd23f server_cookie=9263dbdeea630d2a in_seq=0 out_seq=0 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.919+0000 7efbf27fc640 10 client.17904 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efbf27fc640 1 -- 172.21.15.38:0/1988087664 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7a21220b0 con 0x55e7a21ca450 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efbf27fc640 10 client.17904 handle_command_reply: tid=0 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 client.17904 shutdown 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 msgr2=0x55e7a21b7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7efc0409bd70 tx=0x7efc04022d50 comp rx=0 tx=0).stop 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 shutdown_connections 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a21ca450 0x55e7a21cc840 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21b7d20 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b1440 0x55e7a21b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efbf27fc640 0 client.17904 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 --2- 172.21.15.38:0/1988087664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 >> 172.21.15.38:0/1988087664 conn(0x55e7a21ab440 msgr2=0x55e7a21ad190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 shutdown_connections 2024-04-22T00:09:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 1 -- 172.21.15.38:0/1988087664 wait complete. 2024-04-22T00:09:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.920+0000 7efc14407740 20 client.17904 trim_cache size 0 max 16384 2024-04-22T00:09:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 Processor -- start 2024-04-22T00:09:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 -- start start 2024-04-22T00:09:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.938 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 0x55e7a21b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1aea60 con 0x55e7a21b1440 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1b05a0 con 0x55e7a21b0b20 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc14407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1af130 con 0x55e7a21ba210 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41974/0 (socket says 172.21.15.38:41974) 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 -- 172.21.15.38:0/3429601015 learned_addr learned my addr 172.21.15.38:0/3429601015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc0affd640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 0x55e7a21b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 -- 172.21.15.38:0/3429601015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 msgr2=0x55e7a21b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 0x55e7a21b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 -- 172.21.15.38:0/3429601015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 msgr2=0x55e7a21b99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.939 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 -- 172.21.15.38:0/3429601015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7a21b2e60 con 0x55e7a21b1440 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 0x55e7a21b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc0affd640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.938+0000 7efc03fff640 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7efbf4036040 tx=0x7efbf402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67fe256b1cdc122 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc00ff9640 1 -- 172.21.15.38:0/3429601015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4004980 con 0x55e7a21b1440 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc00ff9640 1 -- 172.21.15.38:0/3429601015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbf4005430 con 0x55e7a21b1440 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc00ff9640 1 -- 172.21.15.38:0/3429601015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4003850 con 0x55e7a21b1440 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 -- 172.21.15.38:0/3429601015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 msgr2=0x55e7a21b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7efbf4036040 tx=0x7efbf402e050 comp rx=0 tx=0).stop 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 -- 172.21.15.38:0/3429601015 shutdown_connections 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21ba210 0x55e7a21b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21b1440 0x55e7a21b6280 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 --2- 172.21.15.38:0/3429601015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b0b20 0x55e7a21b99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 -- 172.21.15.38:0/3429601015 >> 172.21.15.38:0/3429601015 conn(0x55e7a21ab440 msgr2=0x55e7a21b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:06.940 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 -- 172.21.15.38:0/3429601015 shutdown_connections 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.939+0000 7efc14407740 1 -- 172.21.15.38:0/3429601015 wait complete. 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 -- start start 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 Processor -- start 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1afed0 con 0x55e7a21ba210 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1af800 con 0x55e7a21b1440 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1aca40 con 0x55e7a21b0b20 2024-04-22T00:09:06.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc14407740 10 client.0 _ll_register_callbacks cb 0x7efbfc00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41986/0 (socket says 172.21.15.38:41986) 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3972421336 learned_addr learned my addr 172.21.15.38:0/3972421336 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc03fff640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0affd640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3972421336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 msgr2=0x55e7a21cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 msgr2=0x55e7a21cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:06.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.940+0000 7efc0b7fe640 1 -- 172.21.15.38:0/3972421336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7a222f680 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc0b7fe640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7efc040049d0 tx=0x7efc040a1550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258f1000908b8b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc040b9050 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9350+0+0 (secure 0 0 0) 0x7efc0409b230 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbf4005430 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efc040ad050 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 10 client.17910 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7a22308e0 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbf4005430 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efc04086aa0 con 0x55e7a21ba210 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 10 client.17910 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 10 client.17910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 20 client.17910 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a2230db0 0x55e7a22331a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:06.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 4 client.17910 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:09:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.941+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7a211c400 con 0x55e7a2230db0 2024-04-22T00:09:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.942+0000 7efc0affd640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a2230db0 0x55e7a22331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.942+0000 7efc0affd640 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a2230db0 0x55e7a22331a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1df7c26c74f2d9 server_cookie=d7f1d91e2e06cc50 in_seq=0 out_seq=0 2024-04-22T00:09:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:06.942+0000 7efbf27fc640 10 client.17910 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.483+0000 7efbf27fc640 1 -- 172.21.15.38:0/3972421336 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7a211c400 con 0x55e7a2230db0 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.483+0000 7efbf27fc640 10 client.17910 handle_command_reply: tid=0 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 client.17910 shutdown 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 msgr2=0x55e7a21cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7efc040049d0 tx=0x7efc040a1550 comp rx=0 tx=0).stop 2024-04-22T00:09:07.481 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 shutdown_connections 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55e7a2230db0 0x55e7a22331a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efbf27fc640 0 client.17910 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7a21ba210 0x55e7a21cdd70 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e7a21b1440 0x55e7a21cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.484+0000 7efc14407740 1 --2- 172.21.15.38:0/3972421336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7a21b0b20 0x55e7a21cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.485+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 >> 172.21.15.38:0/3972421336 conn(0x55e7a21ab440 msgr2=0x55e7a21c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.485+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 shutdown_connections 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.485+0000 7efc14407740 1 -- 172.21.15.38:0/3972421336 wait complete. 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.485+0000 7efc14407740 20 client.17910 trim_cache size 0 max 16384 2024-04-22T00:09:07.482 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:09:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:09:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:09:07.483 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 msgr2=0x7efbf80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 0x7efbf80a52e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efbf4005ab0 tx=0x7efbf402c980 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 msgr2=0x7efc0c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7efc0400ea60 tx=0x7efc0400ef20 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 shutdown_connections 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efbf80a2e20 0x7efbf80a52e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc0c107760 0x7efc0c19c830 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc0c102110 0x7efc0c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 --2- 172.21.15.38:0/3455456627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc0c101750 0x7efc0c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:07.484 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 >> 172.21.15.38:0/3455456627 conn(0x7efc0c078bd0 msgr2=0x7efc0c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 shutdown_connections 2024-04-22T00:09:07.485 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:07.486+0000 7efc1173e640 1 -- 172.21.15.38:0/3455456627 wait complete. 2024-04-22T00:09:08.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:08 smithi038 ceph-mon[27972]: pgmap v214: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-04-22T00:09:08.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:08 smithi203 ceph-mon[31431]: pgmap v214: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-04-22T00:09:08.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:08 smithi111 ceph-mon[31543]: pgmap v214: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-04-22T00:09:10.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:10 smithi038 ceph-mon[27972]: pgmap v215: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-04-22T00:09:10.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:10 smithi203 ceph-mon[31431]: pgmap v215: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-04-22T00:09:10.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:10 smithi111 ceph-mon[31543]: pgmap v215: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-04-22T00:09:11.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:09:12.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:12 smithi111 ceph-mon[31543]: pgmap v216: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.6 MiB/s wr, 2.36k op/s 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:12 smithi111 ceph-mon[31543]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:12 smithi203 ceph-mon[31431]: pgmap v216: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.6 MiB/s wr, 2.36k op/s 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:12 smithi203 ceph-mon[31431]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:12 smithi038 ceph-mon[27972]: pgmap v216: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.6 MiB/s wr, 2.36k op/s 2024-04-22T00:09:12.668 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:12 smithi038 ceph-mon[27972]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- 172.21.15.38:0/2902973768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108107720 msgr2=0x7f6108107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- 172.21.15.38:0/2902973768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108107720 0x7f6108107b00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009930 tx=0x7f60f802bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- 172.21.15.38:0/2902973768 shutdown_connections 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- 172.21.15.38:0/2902973768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 0x7f610810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- 172.21.15.38:0/2902973768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108101710 0x7f6108101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- 172.21.15.38:0/2902973768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108107720 0x7f6108107b00 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- 172.21.15.38:0/2902973768 >> 172.21.15.38:0/2902973768 conn(0x7f6108078b90 msgr2=0x7f6108078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- 172.21.15.38:0/2902973768 shutdown_connections 2024-04-22T00:09:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- 172.21.15.38:0/2902973768 wait complete. 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 Processor -- start 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 -- start start 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 0x7f6108198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.779+0000 7f610db13640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108107720 0x7f610819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f610db13640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6108111b30 con 0x7f6108101710 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f610db13640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61081119b0 con 0x7f6108107720 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f610db13640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6108111cb0 con 0x7f61081020d0 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41994/0 (socket says 172.21.15.38:41994) 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 -- 172.21.15.38:0/57023577 learned_addr learned my addr 172.21.15.38:0/57023577 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f6106ffd640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 0x7f6108198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f6107fff640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108107720 0x7f610819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 -- 172.21.15.38:0/57023577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 msgr2=0x7f6108198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 0x7f6108198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 -- 172.21.15.38:0/57023577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108107720 msgr2=0x7f610819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108107720 0x7f610819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 -- 172.21.15.38:0/57023577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610819d020 con 0x7f6108101710 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f61077fe640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009230 tx=0x7f60f802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f6104ff9640 1 -- 172.21.15.38:0/57023577 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8041070 con 0x7f6108101710 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f6104ff9640 1 -- 172.21.15.38:0/57023577 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f8004450 con 0x7f6108101710 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610819d2b0 con 0x7f6108101710 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f6104ff9640 1 -- 172.21.15.38:0/57023577 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8034440 con 0x7f6108101710 2024-04-22T00:09:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.780+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081a4ba0 con 0x7f6108101710 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.782+0000 7f6104ff9640 1 -- 172.21.15.38:0/57023577 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f60f8044070 con 0x7f6108101710 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 Processor -- start 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 -- start start 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b43ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b46820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562636a9f560 con 0x562636b3b850 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562636a47730 con 0x562636b3af30 2024-04-22T00:09:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61107dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562636a9b520 con 0x562636b44430 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6104ff9640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b43ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:37536/0 (socket says 172.21.15.38:37536) 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 -- 172.21.15.38:0/541300128 learned_addr learned my addr 172.21.15.38:0/541300128 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b43ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42002/0 (socket says 172.21.15.38:42002) 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6106ffd640 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b46820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6104ff9640 1 -- 172.21.15.38:0/57023577 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f60f80f1150 con 0x7f6108101710 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 -- 172.21.15.38:0/541300128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 msgr2=0x562636b46820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b46820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 -- 172.21.15.38:0/541300128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 msgr2=0x562636b43ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b43ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.783+0000 7f6107fff640 1 -- 172.21.15.38:0/541300128 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56263687d7d0 con 0x562636b3af30 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.784+0000 7f6107fff640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 0x7f60f00a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6100010c40 tx=0x7f610000c000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.784+0000 7f6107fff640 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6100012d00 tx=0x7f6100010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78edc8a322696e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.784+0000 7f60edffb640 1 -- 172.21.15.38:0/541300128 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100024070 con 0x562636b3af30 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.784+0000 7f60edffb640 1 -- 172.21.15.38:0/541300128 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6100009d20 con 0x562636b3af30 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.784+0000 7f60edffb640 1 -- 172.21.15.38:0/541300128 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100005280 con 0x562636b3af30 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- 172.21.15.38:0/541300128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 msgr2=0x562636b3b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6100012d00 tx=0x7f6100010740 comp rx=0 tx=0).stop 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- 172.21.15.38:0/541300128 shutdown_connections 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b46820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b43ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- 172.21.15.38:0/541300128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b3b310 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- 172.21.15.38:0/541300128 >> 172.21.15.38:0/541300128 conn(0x562636b35660 msgr2=0x562636b35a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- 172.21.15.38:0/541300128 shutdown_connections 2024-04-22T00:09:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- 172.21.15.38:0/541300128 wait complete. 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- start start 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 Processor -- start 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 0x562636b41980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 0x562636b41ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562636b471c0 con 0x562636b3af30 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562636a5ae00 con 0x562636b44430 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562636b47340 con 0x562636b3b850 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.785+0000 7f61107dc740 10 client.0 _ll_register_callbacks cb 0x562636a95eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 0x562636b41ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61077fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42014/0 (socket says 172.21.15.38:42014) 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61077fe640 1 -- 172.21.15.38:0/3862858236 learned_addr learned my addr 172.21.15.38:0/3862858236 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6106ffd640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 0x562636b41980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 -- 172.21.15.38:0/3862858236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 msgr2=0x562636b41980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 0x562636b41980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 -- 172.21.15.38:0/3862858236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 msgr2=0x562636b41440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 -- 172.21.15.38:0/3862858236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626367b33f0 con 0x562636b44430 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61077fe640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6106ffd640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 0x562636b41980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f6107fff640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 0x562636b41ec0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6100006010 tx=0x7f6100010d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad009b6651b51dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f60e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6100024040 con 0x562636b44430 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562636aacaa0 con 0x562636b44430 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f6100004c10 con 0x562636b44430 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6100004e10 con 0x562636b44430 2024-04-22T00:09:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61107dc740 10 client.27260 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.786+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626367730f0 con 0x562636b44430 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562636aacaa0 con 0x562636b44430 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f610000f040 con 0x562636b44430 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 10 client.27260 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 10 client.27260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 20 client.27260 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636b546d0 0x562636b56ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 4 client.27260 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562636aa6df0 con 0x562636b546d0 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61077fe640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636b546d0 0x562636b56ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f60e6ffd640 10 client.27260 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.787+0000 7f61077fe640 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636b546d0 0x562636b56ac0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3ebef677db89449 server_cookie=f570ab4b9a3cf438 in_seq=0 out_seq=0 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f60e6ffd640 1 -- 172.21.15.38:0/3862858236 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562636aa6df0 con 0x562636b546d0 2024-04-22T00:09:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f60e6ffd640 10 client.27260 handle_command_reply: tid=0 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 client.27260 shutdown 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 msgr2=0x562636b41ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 0x562636b41ec0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6100006010 tx=0x7f6100010d70 comp rx=0 tx=0).stop 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 shutdown_connections 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b44430 0x562636b41ec0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b3b850 0x562636b41980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636b546d0 0x562636b56ac0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 --2- 172.21.15.38:0/3862858236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3af30 0x562636b41440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 >> 172.21.15.38:0/3862858236 conn(0x562636b35660 msgr2=0x562636b373b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f60e6ffd640 0 client.27260 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 shutdown_connections 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.788+0000 7f61107dc740 1 -- 172.21.15.38:0/3862858236 wait complete. 2024-04-22T00:09:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.789+0000 7f61107dc740 20 client.27260 trim_cache size 0 max 16384 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.806+0000 7f61107dc740 1 Processor -- start 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.806+0000 7f61107dc740 1 -- start start 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b43b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61081acb80 con 0x562636b3b850 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61081ae6c0 con 0x562636b3af30 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61107dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61081ad250 con 0x562636b44430 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f6106ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b43b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f6106ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42030/0 (socket says 172.21.15.38:42030) 2024-04-22T00:09:12.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f6106ffd640 1 -- 172.21.15.38:0/1588497390 learned_addr learned my addr 172.21.15.38:0/1588497390 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f6107fff640 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 -- 172.21.15.38:0/1588497390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 msgr2=0x562636b3fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 -- 172.21.15.38:0/1588497390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 msgr2=0x562636b50080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f61077fe640 1 -- 172.21.15.38:0/1588497390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562636b40ad0 con 0x562636b3af30 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.807+0000 7f6106ffd640 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.808+0000 7f61077fe640 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b43b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006010 tx=0x7f60f80904b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6e97a2a394153d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.808+0000 7f60ed7fa640 1 -- 172.21.15.38:0/1588497390 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f804b040 con 0x562636b3af30 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.808+0000 7f60ed7fa640 1 -- 172.21.15.38:0/1588497390 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f80bd440 con 0x562636b3af30 2024-04-22T00:09:12.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.808+0000 7f60ed7fa640 1 -- 172.21.15.38:0/1588497390 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f80bb7d0 con 0x562636b3af30 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- 172.21.15.38:0/1588497390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 msgr2=0x562636b43b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b43b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006010 tx=0x7f60f80904b0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- 172.21.15.38:0/1588497390 shutdown_connections 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b50080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 --2- 172.21.15.38:0/1588497390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b43b70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- 172.21.15.38:0/1588497390 >> 172.21.15.38:0/1588497390 conn(0x562636b35660 msgr2=0x562636b41c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- 172.21.15.38:0/1588497390 shutdown_connections 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- 172.21.15.38:0/1588497390 wait complete. 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 -- start start 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.809+0000 7f61107dc740 1 Processor -- start 2024-04-22T00:09:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b4c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61081adff0 con 0x562636b3b850 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f61081ad920 con 0x562636b3af30 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61081aab80 con 0x562636b44430 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61107dc740 10 client.0 _ll_register_callbacks cb 0x7f60fc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42036/0 (socket says 172.21.15.38:42036) 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 -- 172.21.15.38:0/1285150518 learned_addr learned my addr 172.21.15.38:0/1285150518 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6107fff640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61077fe640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b4c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 -- 172.21.15.38:0/1285150518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 msgr2=0x562636b3dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 -- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 msgr2=0x562636b4c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b4c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 -- 172.21.15.38:0/1285150518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562636b3e460 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6107fff640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f61077fe640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b4c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f6106ffd640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00bba0 tx=0x7f60fc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c006c800fca49a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f60e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc022060 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.810+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f60fc01d670 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f80bd440 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f60fc029050 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 10 client.17928 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562636bba750 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f80bd440 con 0x562636b3b850 2024-04-22T00:09:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f60fc010690 con 0x562636b3b850 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 10 client.17928 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 10 client.17928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 20 client.17928 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636bbaf30 0x562636bbd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 4 client.17928 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562636a245a0 con 0x562636bbaf30 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.811+0000 7f61077fe640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636bbaf30 0x562636bbd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f60e6ffd640 10 client.17928 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61077fe640 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636bbaf30 0x562636bbd350 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6734c9df58912991 server_cookie=20386aacaee5bfb6 in_seq=0 out_seq=0 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f60e6ffd640 1 -- 172.21.15.38:0/1285150518 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562636a245a0 con 0x562636bbaf30 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f60e6ffd640 10 client.17928 handle_command_reply: tid=0 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 client.17928 shutdown 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 msgr2=0x562636b4cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f60fc00bba0 tx=0x7f60fc0035f0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 shutdown_connections 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562636b44430 0x562636b3dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x562636bbaf30 0x562636bbd350 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562636b3b850 0x562636b4cad0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 --2- 172.21.15.38:0/1285150518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562636b3af30 0x562636b4c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f60e6ffd640 0 client.17928 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 >> 172.21.15.38:0/1285150518 conn(0x562636b35660 msgr2=0x562636b33e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 shutdown_connections 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 1 -- 172.21.15.38:0/1285150518 wait complete. 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.812+0000 7f61107dc740 20 client.17928 trim_cache size 0 max 16384 2024-04-22T00:09:12.814 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:09:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:09:12.815 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 msgr2=0x7f60f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 0x7f60f00a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6100010c40 tx=0x7f610000c000 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 msgr2=0x7f6108198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009230 tx=0x7f60f802cc70 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 shutdown_connections 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6108107720 0x7f610819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61081020d0 0x7f6108198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 --2- 172.21.15.38:0/57023577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6108101710 0x7f6108198030 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 >> 172.21.15.38:0/57023577 conn(0x7f6108078b90 msgr2=0x7f61080ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 shutdown_connections 2024-04-22T00:09:12.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.813+0000 7f610db13640 1 -- 172.21.15.38:0/57023577 wait complete. 2024-04-22T00:09:12.821 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:09:12.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:09:12.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 -- 172.21.15.38:0/2692648331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27601029d0 msgr2=0x7f2760102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 --2- 172.21.15.38:0/2692648331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27601029d0 0x7f2760102db0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2750009900 tx=0x7f275002bd60 comp rx=0 tx=0).stop 2024-04-22T00:09:12.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 -- 172.21.15.38:0/2692648331 shutdown_connections 2024-04-22T00:09:12.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 --2- 172.21.15.38:0/2692648331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f276010bd10 0x7f276010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 --2- 172.21.15.38:0/2692648331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601032f0 0x7f276010b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 --2- 172.21.15.38:0/2692648331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27601029d0 0x7f2760102db0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 -- 172.21.15.38:0/2692648331 >> 172.21.15.38:0/2692648331 conn(0x7f2760077740 msgr2=0x7f2760077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:12.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 -- 172.21.15.38:0/2692648331 shutdown_connections 2024-04-22T00:09:12.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.930+0000 7f2765242640 1 -- 172.21.15.38:0/2692648331 wait complete. 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 Processor -- start 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 -- start start 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 0x7f27601980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 0x7f2760198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2760111b60 con 0x7f276010bd10 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f27601119e0 con 0x7f27601032f0 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f2765242640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2760111ce0 con 0x7f27601029d0 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f275f59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f275ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 0x7f27601980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f275e59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 0x7f2760198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f275f59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42060/0 (socket says 172.21.15.38:42060) 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.931+0000 7f275f59e640 1 -- 172.21.15.38:0/3084554265 learned_addr learned my addr 172.21.15.38:0/3084554265 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 -- 172.21.15.38:0/3084554265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 msgr2=0x7f2760198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 0x7f2760198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 -- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 msgr2=0x7f276019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 -- 172.21.15.38:0/3084554265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276019d0d0 con 0x7f27601029d0 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275f59e640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275e59c640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 0x7f2760198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f275ed9d640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 0x7f27601980e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2750009230 tx=0x7f275002dc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750043020 con 0x7f27601029d0 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276019d360 con 0x7f27601029d0 2024-04-22T00:09:12.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27500054e0 con 0x7f27601029d0 2024-04-22T00:09:12.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750003c20 con 0x7f27601029d0 2024-04-22T00:09:12.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.932+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27601a4c50 con 0x7f27601029d0 2024-04-22T00:09:12.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.933+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2760068560 con 0x7f27601029d0 2024-04-22T00:09:12.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.935+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2750046020 con 0x7f27601029d0 2024-04-22T00:09:12.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.935+0000 7f2747fff640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 0x7f27480a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:12.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.935+0000 7f275e59c640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 0x7f27480a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:12.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.935+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f27500f1980 con 0x7f27601029d0 2024-04-22T00:09:12.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.936+0000 7f275e59c640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 0x7f27480a5240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f27600ff700 tx=0x7f2754005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:12.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:12.939+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f275004d020 con 0x7f27601029d0 2024-04-22T00:09:13.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.166+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2760106e90 con 0x7f27601029d0 2024-04-22T00:09:13.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.167+0000 7f2747fff640 1 -- 172.21.15.38:0/3084554265 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11935 (secure 0 0 0) 0x7f27500be2c0 con 0x7f27601029d0 2024-04-22T00:09:13.163 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:09:13.163 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:06:00.846013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24487,24500,24460,24455,24448]},"id":1}]} 2024-04-22T00:09:13.163 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 35 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 msgr2=0x7f27480a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 0x7f27480a5240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f27600ff700 tx=0x7f2754005f90 comp rx=0 tx=0).stop 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 msgr2=0x7f27601980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 0x7f27601980e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2750009230 tx=0x7f275002dc70 comp rx=0 tx=0).stop 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 shutdown_connections 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f27480a2d80 0x7f27480a5240 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f276010bd10 0x7f276019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27601032f0 0x7f2760198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 --2- 172.21.15.38:0/3084554265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27601029d0 0x7f27601980e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 >> 172.21.15.38:0/3084554265 conn(0x7f2760077740 msgr2=0x7f27600fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 shutdown_connections 2024-04-22T00:09:13.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.169+0000 7f2765242640 1 -- 172.21.15.38:0/3084554265 wait complete. 2024-04-22T00:09:13.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:09:13.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 -- 172.21.15.38:0/2300438279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100e60 msgr2=0x7ff1b810dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/2300438279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100e60 0x7ff1b810dbb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009900 tx=0x7ff1ac02bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 -- 172.21.15.38:0/2300438279 shutdown_connections 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/2300438279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b810e0f0 0x7ff1b81104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/2300438279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100e60 0x7ff1b810dbb0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/2300438279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b8100540 0x7ff1b8100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 -- 172.21.15.38:0/2300438279 >> 172.21.15.38:0/2300438279 conn(0x7ff1b8077740 msgr2=0x7ff1b8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 -- 172.21.15.38:0/2300438279 shutdown_connections 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.285+0000 7ff1bcb35640 1 -- 172.21.15.38:0/2300438279 wait complete. 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 Processor -- start 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 -- start start 2024-04-22T00:09:13.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b8100e60 0x7ff1b819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 0x7ff1b819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113d20 con 0x7ff1b8100540 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113ba0 con 0x7ff1b810e0f0 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1bcb35640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b8113ea0 con 0x7ff1b8100e60 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42078/0 (socket says 172.21.15.38:42078) 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1b659c640 1 -- 172.21.15.38:0/1464421068 learned_addr learned my addr 172.21.15.38:0/1464421068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 0x7ff1b819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.286+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b8100e60 0x7ff1b819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 -- 172.21.15.38:0/1464421068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b8100e60 msgr2=0x7ff1b819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b8100e60 0x7ff1b819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 -- 172.21.15.38:0/1464421068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 msgr2=0x7ff1b819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 0x7ff1b819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 -- 172.21.15.38:0/1464421068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b819f210 con 0x7ff1b8100540 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 0x7ff1b819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000d950 tx=0x7ff1b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1a77fe640 1 -- 172.21.15.38:0/1464421068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0017070 con 0x7ff1b8100540 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1a77fe640 1 -- 172.21.15.38:0/1464421068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b0009c30 con 0x7ff1b8100540 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b819f500 con 0x7ff1b8100540 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1a77fe640 1 -- 172.21.15.38:0/1464421068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000f6b0 con 0x7ff1b8100540 2024-04-22T00:09:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.287+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b81a6d90 con 0x7ff1b8100540 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.289+0000 7ff1a77fe640 1 -- 172.21.15.38:0/1464421068 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff1b00109d0 con 0x7ff1b8100540 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.289+0000 7ff1bf7f4740 1 Processor -- start 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 -- start start 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d379400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1a77fe640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d384910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d2dd530 con 0x55ce1d379940 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d285830 con 0x55ce1d379020 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d2d94f0 con 0x55ce1d382520 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42094/0 (socket says 172.21.15.38:42094) 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1123432807 learned_addr learned my addr 172.21.15.38:0/1123432807 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d379400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1a77fe640 1 -- 172.21.15.38:0/1464421068 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7ff1b00cf5a0 con 0x7ff1b8100540 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d384910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1123432807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 msgr2=0x55ce1d384910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d384910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1123432807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 msgr2=0x55ce1d379400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d379400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1123432807 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1d2a92c0 con 0x55ce1d379940 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d379400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8005cc0 tx=0x7ff1a80139f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.290+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac02c170 tx=0x7ff1ac02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e500e551a96ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1a4ff9640 1 -- 172.21.15.38:0/1123432807 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac02ca80 con 0x55ce1d379940 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1a4ff9640 1 -- 172.21.15.38:0/1123432807 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1ac02cc20 con 0x55ce1d379940 2024-04-22T00:09:13.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1a4ff9640 1 -- 172.21.15.38:0/1123432807 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0048c0 con 0x55ce1d379940 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1123432807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 msgr2=0x55ce1d381fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac02c170 tx=0x7ff1ac02c5d0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1123432807 shutdown_connections 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d384910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d381fe0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1123432807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d379400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1123432807 >> 172.21.15.38:0/1123432807 conn(0x55ce1d373750 msgr2=0x55ce1d373b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1123432807 shutdown_connections 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.291+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1123432807 wait complete. 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 -- start start 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 Processor -- start 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 0x55ce1d37f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 0x55ce1d37f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d3852b0 con 0x55ce1d379020 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d298f10 con 0x55ce1d382520 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d385430 con 0x55ce1d379940 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1bf7f4740 10 client.0 _ll_register_callbacks cb 0x55ce1cff0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42108/0 (socket says 172.21.15.38:42108) 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 -- 172.21.15.38:0/3406971933 learned_addr learned my addr 172.21.15.38:0/3406971933 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 0x55ce1d37f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 0x55ce1d37f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 -- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 msgr2=0x55ce1d37f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 0x55ce1d37f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 -- 172.21.15.38:0/3406971933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 msgr2=0x55ce1d37f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 0x55ce1d37f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 -- 172.21.15.38:0/3406971933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1d26ac80 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 0x55ce1d37f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 0x55ce1d37f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff1b659c640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0098040 tx=0x7ff1b009b2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802a9493d658bc25 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.292+0000 7ff19dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0020070 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7ff1b009a1a0 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1ac02cc20 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff1b0086070 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 10 client.17946 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ce1cff13f0 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1ac02cc20 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff1b0099c30 con 0x55ce1d379020 2024-04-22T00:09:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 10 client.17946 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 10 client.17946 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 20 client.17946 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3923d0 0x55ce1d394840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 4 client.17946 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.293+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce1d2eaa70 con 0x55ce1d3923d0 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.294+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3923d0 0x55ce1d394840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.294+0000 7ff19dffb640 10 client.17946 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.294+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3923d0 0x55ce1d394840 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d07dcfc9a5b59318 server_cookie=52a1cc2cae4125b8 in_seq=0 out_seq=0 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff19dffb640 1 -- 172.21.15.38:0/3406971933 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce1d2eaa70 con 0x55ce1d3923d0 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff19dffb640 10 client.17946 handle_command_reply: tid=0 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 client.17946 shutdown 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 msgr2=0x55ce1d381e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0098040 tx=0x7ff1b009b2a0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 shutdown_connections 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3923d0 0x55ce1d394840 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d382520 0x55ce1d37f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379940 0x55ce1d37f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff19dffb640 0 client.17946 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/3406971933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379020 0x55ce1d381e30 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 >> 172.21.15.38:0/3406971933 conn(0x55ce1d373750 msgr2=0x55ce1d3754a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 shutdown_connections 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/3406971933 wait complete. 2024-04-22T00:09:13.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.295+0000 7ff1bf7f4740 20 client.17946 trim_cache size 0 max 16384 2024-04-22T00:09:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 Processor -- start 2024-04-22T00:09:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 -- start start 2024-04-22T00:09:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379020 0x55ce1d38e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379940 0x55ce1d3818f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b0ea0 con 0x55ce1d382520 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b29e0 con 0x55ce1d379940 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.308+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b1570 con 0x55ce1d379020 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42124/0 (socket says 172.21.15.38:42124) 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 -- 172.21.15.38:0/1883523958 learned_addr learned my addr 172.21.15.38:0/1883523958 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379940 0x55ce1d3818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379020 0x55ce1d38e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 -- 172.21.15.38:0/1883523958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379020 msgr2=0x55ce1d38e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379020 0x55ce1d38e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 -- 172.21.15.38:0/1883523958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379940 msgr2=0x55ce1d3818f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379940 0x55ce1d3818f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 -- 172.21.15.38:0/1883523958 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1d0bb7d0 con 0x55ce1d382520 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff1b806aed0 tx=0x7ff1a8013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b737b4907782ec65 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff19ffff640 1 -- 172.21.15.38:0/1883523958 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8004d20 con 0x55ce1d382520 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff19ffff640 1 -- 172.21.15.38:0/1883523958 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a8005500 con 0x55ce1d382520 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.309+0000 7ff19ffff640 1 -- 172.21.15.38:0/1883523958 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a80207b0 con 0x55ce1d382520 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1883523958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 msgr2=0x55ce1d381e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff1b806aed0 tx=0x7ff1a8013f90 comp rx=0 tx=0).stop 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1883523958 shutdown_connections 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d382520 0x55ce1d381e30 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379940 0x55ce1d3818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1883523958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d379020 0x55ce1d38e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1883523958 >> 172.21.15.38:0/1883523958 conn(0x55ce1d373750 msgr2=0x55ce1d3753c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1883523958 shutdown_connections 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1883523958 wait complete. 2024-04-22T00:09:13.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- start start 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 Processor -- start 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d386010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d386a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b2310 con 0x55ce1d379940 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81b1c40 con 0x55ce1d379020 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.310+0000 7ff1bf7f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b81aeea0 con 0x55ce1d382520 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1bf7f4740 10 client.0 _ll_register_callbacks cb 0x7ff1b00979e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42134/0 (socket says 172.21.15.38:42134) 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1364604727 learned_addr learned my addr 172.21.15.38:0/1364604727 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d386010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d386a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 msgr2=0x55ce1d386a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d386a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1364604727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 msgr2=0x55ce1d386010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d386010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 -- 172.21.15.38:0/1364604727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1d3f7980 con 0x55ce1d379940 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d386010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b6d9d640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d386a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1b5d9b640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0055e0 tx=0x7ff1ac0054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b86ffc3c9ff511 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff19dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac04dd50 con 0x55ce1d379940 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7ff1ac0040c0 con 0x55ce1d379940 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1a8005500 con 0x55ce1d379940 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff1ac0042c0 con 0x55ce1d379940 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1bf7f4740 10 client.17952 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.311+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ce1d3f8b60 con 0x55ce1d379940 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a8005500 con 0x55ce1d379940 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff1ac0035f0 con 0x55ce1d379940 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 10 client.17952 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 10 client.17952 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 20 client.17952 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3f8e50 0x55ce1d3fb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 4 client.17952 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce1d2e4dc0 con 0x55ce1d3f8e50 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3f8e50 0x55ce1d3fb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff19dffb640 10 client.17952 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff1b659c640 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3f8e50 0x55ce1d3fb240 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7f66d985750bfc04 server_cookie=6402bd724fbdc7c8 in_seq=0 out_seq=0 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.312+0000 7ff19dffb640 1 -- 172.21.15.38:0/1364604727 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce1d2e4dc0 con 0x55ce1d3f8e50 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff19dffb640 10 client.17952 handle_command_reply: tid=0 2024-04-22T00:09:13.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 client.17952 shutdown 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 msgr2=0x55ce1d386550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0055e0 tx=0x7ff1ac0054d0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 shutdown_connections 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55ce1d3f8e50 0x55ce1d3fb240 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce1d382520 0x55ce1d386a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce1d379940 0x55ce1d386550 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 --2- 172.21.15.38:0/1364604727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ce1d379020 0x55ce1d386010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff19dffb640 0 client.17952 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 >> 172.21.15.38:0/1364604727 conn(0x55ce1d373750 msgr2=0x55ce1d379e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 shutdown_connections 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 1 -- 172.21.15.38:0/1364604727 wait complete. 2024-04-22T00:09:13.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bf7f4740 20 client.17952 trim_cache size 0 max 16384 2024-04-22T00:09:13.317 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:13.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 msgr2=0x7ff1a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8005cc0 tx=0x7ff1a80139f0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 msgr2=0x7ff1b819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000d950 tx=0x7ff1b000de10 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 shutdown_connections 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1a00a2d00 0x7ff1a00a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b810e0f0 0x7ff1b819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b8100e60 0x7ff1b819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 --2- 172.21.15.38:0/1464421068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1b8100540 0x7ff1b819a220 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 >> 172.21.15.38:0/1464421068 conn(0x7ff1b8077740 msgr2=0x7ff1b810f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 shutdown_connections 2024-04-22T00:09:13.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.313+0000 7ff1bcb35640 1 -- 172.21.15.38:0/1464421068 wait complete. 2024-04-22T00:09:13.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.430+0000 7f83e6220640 1 -- 172.21.15.38:0/3501291530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 msgr2=0x7f83e0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.430+0000 7f83e6220640 1 --2- 172.21.15.38:0/3501291530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0100aa0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009930 tx=0x7f83c802bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- 172.21.15.38:0/3501291530 shutdown_connections 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- 172.21.15.38:0/3501291530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 0x7f83e010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- 172.21.15.38:0/3501291530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0100aa0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- 172.21.15.38:0/3501291530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 0x7f83e0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- 172.21.15.38:0/3501291530 >> 172.21.15.38:0/3501291530 conn(0x7f83e00776a0 msgr2=0x7f83e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- 172.21.15.38:0/3501291530 shutdown_connections 2024-04-22T00:09:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- 172.21.15.38:0/3501291530 wait complete. 2024-04-22T00:09:13.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 Processor -- start 2024-04-22T00:09:13.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- start start 2024-04-22T00:09:13.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 0x7f83e0199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 0x7f83e01a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0110a40 con 0x7f83e0100620 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01108c0 con 0x7f83e0106630 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83e6220640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0110bc0 con 0x7f83e0100fe0 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.431+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42154/0 (socket says 172.21.15.38:42154) 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 -- 172.21.15.38:0/623438169 learned_addr learned my addr 172.21.15.38:0/623438169 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83d7fff640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 0x7f83e0199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83dffff640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 0x7f83e01a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 -- 172.21.15.38:0/623438169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 msgr2=0x7f83e0199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 0x7f83e0199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 -- 172.21.15.38:0/623438169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 msgr2=0x7f83e01a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 0x7f83e01a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 -- 172.21.15.38:0/623438169 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e01a43e0 con 0x7f83e0100620 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83df7fe640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d950 tx=0x7f83d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83dd7fa640 1 -- 172.21.15.38:0/623438169 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0017030 con 0x7f83e0100620 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83dd7fa640 1 -- 172.21.15.38:0/623438169 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d0004020 con 0x7f83e0100620 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83dd7fa640 1 -- 172.21.15.38:0/623438169 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00047a0 con 0x7f83e0100620 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e01a45b0 con 0x7f83e0100620 2024-04-22T00:09:13.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.432+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e01a4ad0 con 0x7f83e0100620 2024-04-22T00:09:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 Processor -- start 2024-04-22T00:09:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83dd7fa640 1 -- 172.21.15.38:0/623438169 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f83d00041d0 con 0x7f83e0100620 2024-04-22T00:09:13.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 -- start start 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49020 0x555a3ad49400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad54910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a3acad530 con 0x555a3ad49940 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555a3ac55830 con 0x555a3ad49020 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83e8e09740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a3aca94f0 con 0x555a3ad52520 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42162/0 (socket says 172.21.15.38:42162) 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 -- 172.21.15.38:0/1597844110 learned_addr learned my addr 172.21.15.38:0/1597844110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83d7fff640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad54910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83dffff640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49020 0x555a3ad49400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83dd7fa640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 0x7f83cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 -- 172.21.15.38:0/1597844110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 msgr2=0x555a3ad54910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad54910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 -- 172.21.15.38:0/1597844110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49020 msgr2=0x555a3ad49400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49020 0x555a3ad49400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83df7fe640 1 -- 172.21.15.38:0/1597844110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a3ac792c0 con 0x555a3ad49940 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83d7fff640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad54910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.435+0000 7f83dffff640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 0x7f83cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83dd7fa640 1 -- 172.21.15.38:0/623438169 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f83d00cf1e0 con 0x7f83e0100620 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83df7fe640 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f83d00cc4b0 tx=0x7f83d006a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46111202fd60fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83d5ffb640 1 -- 172.21.15.38:0/1597844110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d009b0e0 con 0x555a3ad49940 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83dffff640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 0x7f83cc0a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83d8005fd0 tx=0x7f83d8004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83d5ffb640 1 -- 172.21.15.38:0/1597844110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d009ba10 con 0x555a3ad49940 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.436+0000 7f83d5ffb640 1 -- 172.21.15.38:0/1597844110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00d2e60 con 0x555a3ad49940 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- 172.21.15.38:0/1597844110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 msgr2=0x555a3ad51fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f83d00cc4b0 tx=0x7f83d006a240 comp rx=0 tx=0).stop 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- 172.21.15.38:0/1597844110 shutdown_connections 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad54910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad51fe0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- 172.21.15.38:0/1597844110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49020 0x555a3ad49400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- 172.21.15.38:0/1597844110 >> 172.21.15.38:0/1597844110 conn(0x555a3ad43750 msgr2=0x555a3ad43b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- 172.21.15.38:0/1597844110 shutdown_connections 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- 172.21.15.38:0/1597844110 wait complete. 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- start start 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 Processor -- start 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad51f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad4fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a3ad552b0 con 0x555a3ad49940 2024-04-22T00:09:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555a3ac68f10 con 0x555a3ad52520 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a3ad55430 con 0x555a3ad49020 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83e8e09740 10 client.0 _ll_register_callbacks cb 0x555a3a9c0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42168/0 (socket says 172.21.15.38:42168) 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.437+0000 7f83d7fff640 1 -- 172.21.15.38:0/2239343270 learned_addr learned my addr 172.21.15.38:0/2239343270 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83df7fe640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad51f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83dffff640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad4fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 -- 172.21.15.38:0/2239343270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 msgr2=0x555a3ad51f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad51f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 -- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 msgr2=0x555a3ad4fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad4fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 -- 172.21.15.38:0/2239343270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a3ac3ac80 con 0x555a3ad49940 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83d7fff640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f83c802cfd0 tx=0x7f83c802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca9c62f4982e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83beffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8004240 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f83c80043e0 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d009ba10 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83c80046c0 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83e8e09740 10 client.17964 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555a3a9c13f0 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.438+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d009ba10 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f83c80035f0 con 0x555a3ad49940 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 10 client.17964 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 10 client.17964 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 20 client.17964 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3ad62660 0x555a3ad64ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 4 client.17964 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a3acbaa70 con 0x555a3ad62660 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83df7fe640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3ad62660 0x555a3ad64ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83beffd640 10 client.17964 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.439+0000 7f83df7fe640 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3ad62660 0x555a3ad64ac0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c2bbed2674b2e7c server_cookie=52c7695ef8916df8 in_seq=0 out_seq=0 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83beffd640 1 -- 172.21.15.38:0/2239343270 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a3acbaa70 con 0x555a3ad62660 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83beffd640 10 client.17964 handle_command_reply: tid=0 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 client.17964 shutdown 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 msgr2=0x555a3ad4f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f83c802cfd0 tx=0x7f83c802ce30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 shutdown_connections 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad4fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3ad62660 0x555a3ad64ac0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad4f310 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 --2- 172.21.15.38:0/2239343270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad51f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 >> 172.21.15.38:0/2239343270 conn(0x555a3ad43750 msgr2=0x555a3ad454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83beffd640 0 client.17964 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 shutdown_connections 2024-04-22T00:09:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 1 -- 172.21.15.38:0/2239343270 wait complete. 2024-04-22T00:09:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.440+0000 7f83e8e09740 20 client.17964 trim_cache size 0 max 16384 2024-04-22T00:09:13.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.453+0000 7f83e8e09740 1 Processor -- start 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 -- start start 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 0x555a3ad426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad6a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01aed10 con 0x555a3ad49020 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01b0850 con 0x555a3ad49940 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83e8e09740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01af3e0 con 0x555a3ad52520 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42174/0 (socket says 172.21.15.38:42174) 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 -- 172.21.15.38:0/3794202711 learned_addr learned my addr 172.21.15.38:0/3794202711 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83d7fff640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 0x555a3ad426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 -- 172.21.15.38:0/3794202711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 msgr2=0x555a3ad6a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83dffff640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad6a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad6a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 -- 172.21.15.38:0/3794202711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 msgr2=0x555a3ad426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 0x555a3ad426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 -- 172.21.15.38:0/3794202711 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a3aa8b7d0 con 0x555a3ad49020 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83d7fff640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 0x555a3ad426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83dffff640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad6a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83df7fe640 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f83d00cc5f0 tx=0x7f83d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1248fb7021c352c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83d57fa640 1 -- 172.21.15.38:0/3794202711 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d009ac70 con 0x555a3ad49020 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83d57fa640 1 -- 172.21.15.38:0/3794202711 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d009ba10 con 0x555a3ad49020 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.454+0000 7f83d57fa640 1 -- 172.21.15.38:0/3794202711 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d009ae50 con 0x555a3ad49020 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- 172.21.15.38:0/3794202711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 msgr2=0x555a3ad5e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f83d00cc5f0 tx=0x7f83d0000bc0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- 172.21.15.38:0/3794202711 shutdown_connections 2024-04-22T00:09:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad52520 0x555a3ad6a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad49940 0x555a3ad426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 --2- 172.21.15.38:0/3794202711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49020 0x555a3ad5e170 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- 172.21.15.38:0/3794202711 >> 172.21.15.38:0/3794202711 conn(0x555a3ad43750 msgr2=0x555a3ad41fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- 172.21.15.38:0/3794202711 shutdown_connections 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- 172.21.15.38:0/3794202711 wait complete. 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 -- start start 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.455+0000 7f83e8e09740 1 Processor -- start 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad56140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad56bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01b0180 con 0x555a3ad49940 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01afab0 con 0x555a3ad52520 2024-04-22T00:09:13.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83e01acd10 con 0x555a3ad49020 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 10 client.0 _ll_register_callbacks cb 0x7f83c802c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42190/0 (socket says 172.21.15.38:42190) 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 -- 172.21.15.38:0/4129463673 learned_addr learned my addr 172.21.15.38:0/4129463673 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83dffff640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad56bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83df7fe640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad56140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 -- 172.21.15.38:0/4129463673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 msgr2=0x555a3ad56140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad56140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 -- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 msgr2=0x555a3ad56bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad56bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 -- 172.21.15.38:0/4129463673 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a3adc7c50 con 0x555a3ad49940 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83dffff640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad56bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83d7fff640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x555a3ad508e0 tx=0x7f83c8003670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2892c2ee542ccfb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83beffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8004a60 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f83c8004c00 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d009ba10 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83c8062050 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 10 client.17970 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.456+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555a3adc8ef0 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d009ba10 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f83c805d030 con 0x555a3ad49940 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 10 client.17970 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 10 client.17970 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 20 client.17970 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3adc9710 0x555a3adcbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 4 client.17970 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:13.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a3acb4dc0 con 0x555a3adc9710 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83df7fe640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3adc9710 0x555a3adcbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83beffd640 10 client.17970 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.457+0000 7f83df7fe640 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3adc9710 0x555a3adcbb30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b61b44f1982c4a4b server_cookie=58b8d7203b853280 in_seq=0 out_seq=0 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83beffd640 1 -- 172.21.15.38:0/4129463673 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a3acb4dc0 con 0x555a3adc9710 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83beffd640 10 client.17970 handle_command_reply: tid=0 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 client.17970 shutdown 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 msgr2=0x555a3ad56680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x555a3ad508e0 tx=0x7f83c8003670 comp rx=0 tx=0).stop 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 shutdown_connections 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555a3ad52520 0x555a3ad56bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x555a3adc9710 0x555a3adcbb30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a3ad49940 0x555a3ad56680 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 --2- 172.21.15.38:0/4129463673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a3ad49020 0x555a3ad56140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83beffd640 0 client.17970 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:13.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 >> 172.21.15.38:0/4129463673 conn(0x555a3ad43750 msgr2=0x555a3ad45410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 shutdown_connections 2024-04-22T00:09:13.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 1 -- 172.21.15.38:0/4129463673 wait complete. 2024-04-22T00:09:13.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e8e09740 20 client.17970 trim_cache size 0 max 16384 2024-04-22T00:09:13.462 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 msgr2=0x7f83cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 0x7f83cc0a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83d8005fd0 tx=0x7f83d8004a40 comp rx=0 tx=0).stop 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 msgr2=0x7f83e0198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d950 tx=0x7f83d000de10 comp rx=0 tx=0).stop 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 shutdown_connections 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f83cc0a2d00 0x7f83cc0a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83e0106630 0x7f83e01a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83e0100fe0 0x7f83e0199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 --2- 172.21.15.38:0/623438169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100620 0x7f83e0198e20 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.458+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 >> 172.21.15.38:0/623438169 conn(0x7f83e00776a0 msgr2=0x7f83e01031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.459+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 shutdown_connections 2024-04-22T00:09:13.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.459+0000 7f83e6220640 1 -- 172.21.15.38:0/623438169 wait complete. 2024-04-22T00:09:13.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 -- 172.21.15.38:0/3161898124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5201077a0 msgr2=0x7fe520107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 --2- 172.21.15.38:0/3161898124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5201077a0 0x7fe520107b80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe518009930 tx=0x7fe51802bc20 comp rx=0 tx=0).stop 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 -- 172.21.15.38:0/3161898124 shutdown_connections 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 --2- 172.21.15.38:0/3161898124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe520102150 0x7fe52010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 --2- 172.21.15.38:0/3161898124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 0x7fe520101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 --2- 172.21.15.38:0/3161898124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5201077a0 0x7fe520107b80 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 -- 172.21.15.38:0/3161898124 >> 172.21.15.38:0/3161898124 conn(0x7fe5200fd2a0 msgr2=0x7fe5200ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 -- 172.21.15.38:0/3161898124 shutdown_connections 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.595+0000 7fe52525f640 1 -- 172.21.15.38:0/3161898124 wait complete. 2024-04-22T00:09:13.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 Processor -- start 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 -- start start 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 0x7fe520197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 0x7fe52019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe520111b70 con 0x7fe520102150 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201119f0 con 0x7fe5201077a0 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.596+0000 7fe52525f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe520111cf0 con 0x7fe520101790 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42208/0 (socket says 172.21.15.38:42208) 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 -- 172.21.15.38:0/1477573002 learned_addr learned my addr 172.21.15.38:0/1477573002 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 0x7fe520197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 0x7fe52019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 -- 172.21.15.38:0/1477573002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 msgr2=0x7fe520197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 0x7fe520197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 -- 172.21.15.38:0/1477573002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 msgr2=0x7fe52019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 0x7fe52019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 -- 172.21.15.38:0/1477573002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52019cfe0 con 0x7fe520102150 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 0x7fe52019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe51e59c640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c920 tx=0x7fe50800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe517fff640 1 -- 172.21.15.38:0/1477573002 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508007c70 con 0x7fe520102150 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52019d2d0 con 0x7fe520102150 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe517fff640 1 -- 172.21.15.38:0/1477573002 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe508007e10 con 0x7fe520102150 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe517fff640 1 -- 172.21.15.38:0/1477573002 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508018890 con 0x7fe520102150 2024-04-22T00:09:13.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.597+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5201a4b60 con 0x7fe520102150 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe517fff640 1 -- 172.21.15.38:0/1477573002 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe508018b30 con 0x7fe520102150 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 Processor -- start 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 -- start start 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559375be0530 con 0x559375c85520 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559375b88830 con 0x559375c7c020 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe527e42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559375bdc4f0 con 0x559375c7c940 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51f59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50784/0 (socket says 172.21.15.38:50784) 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42220/0 (socket says 172.21.15.38:42220) 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51f59e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:37740/0 (socket says 172.21.15.38:37740) 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.600+0000 7fe51ed9d640 1 -- 172.21.15.38:0/4154951641 learned_addr learned my addr 172.21.15.38:0/4154951641 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe517fff640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 0x7fe50c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 -- 172.21.15.38:0/4154951641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 msgr2=0x559375c84fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 -- 172.21.15.38:0/4154951641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 msgr2=0x559375c87910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 -- 172.21.15.38:0/4154951641 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559375bac2c0 con 0x559375c7c020 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51e59c640 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe517fff640 1 -- 172.21.15.38:0/1477573002 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7fe5080cfa10 con 0x7fe520102150 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 0x7fe50c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe51f59e640 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe510008ea0 tx=0x7fe510008f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd0295e6b577fcf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe5157fa640 1 -- 172.21.15.38:0/4154951641 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510011e60 con 0x559375c7c020 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe5157fa640 1 -- 172.21.15.38:0/4154951641 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51001adc0 con 0x559375c7c020 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.601+0000 7fe5157fa640 1 -- 172.21.15.38:0/4154951641 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510019c10 con 0x559375c7c020 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- 172.21.15.38:0/4154951641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 msgr2=0x559375c7c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 0x7fe50c0a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe51001e000 tx=0x7fe51001f000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe510008ea0 tx=0x7fe510008f90 comp rx=0 tx=0).stop 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- 172.21.15.38:0/4154951641 shutdown_connections 2024-04-22T00:09:13.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375c87910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c84fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 --2- 172.21.15.38:0/4154951641 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7c400 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- 172.21.15.38:0/4154951641 >> 172.21.15.38:0/4154951641 conn(0x559375c76750 msgr2=0x559375c76b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- 172.21.15.38:0/4154951641 shutdown_connections 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- 172.21.15.38:0/4154951641 wait complete. 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.602+0000 7fe527e42740 1 -- start start 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 Processor -- start 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c940 0x559375c821c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c85520 0x559375c82700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559375c88310 con 0x559375c7c020 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559375b9bf10 con 0x559375c7c940 2024-04-22T00:09:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559375c88490 con 0x559375c85520 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe527e42740 10 client.0 _ll_register_callbacks cb 0x5593758f3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42230/0 (socket says 172.21.15.38:42230) 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 -- 172.21.15.38:0/2915734021 learned_addr learned my addr 172.21.15.38:0/2915734021 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51e59c640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c940 0x559375c821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 -- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c85520 msgr2=0x559375c82700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c85520 0x559375c82700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 -- 172.21.15.38:0/2915734021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c940 msgr2=0x559375c821c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c940 0x559375c821c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 -- 172.21.15.38:0/2915734021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559375b6dc80 con 0x559375c7c020 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe51802c160 tx=0x7fe518005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7617ae7fc125b261 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe5067fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518042070 con 0x559375c7c020 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.603+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7fe5180046c0 con 0x559375c7c020 2024-04-22T00:09:13.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe51001adc0 con 0x559375c7c020 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe5180048c0 con 0x559375c7c020 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 10 client.17982 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5593758f43f0 con 0x559375c7c020 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51001adc0 con 0x559375c7c020 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe5180037b0 con 0x559375c7c020 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 10 client.17982 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 10 client.17982 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 20 client.17982 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375c95550 0x559375c979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 4 client.17982 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559375beda70 con 0x559375c95550 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.604+0000 7fe51e59c640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375c95550 0x559375c979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.605+0000 7fe5067fc640 10 client.17982 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:13.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.605+0000 7fe51e59c640 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375c95550 0x559375c979c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4bab19916d8c2cc6 server_cookie=c465a499c89f7ad0 in_seq=0 out_seq=0 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.605+0000 7fe5067fc640 1 -- 172.21.15.38:0/2915734021 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559375beda70 con 0x559375c95550 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.605+0000 7fe5067fc640 10 client.17982 handle_command_reply: tid=0 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 client.17982 shutdown 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 msgr2=0x559375c84e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fe51802c160 tx=0x7fe518005e00 comp rx=0 tx=0).stop 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 shutdown_connections 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375c95550 0x559375c979c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c85520 0x559375c82700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c940 0x559375c821c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe5067fc640 0 client.17982 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 --2- 172.21.15.38:0/2915734021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c84e30 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 >> 172.21.15.38:0/2915734021 conn(0x559375c76750 msgr2=0x559375c784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 shutdown_connections 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 1 -- 172.21.15.38:0/2915734021 wait complete. 2024-04-22T00:09:13.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.606+0000 7fe527e42740 20 client.17982 trim_cache size 0 max 16384 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.619+0000 7fe527e42740 1 Processor -- start 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.619+0000 7fe527e42740 1 -- start start 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c848f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 0x559375c84e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201aec50 con 0x559375c7c020 2024-04-22T00:09:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201b0790 con 0x559375c85520 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe527e42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201af320 con 0x559375c7c940 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42246/0 (socket says 172.21.15.38:42246) 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 -- 172.21.15.38:0/3853123117 learned_addr learned my addr 172.21.15.38:0/3853123117 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 -- 172.21.15.38:0/3853123117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 msgr2=0x559375c848f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51e59c640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c848f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51f59e640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 0x559375c84e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c848f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 -- 172.21.15.38:0/3853123117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 msgr2=0x559375c84e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 0x559375c84e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 -- 172.21.15.38:0/3853123117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593759be7d0 con 0x559375c7c020 2024-04-22T00:09:13.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51e59c640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51f59e640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 0x559375c84e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.620+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe518000c00 tx=0x7fe518009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28974d11282fbab server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe514ff9640 1 -- 172.21.15.38:0/3853123117 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518042070 con 0x559375c7c020 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe514ff9640 1 -- 172.21.15.38:0/3853123117 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51802cd30 con 0x559375c7c020 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe514ff9640 1 -- 172.21.15.38:0/3853123117 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518004f10 con 0x559375c7c020 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 -- 172.21.15.38:0/3853123117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 msgr2=0x559375c91200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fe518000c00 tx=0x7fe518009230 comp rx=0 tx=0).stop 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 -- 172.21.15.38:0/3853123117 shutdown_connections 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c85520 0x559375c84e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 --2- 172.21.15.38:0/3853123117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c7c020 0x559375c91200 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 -- 172.21.15.38:0/3853123117 >> 172.21.15.38:0/3853123117 conn(0x559375c76750 msgr2=0x559375c7d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 -- 172.21.15.38:0/3853123117 shutdown_connections 2024-04-22T00:09:13.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.621+0000 7fe527e42740 1 -- 172.21.15.38:0/3853123117 wait complete. 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 -- start start 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 Processor -- start 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c7f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201b00c0 con 0x559375c85520 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201af9f0 con 0x559375c7c020 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5201acc50 con 0x559375c7c940 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe527e42740 10 client.0 _ll_register_callbacks cb 0x7fe518005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42252/0 (socket says 172.21.15.38:42252) 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 -- 172.21.15.38:0/1550187898 learned_addr learned my addr 172.21.15.38:0/1550187898 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51e59c640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c7f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 -- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 msgr2=0x559375c7f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c7f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 -- 172.21.15.38:0/1550187898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 msgr2=0x559375c7f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.622+0000 7fe51f59e640 1 -- 172.21.15.38:0/1550187898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559375cfb040 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe51f59e640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe51001cf50 tx=0x7fe51001cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19c7f05cc4041a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510011e60 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe51802cd30 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7fe510020040 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe510038050 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 10 client.17988 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559375cfc0f0 con 0x559375c85520 2024-04-22T00:09:13.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51802cd30 con 0x559375c85520 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe510004c80 con 0x559375c85520 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 10 client.17988 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 10 client.17988 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 20 client.17988 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375cfc5c0 0x559375cfe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 4 client.17988 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.623+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559375c82a90 con 0x559375cfc5c0 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375cfc5c0 0x559375cfe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe51ed9d640 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375cfc5c0 0x559375cfe9e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7054defde6a62ae4 server_cookie=36769657ed37389a in_seq=0 out_seq=0 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe5067fc640 10 client.17988 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe5067fc640 1 -- 172.21.15.38:0/1550187898 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559375c82a90 con 0x559375cfc5c0 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe5067fc640 10 client.17988 handle_command_reply: tid=0 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe527e42740 1 client.17988 shutdown 2024-04-22T00:09:13.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 msgr2=0x559375cfab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.624+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fe51001cf50 tx=0x7fe51001cf80 comp rx=0 tx=0).stop 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 shutdown_connections 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559375cfc5c0 0x559375cfe9e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559375c85520 0x559375cfab00 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559375c7c940 0x559375c7f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe5067fc640 0 client.17988 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 --2- 172.21.15.38:0/1550187898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559375c7c020 0x559375c7f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 >> 172.21.15.38:0/1550187898 conn(0x559375c76750 msgr2=0x559375c7ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 shutdown_connections 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 1 -- 172.21.15.38:0/1550187898 wait complete. 2024-04-22T00:09:13.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe527e42740 20 client.17988 trim_cache size 0 max 16384 2024-04-22T00:09:13.628 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:13.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 msgr2=0x7fe50c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 0x7fe50c0a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe51001e000 tx=0x7fe51001f000 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 msgr2=0x7fe520198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c920 tx=0x7fe50800cde0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 shutdown_connections 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe50c0a2d00 0x7fe50c0a51c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5201077a0 0x7fe52019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe520102150 0x7fe520198530 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 --2- 172.21.15.38:0/1477573002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe520101790 0x7fe520197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 >> 172.21.15.38:0/1477573002 conn(0x7fe5200fd2a0 msgr2=0x7fe5200fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 shutdown_connections 2024-04-22T00:09:13.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.625+0000 7fe52525f640 1 -- 172.21.15.38:0/1477573002 wait complete. 2024-04-22T00:09:13.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:09:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 -- 172.21.15.38:0/125397328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e806a6e0 msgr2=0x7f22e810dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 --2- 172.21.15.38:0/125397328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e806a6e0 0x7f22e810dc10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009900 tx=0x7f22d802bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 -- 172.21.15.38:0/125397328 shutdown_connections 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 --2- 172.21.15.38:0/125397328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e8110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 --2- 172.21.15.38:0/125397328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e806a6e0 0x7f22e810dc10 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 --2- 172.21.15.38:0/125397328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e8069dc0 0x7f22e806a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 -- 172.21.15.38:0/125397328 >> 172.21.15.38:0/125397328 conn(0x7f22e80776a0 msgr2=0x7f22e8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 -- 172.21.15.38:0/125397328 shutdown_connections 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.740+0000 7f22ef397640 1 -- 172.21.15.38:0/125397328 wait complete. 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 Processor -- start 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 -- start start 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e806a6e0 0x7f22e819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113d10 con 0x7f22e8069dc0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113b90 con 0x7f22e806a6e0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ef397640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22e8113e90 con 0x7f22e810e150 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ed133640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ed133640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53372/0 (socket says 172.21.15.38:53372) 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.741+0000 7f22ed133640 1 -- 172.21.15.38:0/3442345587 learned_addr learned my addr 172.21.15.38:0/3442345587 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ec932640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e806a6e0 0x7f22e819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed934640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 -- 172.21.15.38:0/3442345587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 msgr2=0x7f22e819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 -- 172.21.15.38:0/3442345587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e806a6e0 msgr2=0x7f22e819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e806a6e0 0x7f22e819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 -- 172.21.15.38:0/3442345587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e819f200 con 0x7f22e8069dc0 2024-04-22T00:09:13.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed934640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ed133640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d950 tx=0x7f22e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22de7fc640 1 -- 172.21.15.38:0/3442345587 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4017070 con 0x7f22e8069dc0 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22de7fc640 1 -- 172.21.15.38:0/3442345587 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e4009c30 con 0x7f22e8069dc0 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e819f4f0 con 0x7f22e8069dc0 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22de7fc640 1 -- 172.21.15.38:0/3442345587 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e400f6b0 con 0x7f22e8069dc0 2024-04-22T00:09:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.742+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e81a6d80 con 0x7f22e8069dc0 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.744+0000 7f22de7fc640 1 -- 172.21.15.38:0/3442345587 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f22e40109d0 con 0x7f22e8069dc0 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 Processor -- start 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 -- start start 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd712fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd713520 0x55c4bd715910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd66e530 con 0x55c4bd70a020 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd616830 con 0x55c4bd70a940 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd66a4f0 con 0x55c4bd713520 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22de7fc640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 0x7f22d40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53388/0 (socket says 172.21.15.38:53388) 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 -- 172.21.15.38:0/1408031153 learned_addr learned my addr 172.21.15.38:0/1408031153 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed133640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd712fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ec932640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd713520 0x55c4bd715910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22de7fc640 1 -- 172.21.15.38:0/3442345587 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f22e40cf4e0 con 0x7f22e8069dc0 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 0x7f22d40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 -- 172.21.15.38:0/1408031153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd713520 msgr2=0x55c4bd715910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd713520 0x55c4bd715910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 -- 172.21.15.38:0/1408031153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 msgr2=0x55c4bd712fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd712fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed934640 1 -- 172.21.15.38:0/1408031153 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4bd63a2c0 con 0x55c4bd70a020 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.745+0000 7f22ed133640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd712fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.746+0000 7f22ed934640 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f22e0015dc0 tx=0x7f22e00127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9a348dab7a5ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.746+0000 7f22ed934640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 0x7f22d40a5240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22e0012cc0 tx=0x7f22e001b000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.746+0000 7f22d37fe640 1 -- 172.21.15.38:0/1408031153 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e00157f0 con 0x55c4bd70a020 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.746+0000 7f22d37fe640 1 -- 172.21.15.38:0/1408031153 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e0010c10 con 0x55c4bd70a020 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.746+0000 7f22d37fe640 1 -- 172.21.15.38:0/1408031153 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e001e9c0 con 0x55c4bd70a020 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1408031153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 msgr2=0x55c4bd70a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f22e0015dc0 tx=0x7f22e00127c0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1408031153 shutdown_connections 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd713520 0x55c4bd715910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd712fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1408031153 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd70a400 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1408031153 >> 172.21.15.38:0/1408031153 conn(0x55c4bd704750 msgr2=0x55c4bd704b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1408031153 shutdown_connections 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1408031153 wait complete. 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- start start 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 Processor -- start 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd712ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd710280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd7162b0 con 0x55c4bd713520 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd629f10 con 0x55c4bd70a940 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.747+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c4bd716430 con 0x55c4bd70a020 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22f1f7c740 10 client.0 _ll_register_callbacks cb 0x55c4bd381be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53400/0 (socket says 172.21.15.38:53400) 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 -- 172.21.15.38:0/1528043247 learned_addr learned my addr 172.21.15.38:0/1528043247 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed133640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd712ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ec932640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd710280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 -- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 msgr2=0x55c4bd712ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd712ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 -- 172.21.15.38:0/1528043247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 msgr2=0x55c4bd710280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd710280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 -- 172.21.15.38:0/1528043247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4bd5fbc80 con 0x55c4bd713520 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ec932640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd710280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22ed934640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f22e001eb70 tx=0x7f22e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53698c071f603ed server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e0010dc0 con 0x55c4bd713520 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f22e0003970 con 0x55c4bd713520 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.748+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22e0010c10 con 0x55c4bd713520 2024-04-22T00:09:13.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22e0003b70 con 0x55c4bd713520 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 10 client.18000 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c4bd3823f0 con 0x55c4bd713520 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e0010c10 con 0x55c4bd713520 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f22e0004750 con 0x55c4bd713520 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 10 client.18000 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 10 client.18000 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 20 client.18000 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd725900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 4 client.18000 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4bd67ba70 con 0x55c4bd723510 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.749+0000 7f22ed133640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd725900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.750+0000 7f22d0ff9640 10 client.18000 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.750+0000 7f22ed133640 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd725900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c1c80f30e39d8cc7 server_cookie=a7b39f65647fbd0d in_seq=0 out_seq=0 2024-04-22T00:09:13.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.750+0000 7f22d0ff9640 1 -- 172.21.15.38:0/1528043247 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4bd67ba70 con 0x55c4bd723510 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.750+0000 7f22d0ff9640 10 client.18000 handle_command_reply: tid=0 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.750+0000 7f22f1f7c740 1 client.18000 shutdown 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 msgr2=0x55c4bd710890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f22e001eb70 tx=0x7f22e0009840 comp rx=0 tx=0).stop 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 shutdown_connections 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd725900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22d0ff9640 0 client.18000 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd713520 0x55c4bd710890 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd70a940 0x55c4bd710280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/1528043247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd712ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 >> 172.21.15.38:0/1528043247 conn(0x55c4bd704750 msgr2=0x55c4bd7064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 shutdown_connections 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 1 -- 172.21.15.38:0/1528043247 wait complete. 2024-04-22T00:09:13.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.751+0000 7f22f1f7c740 20 client.18000 trim_cache size 0 max 16384 2024-04-22T00:09:13.765 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 Processor -- start 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 -- start start 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a940 0x55c4bd7129b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd712ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81b0f70 con 0x55c4bd70a020 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81b2ab0 con 0x55c4bd713520 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81b1640 con 0x55c4bd70a940 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22ed133640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22ed133640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53416/0 (socket says 172.21.15.38:53416) 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.764+0000 7f22ed133640 1 -- 172.21.15.38:0/2527441883 learned_addr learned my addr 172.21.15.38:0/2527441883 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ec932640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a940 0x55c4bd7129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed934640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd712ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.766 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 -- 172.21.15.38:0/2527441883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a940 msgr2=0x55c4bd7129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a940 0x55c4bd7129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 -- 172.21.15.38:0/2527441883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 msgr2=0x55c4bd712ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd712ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 -- 172.21.15.38:0/2527441883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4bd44c7d0 con 0x55c4bd70a020 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed934640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd712ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22ed133640 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f22e409b2b0 tx=0x7f22e406e620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ee94b11b114706 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22d2ffd640 1 -- 172.21.15.38:0/2527441883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4020040 con 0x55c4bd70a020 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22d2ffd640 1 -- 172.21.15.38:0/2527441883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e409f960 con 0x55c4bd70a020 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.765+0000 7f22d2ffd640 1 -- 172.21.15.38:0/2527441883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40cca60 con 0x55c4bd70a020 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- 172.21.15.38:0/2527441883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 msgr2=0x55c4bd71f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f22e409b2b0 tx=0x7f22e406e620 comp rx=0 tx=0).stop 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- 172.21.15.38:0/2527441883 shutdown_connections 2024-04-22T00:09:13.767 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd712ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a940 0x55c4bd7129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/2527441883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a020 0x55c4bd71f150 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- 172.21.15.38:0/2527441883 >> 172.21.15.38:0/2527441883 conn(0x55c4bd704750 msgr2=0x55c4bd703850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- 172.21.15.38:0/2527441883 shutdown_connections 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- 172.21.15.38:0/2527441883 wait complete. 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- start start 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 Processor -- start 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd728b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd726430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81b23e0 con 0x55c4bd70a940 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81b1d10 con 0x55c4bd713520 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22e81aef70 con 0x55c4bd70a020 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22f1f7c740 10 client.0 _ll_register_callbacks cb 0x7f22e400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.766+0000 7f22ec932640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53430/0 (socket says 172.21.15.38:53430) 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 -- 172.21.15.38:0/143066450 learned_addr learned my addr 172.21.15.38:0/143066450 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ed934640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd726430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ed133640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd728b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 -- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 msgr2=0x55c4bd728b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd728b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 -- 172.21.15.38:0/143066450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 msgr2=0x55c4bd726430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd726430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 -- 172.21.15.38:0/143066450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4bd726a70 con 0x55c4bd70a940 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22ec932640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f22d80096c0 tx=0x7f22d802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dad5696a787135 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d802c6b0 con 0x55c4bd70a940 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f22d8005220 con 0x55c4bd70a940 2024-04-22T00:09:13.769 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22e409f960 con 0x55c4bd70a940 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22d8005420 con 0x55c4bd70a940 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22f1f7c740 10 client.18006 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.767+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c4bd789530 con 0x55c4bd70a940 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e409f960 con 0x55c4bd70a940 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f22d8003710 con 0x55c4bd70a940 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 10 client.18006 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 10 client.18006 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 20 client.18006 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd78bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 4 client.18006 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4bd675dc0 con 0x55c4bd723510 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.768+0000 7f22ed133640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd78bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22d0ff9640 10 client.18006 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22ed133640 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd78bd40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e91e10e7c57e7e3 server_cookie=197edf404aeae5f3 in_seq=0 out_seq=0 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22d0ff9640 1 -- 172.21.15.38:0/143066450 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4bd675dc0 con 0x55c4bd723510 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22d0ff9640 10 client.18006 handle_command_reply: tid=0 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 client.18006 shutdown 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 msgr2=0x55c4bd725ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f22d80096c0 tx=0x7f22d802ce30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 shutdown_connections 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55c4bd723510 0x55c4bd78bd40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4bd713520 0x55c4bd726430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c4bd70a940 0x55c4bd725ef0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22d0ff9640 0 client.18006 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 --2- 172.21.15.38:0/143066450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4bd70a020 0x55c4bd728b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 >> 172.21.15.38:0/143066450 conn(0x55c4bd704750 msgr2=0x55c4bd705a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 shutdown_connections 2024-04-22T00:09:13.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 1 -- 172.21.15.38:0/143066450 wait complete. 2024-04-22T00:09:13.772 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.769+0000 7f22f1f7c740 20 client.18006 trim_cache size 0 max 16384 2024-04-22T00:09:13.772 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 msgr2=0x7f22d40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 0x7f22d40a5240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22e0012cc0 tx=0x7f22e001b000 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 msgr2=0x7f22e819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d950 tx=0x7f22e400de10 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 shutdown_connections 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f22d40a2d80 0x7f22d40a5240 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22e810e150 0x7f22e819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22e806a6e0 0x7f22e819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 --2- 172.21.15.38:0/3442345587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22e8069dc0 0x7f22e819a210 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 >> 172.21.15.38:0/3442345587 conn(0x7f22e80776a0 msgr2=0x7f22e810f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 shutdown_connections 2024-04-22T00:09:13.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.770+0000 7f22ef397640 1 -- 172.21.15.38:0/3442345587 wait complete. 2024-04-22T00:09:13.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:09:13.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 -- 172.21.15.38:0/2177856448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0504100610 msgr2=0x7f0504100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 --2- 172.21.15.38:0/2177856448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0504100610 0x7f0504100a90 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009930 tx=0x7f04f402bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:13.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 -- 172.21.15.38:0/2177856448 shutdown_connections 2024-04-22T00:09:13.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 --2- 172.21.15.38:0/2177856448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 0x7f050410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 --2- 172.21.15.38:0/2177856448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0504100610 0x7f0504100a90 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 --2- 172.21.15.38:0/2177856448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05041066b0 0x7f0504106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 -- 172.21.15.38:0/2177856448 >> 172.21.15.38:0/2177856448 conn(0x7f0504077740 msgr2=0x7f0504077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 -- 172.21.15.38:0/2177856448 shutdown_connections 2024-04-22T00:09:13.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.885+0000 7f050b666640 1 -- 172.21.15.38:0/2177856448 wait complete. 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 Processor -- start 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 -- start start 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0504100610 0x7f0504197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 0x7f0504198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0504111b30 con 0x7f05041066b0 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f05041119b0 con 0x7f0504100fd0 2024-04-22T00:09:13.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f050b666640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0504111cb0 con 0x7f0504100610 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53456/0 (socket says 172.21.15.38:53456) 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 -- 172.21.15.38:0/3809447206 learned_addr learned my addr 172.21.15.38:0/3809447206 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509402640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0504100610 0x7f0504197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0508c01640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 0x7f0504198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 -- 172.21.15.38:0/3809447206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0504100610 msgr2=0x7f0504197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0504100610 0x7f0504197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 -- 172.21.15.38:0/3809447206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 msgr2=0x7f0504198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 0x7f0504198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 -- 172.21.15.38:0/3809447206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050419cfc0 con 0x7f05041066b0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.886+0000 7f0509c03640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00eaa0 tx=0x7f04fc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.887+0000 7f04fa7fc640 1 -- 172.21.15.38:0/3809447206 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00cdb0 con 0x7f05041066b0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.887+0000 7f04fa7fc640 1 -- 172.21.15.38:0/3809447206 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04fc005020 con 0x7f05041066b0 2024-04-22T00:09:13.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.887+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050419d2b0 con 0x7f05041066b0 2024-04-22T00:09:13.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.887+0000 7f04fa7fc640 1 -- 172.21.15.38:0/3809447206 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc0116b0 con 0x7f05041066b0 2024-04-22T00:09:13.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.887+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05041a4b40 con 0x7f05041066b0 2024-04-22T00:09:13.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.889+0000 7f04fa7fc640 1 -- 172.21.15.38:0/3809447206 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f04fc005560 con 0x7f05041066b0 2024-04-22T00:09:13.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.889+0000 7f050e249740 1 Processor -- start 2024-04-22T00:09:13.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.889+0000 7f050e249740 1 -- start start 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x55579574cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557956a8530 con 0x555795744020 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555795650830 con 0x55579574d520 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f050e249740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557956a44f0 con 0x555795744940 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f04fa7fc640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 0x7f04f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53464/0 (socket says 172.21.15.38:53464) 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 -- 172.21.15.38:0/3983599981 learned_addr learned my addr 172.21.15.38:0/3983599981 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0509c03640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 0x7f04f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0509402640 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x55579574cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f04fa7fc640 1 -- 172.21.15.38:0/3809447206 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f04fc0cfa40 con 0x7f05041066b0 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 -- 172.21.15.38:0/3983599981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 msgr2=0x55579574cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x55579574cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 -- 172.21.15.38:0/3983599981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 msgr2=0x55579574f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 -- 172.21.15.38:0/3983599981 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557956742c0 con 0x555795744020 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0508c01640 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009900 tx=0x7f04f4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fea3bd9f6573396 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.890+0000 7f0509c03640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 0x7f04f00a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f04fc098590 tx=0x7f04fc01eb40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f04ef7fe640 1 -- 172.21.15.38:0/3983599981 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4049070 con 0x555795744020 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f04ef7fe640 1 -- 172.21.15.38:0/3983599981 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f40046c0 con 0x555795744020 2024-04-22T00:09:13.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f04ef7fe640 1 -- 172.21.15.38:0/3983599981 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4005150 con 0x555795744020 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 -- 172.21.15.38:0/3983599981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 msgr2=0x555795744400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009900 tx=0x7f04f4005d50 comp rx=0 tx=0).stop 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 -- 172.21.15.38:0/3983599981 shutdown_connections 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x55579574cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 --2- 172.21.15.38:0/3983599981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x555795744400 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 -- 172.21.15.38:0/3983599981 >> 172.21.15.38:0/3983599981 conn(0x55579573e750 msgr2=0x55579573eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 -- 172.21.15.38:0/3983599981 shutdown_connections 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.891+0000 7f050e249740 1 -- 172.21.15.38:0/3983599981 wait complete. 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 -- start start 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 Processor -- start 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x55579574cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557957502b0 con 0x555795744940 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555795663f10 con 0x55579574d520 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555795750430 con 0x555795744020 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f050e249740 10 client.0 _ll_register_callbacks cb 0x5557953bbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f0509402640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f0509402640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53478/0 (socket says 172.21.15.38:53478) 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f0509402640 1 -- 172.21.15.38:0/1874360828 learned_addr learned my addr 172.21.15.38:0/1874360828 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f0508c01640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x55579574cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.892+0000 7f0508c01640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 -- 172.21.15.38:0/1874360828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 msgr2=0x55579574cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x55579574cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 -- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 msgr2=0x55579574a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 -- 172.21.15.38:0/1874360828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555795635c80 con 0x555795744940 2024-04-22T00:09:13.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0508c01640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f0509402640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f050000be60 tx=0x7f0500012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b746cd8f35aaa32b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500005220 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f05000053c0 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f40046c0 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0500005600 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f050e249740 10 client.18018 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5557953bc3f0 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f40046c0 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f050001c400 con 0x555795744940 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f050e249740 10 client.18018 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.893+0000 7f050e249740 10 client.18018 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f050e249740 20 client.18018 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55579575d6a0 0x55579575fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f050e249740 4 client.18018 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557956b5a70 con 0x55579575d6a0 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f0508c01640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55579575d6a0 0x55579575fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f04ecff9640 10 client.18018 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.894+0000 7f0508c01640 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55579575d6a0 0x55579575fae0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e9fa087a94b7762 server_cookie=e1de0b37e9f05e5c in_seq=0 out_seq=0 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f04ecff9640 1 -- 172.21.15.38:0/1874360828 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557956b5a70 con 0x55579575d6a0 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f04ecff9640 10 client.18018 handle_command_reply: tid=0 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 client.18018 shutdown 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 msgr2=0x55579574a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f050000be60 tx=0x7f0500012820 comp rx=0 tx=0).stop 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 shutdown_connections 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579574a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55579575d6a0 0x55579575fae0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x55579574a280 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 --2- 172.21.15.38:0/1874360828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x55579574cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 >> 172.21.15.38:0/1874360828 conn(0x55579573e750 msgr2=0x5557957404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f04ecff9640 0 client.18018 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 shutdown_connections 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 1 -- 172.21.15.38:0/1874360828 wait complete. 2024-04-22T00:09:13.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.895+0000 7f050e249740 20 client.18018 trim_cache size 0 max 16384 2024-04-22T00:09:13.910 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 Processor -- start 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 -- start start 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x5557957493e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579573d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05041aeb20 con 0x555795744020 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f05041b0660 con 0x55579574d520 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f050e249740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05041af1f0 con 0x555795744940 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f0508c01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f0508c01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53484/0 (socket says 172.21.15.38:53484) 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.909+0000 7f0508c01640 1 -- 172.21.15.38:0/2537491609 learned_addr learned my addr 172.21.15.38:0/2537491609 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0509402640 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x5557957493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 -- 172.21.15.38:0/2537491609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 msgr2=0x5557957493e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x5557957493e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 -- 172.21.15.38:0/2537491609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 msgr2=0x55579573d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579573d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 -- 172.21.15.38:0/2537491609 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555795744770 con 0x555795744020 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f0508c01640 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005ab0 tx=0x7f04f402c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edfc779cfc5755d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f04eeffd640 1 -- 172.21.15.38:0/2537491609 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4049040 con 0x555795744020 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f04eeffd640 1 -- 172.21.15.38:0/2537491609 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f4004330 con 0x555795744020 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.910+0000 7f04eeffd640 1 -- 172.21.15.38:0/2537491609 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4005240 con 0x555795744020 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- 172.21.15.38:0/2537491609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 msgr2=0x5557957591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005ab0 tx=0x7f04f402c6b0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- 172.21.15.38:0/2537491609 shutdown_connections 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x55579573d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744940 0x5557957493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- 172.21.15.38:0/2537491609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744020 0x5557957591a0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- 172.21.15.38:0/2537491609 >> 172.21.15.38:0/2537491609 conn(0x55579573e750 msgr2=0x55579573cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- 172.21.15.38:0/2537491609 shutdown_connections 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- 172.21.15.38:0/2537491609 wait complete. 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- start start 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 Processor -- start 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x5557957558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x5557957511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05041aff90 con 0x555795744940 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f05041af8c0 con 0x55579574d520 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05041acb20 con 0x555795744020 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f050e249740 10 client.0 _ll_register_callbacks cb 0x7f050000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.911+0000 7f0509402640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53486/0 (socket says 172.21.15.38:53486) 2024-04-22T00:09:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 -- 172.21.15.38:0/969714384 learned_addr learned my addr 172.21.15.38:0/969714384 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0508c01640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x5557957558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0508c01640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x5557957511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 -- 172.21.15.38:0/969714384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 msgr2=0x5557957558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x5557957558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 -- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 msgr2=0x5557957511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x5557957511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 -- 172.21.15.38:0/969714384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557954867d0 con 0x555795744940 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0508c01640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x5557957511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f0509402640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f050000c010 tx=0x7f0500012e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cecf5869e4f474 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f04ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050002f070 con 0x555795744940 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f05000107d0 con 0x555795744940 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f4004330 con 0x555795744940 2024-04-22T00:09:13.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f050003b050 con 0x555795744940 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f050e249740 10 client.18024 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.912+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5557957c3900 con 0x555795744940 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f4004330 con 0x555795744940 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0500034c50 con 0x555795744940 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 10 client.18024 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 10 client.18024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 20 client.18024 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5557957c4050 0x5557957c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 4 client.18024 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5557956afdc0 con 0x5557957c4050 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f0508c01640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5557957c4050 0x5557957c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f04ecff9640 10 client.18024 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.913+0000 7f0508c01640 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5557957c4050 0x5557957c6470 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ee535dc18ef35c server_cookie=380a2797e532f6b8 in_seq=0 out_seq=0 2024-04-22T00:09:13.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f04ecff9640 1 -- 172.21.15.38:0/969714384 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5557956afdc0 con 0x5557957c4050 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f04ecff9640 10 client.18024 handle_command_reply: tid=0 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 client.18024 shutdown 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 msgr2=0x555795750c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f050000c010 tx=0x7f0500012e60 comp rx=0 tx=0).stop 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 shutdown_connections 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55579574d520 0x5557957511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5557957c4050 0x5557957c6470 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555795744940 0x555795750c70 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 --2- 172.21.15.38:0/969714384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555795744020 0x5557957558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 >> 172.21.15.38:0/969714384 conn(0x55579573e750 msgr2=0x55579573d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f04ecff9640 0 client.18024 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 shutdown_connections 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 1 -- 172.21.15.38:0/969714384 wait complete. 2024-04-22T00:09:13.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.914+0000 7f050e249740 20 client.18024 trim_cache size 0 max 16384 2024-04-22T00:09:13.917 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 msgr2=0x7f04f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 0x7f04f00a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f04fc098590 tx=0x7f04fc01eb40 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 msgr2=0x7f050419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00eaa0 tx=0x7f04fc00ef60 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 shutdown_connections 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04f00a2d00 0x7f04f00a51c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05041066b0 0x7f050419c890 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0504100fd0 0x7f0504198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 --2- 172.21.15.38:0/3809447206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0504100610 0x7f0504197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 >> 172.21.15.38:0/3809447206 conn(0x7f0504077740 msgr2=0x7f05040ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 shutdown_connections 2024-04-22T00:09:13.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:13.915+0000 7f050b666640 1 -- 172.21.15.38:0/3809447206 wait complete. 2024-04-22T00:09:14.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:14 smithi038 ceph-mon[27972]: pgmap v218: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 2.06k op/s 2024-04-22T00:09:14.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3084554265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:14.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:14 smithi203 ceph-mon[31431]: pgmap v218: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 2.06k op/s 2024-04-22T00:09:14.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3084554265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:14.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:14 smithi111 ceph-mon[31543]: pgmap v218: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.4 MiB/s wr, 2.06k op/s 2024-04-22T00:09:14.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3084554265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:14.923 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:09:14.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:09:14.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- 172.21.15.38:0/2531559478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c10e0f0 msgr2=0x7f2e6c1104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 --2- 172.21.15.38:0/2531559478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c1104e0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009900 tx=0x7f2e6402bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- 172.21.15.38:0/2531559478 shutdown_connections 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 --2- 172.21.15.38:0/2531559478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c1104e0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 --2- 172.21.15.38:0/2531559478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c100e60 0x7f2e6c10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 --2- 172.21.15.38:0/2531559478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 0x7f2e6c100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- 172.21.15.38:0/2531559478 >> 172.21.15.38:0/2531559478 conn(0x7f2e6c077740 msgr2=0x7f2e6c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- 172.21.15.38:0/2531559478 shutdown_connections 2024-04-22T00:09:15.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- 172.21.15.38:0/2531559478 wait complete. 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 Processor -- start 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.048+0000 7f2e744f4640 1 -- start start 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 0x7f2e6c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c113d20 con 0x7f2e6c100e60 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c113ba0 con 0x7f2e6c10e0f0 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e744f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c113ea0 con 0x7f2e6c100540 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53502/0 (socket says 172.21.15.38:53502) 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3427832028 learned_addr learned my addr 172.21.15.38:0/3427832028 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e72290640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 0x7f2e6c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3427832028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 msgr2=0x7f2e6c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 0x7f2e6c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3427832028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c10e0f0 msgr2=0x7f2e6c19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3427832028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c19f1e0 con 0x7f2e6c100e60 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.049+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000d950 tx=0x7f2e6000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.050+0000 7f2e6b7fe640 1 -- 172.21.15.38:0/3427832028 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60017070 con 0x7f2e6c100e60 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.050+0000 7f2e6b7fe640 1 -- 172.21.15.38:0/3427832028 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e60009c30 con 0x7f2e6c100e60 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.050+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c19f4d0 con 0x7f2e6c100e60 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.050+0000 7f2e6b7fe640 1 -- 172.21.15.38:0/3427832028 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000f6b0 con 0x7f2e6c100e60 2024-04-22T00:09:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.050+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c1a6d60 con 0x7f2e6c100e60 2024-04-22T00:09:15.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.052+0000 7f2e6b7fe640 1 -- 172.21.15.38:0/3427832028 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2e600109d0 con 0x7f2e6c100e60 2024-04-22T00:09:15.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.052+0000 7f2e771ba740 1 Processor -- start 2024-04-22T00:09:15.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 -- start start 2024-04-22T00:09:15.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce411ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce41d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5572ce375170 con 0x5572ce412720 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5572ce31d7c0 con 0x5572ce41adb0 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e771ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5572ce371130 con 0x5572ce411a40 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e72a91640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e6b7fe640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 0x7f2e540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e72a91640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53514/0 (socket says 172.21.15.38:53514) 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e72a91640 1 -- 172.21.15.38:0/249766406 learned_addr learned my addr 172.21.15.38:0/249766406 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce41d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e72290640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce411ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.053+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 0x7f2e540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 -- 172.21.15.38:0/249766406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 msgr2=0x5572ce411ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce411ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e6b7fe640 1 -- 172.21.15.38:0/3427832028 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f2e600cf360 con 0x7f2e6c100e60 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 -- 172.21.15.38:0/249766406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 msgr2=0x5572ce41d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce41d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 -- 172.21.15.38:0/249766406 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572ce32ee60 con 0x5572ce412720 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce41d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2e64005ab0 tx=0x7f2e6402cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b4b99bf939c517 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e68bfe640 1 -- 172.21.15.38:0/249766406 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64050070 con 0x5572ce412720 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e68bfe640 1 -- 172.21.15.38:0/249766406 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e64004ef0 con 0x5572ce412720 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e68bfe640 1 -- 172.21.15.38:0/249766406 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64003850 con 0x5572ce412720 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.054+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 0x7f2e540a5240 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e64004000 tx=0x7f2e64049000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- 172.21.15.38:0/249766406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 msgr2=0x5572ce411470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2e64005ab0 tx=0x7f2e6402cb00 comp rx=0 tx=0).stop 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- 172.21.15.38:0/249766406 shutdown_connections 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce41d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce411ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 --2- 172.21.15.38:0/249766406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce411470 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- 172.21.15.38:0/249766406 >> 172.21.15.38:0/249766406 conn(0x5572ce3d84f0 msgr2=0x5572ce40c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- 172.21.15.38:0/249766406 shutdown_connections 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- 172.21.15.38:0/249766406 wait complete. 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 -- start start 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.055+0000 7f2e771ba740 1 Processor -- start 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce411a40 0x5572ce417d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5572ce41dc20 con 0x5572ce412720 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5572ce330d40 con 0x5572ce411a40 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5572ce41dda0 con 0x5572ce41adb0 2024-04-22T00:09:15.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 10 client.0 _ll_register_callbacks cb 0x5572ce36c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53528/0 (socket says 172.21.15.38:53528) 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 -- 172.21.15.38:0/2459928663 learned_addr learned my addr 172.21.15.38:0/2459928663 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e72290640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce411a40 0x5572ce417d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e72a91640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 -- 172.21.15.38:0/2459928663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 msgr2=0x5572ce4187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 -- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce411a40 msgr2=0x5572ce417d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce411a40 0x5572ce417d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 -- 172.21.15.38:0/2459928663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572ce1517e0 con 0x5572ce412720 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009610 tx=0x7f2e6009b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53627067f9fb1231 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60020030 con 0x5572ce412720 2024-04-22T00:09:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f2e60099790 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.056+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e64004ef0 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2e6009b490 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 10 client.18036 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572ce1b1f70 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e64004ef0 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2e6009a8e0 con 0x5572ce412720 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 10 client.18036 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 10 client.18036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 20 client.18036 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce42ae80 0x5572ce42d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 4 client.18036 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572ce349450 con 0x5572ce42ae80 2024-04-22T00:09:15.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.057+0000 7f2e72290640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce42ae80 0x5572ce42d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.058+0000 7f2e72290640 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce42ae80 0x5572ce42d290 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67b984096a32b9e3 server_cookie=cd9aa3ef909a551e in_seq=0 out_seq=0 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.058+0000 7f2e51ffb640 10 client.18036 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.058+0000 7f2e51ffb640 1 -- 172.21.15.38:0/2459928663 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572ce349450 con 0x5572ce42ae80 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.058+0000 7f2e51ffb640 10 client.18036 handle_command_reply: tid=0 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.058+0000 7f2e771ba740 1 client.18036 shutdown 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 msgr2=0x5572ce418280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009610 tx=0x7f2e6009b390 comp rx=0 tx=0).stop 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 shutdown_connections 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce42ae80 0x5572ce42d290 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce418280 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e51ffb640 0 client.18036 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 --2- 172.21.15.38:0/2459928663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce411a40 0x5572ce417d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 >> 172.21.15.38:0/2459928663 conn(0x5572ce3d84f0 msgr2=0x5572ce40e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 shutdown_connections 2024-04-22T00:09:15.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 1 -- 172.21.15.38:0/2459928663 wait complete. 2024-04-22T00:09:15.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.059+0000 7f2e771ba740 20 client.18036 trim_cache size 0 max 16384 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 Processor -- start 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 -- start start 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce41a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce40b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1b0e70 con 0x5572ce412720 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1b29b0 con 0x5572ce41adb0 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e771ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1b1540 con 0x5572ce411a40 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53540/0 (socket says 172.21.15.38:53540) 2024-04-22T00:09:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3685288845 learned_addr learned my addr 172.21.15.38:0/3685288845 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e72290640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce41a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce40b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3685288845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 msgr2=0x5572ce41a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce41a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3685288845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 msgr2=0x5572ce40b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce40b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e71a8f640 1 -- 172.21.15.38:0/3685288845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572ce049170 con 0x5572ce412720 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.077+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce40b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f2e60010fd0 tx=0x7f2e6000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45601688394f6912 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e53fff640 1 -- 172.21.15.38:0/3685288845 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60014020 con 0x5572ce412720 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e53fff640 1 -- 172.21.15.38:0/3685288845 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e600a04d0 con 0x5572ce412720 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e53fff640 1 -- 172.21.15.38:0/3685288845 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60020070 con 0x5572ce412720 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 -- 172.21.15.38:0/3685288845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 msgr2=0x5572ce426b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f2e60010fd0 tx=0x7f2e6000bf60 comp rx=0 tx=0).stop 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 -- 172.21.15.38:0/3685288845 shutdown_connections 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce41adb0 0x5572ce40b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce412720 0x5572ce426b50 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3685288845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce411a40 0x5572ce41a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 -- 172.21.15.38:0/3685288845 >> 172.21.15.38:0/3685288845 conn(0x5572ce3d84f0 msgr2=0x5572ce40ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 -- 172.21.15.38:0/3685288845 shutdown_connections 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.078+0000 7f2e771ba740 1 -- 172.21.15.38:0/3685288845 wait complete. 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 -- start start 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 Processor -- start 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce412720 0x5572ce415660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1b22e0 con 0x5572ce411a40 2024-04-22T00:09:15.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1b1c10 con 0x5572ce412720 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c1aee70 con 0x5572ce41adb0 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e771ba740 10 client.0 _ll_register_callbacks cb 0x7f2e6000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e72290640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e72290640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53556/0 (socket says 172.21.15.38:53556) 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e72290640 1 -- 172.21.15.38:0/3905735052 learned_addr learned my addr 172.21.15.38:0/3905735052 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e72a91640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.079+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce412720 0x5572ce415660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 -- 172.21.15.38:0/3905735052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 msgr2=0x5572ce4903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 -- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce412720 msgr2=0x5572ce415660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce412720 0x5572ce415660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 -- 172.21.15.38:0/3905735052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572ce490910 con 0x5572ce411a40 2024-04-22T00:09:15.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e72290640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x5572ce42dad0 tx=0x7f2e5c012b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895cdb32b9048c8a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c004d20 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f2e5c010900 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e600a04d0 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2e5c030050 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 10 client.18042 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572ce491bd0 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e600a04d0 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2e5c00f040 con 0x5572ce411a40 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 10 client.18042 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 10 client.18042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.080+0000 7f2e771ba740 20 client.18042 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce492300 0x5572ce494720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e771ba740 4 client.18042 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5572ce3826b0 con 0x5572ce492300 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce492300 0x5572ce494720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e71a8f640 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce492300 0x5572ce494720 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34afdbd1b852ba0c server_cookie=2d5fc3f29ef46d70 in_seq=0 out_seq=0 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.081+0000 7f2e51ffb640 10 client.18042 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.082+0000 7f2e51ffb640 1 -- 172.21.15.38:0/3905735052 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5572ce3826b0 con 0x5572ce492300 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.082+0000 7f2e51ffb640 10 client.18042 handle_command_reply: tid=0 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.082+0000 7f2e771ba740 1 client.18042 shutdown 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.082+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 msgr2=0x5572ce415120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.082+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x5572ce42dad0 tx=0x7f2e5c012b10 comp rx=0 tx=0).stop 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 shutdown_connections 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5572ce41adb0 0x5572ce4903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5572ce412720 0x5572ce415660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5572ce492300 0x5572ce494720 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 --2- 172.21.15.38:0/3905735052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572ce411a40 0x5572ce415120 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e51ffb640 0 client.18042 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 >> 172.21.15.38:0/3905735052 conn(0x5572ce3d84f0 msgr2=0x5572ce40b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 shutdown_connections 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 1 -- 172.21.15.38:0/3905735052 wait complete. 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e771ba740 20 client.18042 trim_cache size 0 max 16384 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "8f548f89-1a3d-4f45-bd9f-13615382e754", 2024-04-22T00:09:15.085 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:09:15.086 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:15.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 msgr2=0x7f2e540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 0x7f2e540a5240 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e64004000 tx=0x7f2e64049000 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 msgr2=0x7f2e6c19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000d950 tx=0x7f2e6000de10 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 shutdown_connections 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e540a2d80 0x7f2e540a5240 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e6c10e0f0 0x7f2e6c19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e6c100e60 0x7f2e6c19a780 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 --2- 172.21.15.38:0/3427832028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e6c100540 0x7f2e6c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.083+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 >> 172.21.15.38:0/3427832028 conn(0x7f2e6c077740 msgr2=0x7f2e6c10f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.084+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 shutdown_connections 2024-04-22T00:09:15.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.084+0000 7f2e744f4640 1 -- 172.21.15.38:0/3427832028 wait complete. 2024-04-22T00:09:15.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:09:15.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:09:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 -- 172.21.15.38:0/667044838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb406a780 msgr2=0x7efeb410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 --2- 172.21.15.38:0/667044838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb406a780 0x7efeb410e5f0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7efea4009930 tx=0x7efea402bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:15.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 -- 172.21.15.38:0/667044838 shutdown_connections 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 --2- 172.21.15.38:0/667044838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb406a780 0x7efeb410e5f0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 --2- 172.21.15.38:0/667044838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4069dc0 0x7efeb406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 --2- 172.21.15.38:0/667044838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb4107730 0x7efeb4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 -- 172.21.15.38:0/667044838 >> 172.21.15.38:0/667044838 conn(0x7efeb40776a0 msgr2=0x7efeb4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 -- 172.21.15.38:0/667044838 shutdown_connections 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.206+0000 7efebb73e640 1 -- 172.21.15.38:0/667044838 wait complete. 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 Processor -- start 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 -- start start 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb406a780 0x7efeb41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4107730 0x7efeb419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4111b40 con 0x7efeb4069dc0 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41119c0 con 0x7efeb406a780 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efebb73e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4111cc0 con 0x7efeb4107730 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53570/0 (socket says 172.21.15.38:53570) 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 -- 172.21.15.38:0/481708721 learned_addr learned my addr 172.21.15.38:0/481708721 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4107730 0x7efeb419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb406a780 0x7efeb41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 -- 172.21.15.38:0/481708721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4107730 msgr2=0x7efeb419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4107730 0x7efeb419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 -- 172.21.15.38:0/481708721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb406a780 msgr2=0x7efeb41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb406a780 0x7efeb41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.207+0000 7efeb94da640 1 -- 172.21.15.38:0/481708721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb419cf90 con 0x7efeb4069dc0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efeb94da640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7efeac00ebc0 tx=0x7efeac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efeb27fc640 1 -- 172.21.15.38:0/481708721 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac010630 con 0x7efeb4069dc0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efeb27fc640 1 -- 172.21.15.38:0/481708721 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeac010cb0 con 0x7efeb4069dc0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efeb27fc640 1 -- 172.21.15.38:0/481708721 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac018bd0 con 0x7efeb4069dc0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb419d280 con 0x7efeb4069dc0 2024-04-22T00:09:15.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.208+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb41a4b10 con 0x7efeb4069dc0 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.210+0000 7efeb27fc640 1 -- 172.21.15.38:0/481708721 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7efeac0107d0 con 0x7efeb4069dc0 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.210+0000 7efebe406740 1 Processor -- start 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.210+0000 7efebe406740 1 -- start start 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99deba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b99de1e560 con 0x55b99deb9f30 2024-04-22T00:09:15.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b99ddc6730 con 0x55b99dec3430 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efebe406740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b99de1a520 con 0x55b99deba850 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb9cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99deba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb9cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99deba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53580/0 (socket says 172.21.15.38:53580) 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb9cdb640 1 -- 172.21.15.38:0/931367855 learned_addr learned my addr 172.21.15.38:0/931367855 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb94da640 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb27fc640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 0x7efe9c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 -- 172.21.15.38:0/931367855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 msgr2=0x55b99dec2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 -- 172.21.15.38:0/931367855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 msgr2=0x55b99deba310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99deba310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb8cd9640 1 -- 172.21.15.38:0/931367855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99dbfc7d0 con 0x55b99dec3430 2024-04-22T00:09:15.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 0x7efe9c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.211+0000 7efeb27fc640 1 -- 172.21.15.38:0/481708721 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7efeac0cfda0 con 0x7efeb4069dc0 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec5820 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efea8005fd0 tx=0x7efea800bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a503a4a91052ca4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 0x7efe9c0a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efea4006010 tx=0x7efea402e040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efe9b7fe640 1 -- 172.21.15.38:0/931367855 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8003bd0 con 0x55b99dec3430 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efe9b7fe640 1 -- 172.21.15.38:0/931367855 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea8015070 con 0x55b99dec3430 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efe9b7fe640 1 -- 172.21.15.38:0/931367855 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8004d20 con 0x55b99dec3430 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efebe406740 1 -- 172.21.15.38:0/931367855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 msgr2=0x55b99dec5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.212+0000 7efebe406740 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec5820 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efea8005fd0 tx=0x7efea800bb20 comp rx=0 tx=0).stop 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- 172.21.15.38:0/931367855 shutdown_connections 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec5820 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- 172.21.15.38:0/931367855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99deba310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- 172.21.15.38:0/931367855 >> 172.21.15.38:0/931367855 conn(0x55b99deb4660 msgr2=0x55b99deb4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- 172.21.15.38:0/931367855 shutdown_connections 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- 172.21.15.38:0/931367855 wait complete. 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- start start 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 Processor -- start 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b99dec6220 con 0x55b99deb9f30 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b99ddd9e00 con 0x55b99dec3430 2024-04-22T00:09:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b99dec63a0 con 0x55b99deba850 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.213+0000 7efebe406740 10 client.0 _ll_register_callbacks cb 0x55b99de14eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53588/0 (socket says 172.21.15.38:53588) 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb9cdb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 -- 172.21.15.38:0/3919785071 learned_addr learned my addr 172.21.15.38:0/3919785071 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 -- 172.21.15.38:0/3919785071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 msgr2=0x55b99dec09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 -- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 msgr2=0x55b99dec0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 -- 172.21.15.38:0/3919785071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b99db323f0 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efeb94da640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efeac09bc80 tx=0x7efeac009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a37485ede08c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efe98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac010e60 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7efeac09b3d0 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea8015070 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efeac09bac0 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efebe406740 10 client.18054 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b99daf20f0 con 0x55b99deb9f30 2024-04-22T00:09:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.214+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea8015070 con 0x55b99deb9f30 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efeac096af0 con 0x55b99deb9f30 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 10 client.18054 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 10 client.18054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 20 client.18054 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99ded3830 0x55b99ded5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 4 client.18054 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b99de2baa0 con 0x55b99ded3830 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.215+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99ded3830 0x55b99ded5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.216+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99ded3830 0x55b99ded5c20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=772e30ae1644deca server_cookie=32625c0265849734 in_seq=0 out_seq=0 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.216+0000 7efe98ff9640 10 client.18054 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.216+0000 7efe98ff9640 1 -- 172.21.15.38:0/3919785071 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b99de2baa0 con 0x55b99ded3830 2024-04-22T00:09:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.216+0000 7efe98ff9640 10 client.18054 handle_command_reply: tid=0 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.216+0000 7efebe406740 1 client.18054 shutdown 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 msgr2=0x55b99dec0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7efeac09bc80 tx=0x7efeac009200 comp rx=0 tx=0).stop 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 shutdown_connections 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99dec3430 0x55b99dec0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deba850 0x55b99dec09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99ded3830 0x55b99ded5c20 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 --2- 172.21.15.38:0/3919785071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec0480 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efe98ff9640 0 client.18054 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 >> 172.21.15.38:0/3919785071 conn(0x55b99deb4660 msgr2=0x55b99deb63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 shutdown_connections 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 1 -- 172.21.15.38:0/3919785071 wait complete. 2024-04-22T00:09:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.217+0000 7efebe406740 20 client.18054 trim_cache size 0 max 16384 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.234+0000 7efebe406740 1 Processor -- start 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.234+0000 7efebe406740 1 -- start start 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 0x55b99decf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41aecb0 con 0x55b99deb9f30 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41b07f0 con 0x55b99deba850 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efebe406740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41af380 con 0x55b99dec3430 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53590/0 (socket says 172.21.15.38:53590) 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 -- 172.21.15.38:0/898831513 learned_addr learned my addr 172.21.15.38:0/898831513 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 0x55b99decf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 -- 172.21.15.38:0/898831513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 msgr2=0x55b99decf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 0x55b99decf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 -- 172.21.15.38:0/898831513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 msgr2=0x55b99debb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 -- 172.21.15.38:0/898831513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99debf430 con 0x55b99deb9f30 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 0x55b99decf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efeb94da640 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efeac0983e0 tx=0x7efeac09a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c30d7b3447fa35e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efe9affd640 1 -- 172.21.15.38:0/898831513 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac010e60 con 0x55b99deb9f30 2024-04-22T00:09:15.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efe9affd640 1 -- 172.21.15.38:0/898831513 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeac0d35d0 con 0x55b99deb9f30 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.235+0000 7efe9affd640 1 -- 172.21.15.38:0/898831513 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac09b8e0 con 0x55b99deb9f30 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- 172.21.15.38:0/898831513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 msgr2=0x55b99dec2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efeac0983e0 tx=0x7efeac09a490 comp rx=0 tx=0).stop 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- 172.21.15.38:0/898831513 shutdown_connections 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99dec3430 0x55b99decf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 --2- 172.21.15.38:0/898831513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99deb9f30 0x55b99dec2b40 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- 172.21.15.38:0/898831513 >> 172.21.15.38:0/898831513 conn(0x55b99deb4660 msgr2=0x55b99deb2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- 172.21.15.38:0/898831513 shutdown_connections 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- 172.21.15.38:0/898831513 wait complete. 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 -- start start 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.236+0000 7efebe406740 1 Processor -- start 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deb9f30 0x55b99decb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41b0120 con 0x55b99dec3430 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41afa50 con 0x55b99deba850 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41accb0 con 0x55b99deb9f30 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efebe406740 10 client.0 _ll_register_callbacks cb 0x7efeac00eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53592/0 (socket says 172.21.15.38:53592) 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 -- 172.21.15.38:0/2628109552 learned_addr learned my addr 172.21.15.38:0/2628109552 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb94da640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deb9f30 0x55b99decb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 -- 172.21.15.38:0/2628109552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deb9f30 msgr2=0x55b99decb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deb9f30 0x55b99decb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 -- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 msgr2=0x55b99debcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 -- 172.21.15.38:0/2628109552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b99debd790 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb8cd9640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efeb9cdb640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efea402e010 tx=0x7efea40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6561496616afd82 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efe98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.237+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4005110 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeac0d35d0 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7efea40052b0 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efea40054b0 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 10 client.18060 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b99df39a20 con 0x55b99dec3430 2024-04-22T00:09:15.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeac0d35d0 con 0x55b99dec3430 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efea4056630 con 0x55b99dec3430 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 10 client.18060 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 10 client.18060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 20 client.18060 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99df3a170 0x55b99df3c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 4 client.18060 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b99de25df0 con 0x55b99df3a170 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.238+0000 7efeb94da640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99df3a170 0x55b99df3c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efeb94da640 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99df3a170 0x55b99df3c590 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bcd9a08bf4b3e7 server_cookie=87c4ea9f02fd85c5 in_seq=0 out_seq=0 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efe98ff9640 10 client.18060 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efe98ff9640 1 -- 172.21.15.38:0/2628109552 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b99de25df0 con 0x55b99df3a170 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efe98ff9640 10 client.18060 handle_command_reply: tid=0 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 client.18060 shutdown 2024-04-22T00:09:15.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 msgr2=0x55b99debd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7efea402e010 tx=0x7efea40058d0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 shutdown_connections 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55b99df3a170 0x55b99df3c590 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b99dec3430 0x55b99debd250 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b99deba850 0x55b99debcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efe98ff9640 0 client.18060 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 --2- 172.21.15.38:0/2628109552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b99deb9f30 0x55b99decb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 >> 172.21.15.38:0/2628109552 conn(0x55b99deb4660 msgr2=0x55b99deb36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 shutdown_connections 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 1 -- 172.21.15.38:0/2628109552 wait complete. 2024-04-22T00:09:15.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.239+0000 7efebe406740 20 client.18060 trim_cache size 0 max 16384 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "8f548f89-1a3d-4f45-bd9f-13615382e754": { 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "8f548f89-1a3d-4f45-bd9f-13615382e754", 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:09:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:09:15.245 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:15.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 msgr2=0x7efe9c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 0x7efe9c0a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efea4006010 tx=0x7efea402e040 comp rx=0 tx=0).stop 2024-04-22T00:09:15.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 msgr2=0x7efeb4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:15.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7efeac00ebc0 tx=0x7efeac00c6a0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 shutdown_connections 2024-04-22T00:09:15.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7efe9c0a2d00 0x7efe9c0a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb4107730 0x7efeb419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb406a780 0x7efeb41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 --2- 172.21.15.38:0/481708721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efeb4069dc0 0x7efeb4197fa0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 >> 172.21.15.38:0/481708721 conn(0x7efeb40776a0 msgr2=0x7efeb40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 shutdown_connections 2024-04-22T00:09:15.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:15.240+0000 7efebb73e640 1 -- 172.21.15.38:0/481708721 wait complete. 2024-04-22T00:09:15.248 INFO:tasks.cephfs.filesystem:scrub status for tag:8f548f89-1a3d-4f45-bd9f-13615382e754 - {'path': '/', 'tag': '8f548f89-1a3d-4f45-bd9f-13615382e754', 'options': 'recursive,force'} 2024-04-22T00:09:15.580 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:09:15 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:09:15.145+0000 7f8b50b1c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs/ auth{1=1,2=1,3=1,4=1} v375 ap=2 RANDEPHEMERALPIN f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3) n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5) old_inodes=1 (inest sync->mix g=1,2,3,4) caps={25090=pAsLsXsFs/pAsLsXsFsxcral@14} mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 scrubqueue=0 randepin 0x5592113b6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000008/refs v263>,<0x10000000007/.git v266>,<0x10000000006/ffsb v410>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v38>,<0x1/volumes v36>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000008/refs v375>,<0x10000000007/.git v3098>,<0x10000000006/ffsb v2679>,<0x10000000005/tmp v697>,<0x100000001fb/client.0 v676>,<0x100000001fa/ff2f9885-24f2-4e1b-ac4c-2ef80769331d v672>,<0x10000000001/sv_1 v684>,<0x10000000000/qa v661>,<0x1/volumes v643>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-04-22T00:03:30.735647+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","memory_value.dirstat":"f(v1 m2024-04-22T00:03:31.416636+0000 3=0+3)","memory_value.rstat":"n(v3 rc2024-04-22T00:03:31.561633+0000 b73 7=2+5)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-04-22T00:09:16.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: pgmap v219: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.6 MiB/s wr, 1.88k op/s 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:16 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: pgmap v219: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.6 MiB/s wr, 1.88k op/s 2024-04-22T00:09:16.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:16 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: pgmap v219: 129 pgs: 6 active+clean+snaptrim, 108 active+clean, 15 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.6 MiB/s wr, 1.88k op/s 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/ff2f9885-24f2-4e1b-ac4c-2ef80769331d/client.0/tmp/ffsb/.git/refs) see mds.b log and `damage ls` output for details 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:16 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:18.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:18 smithi038 ceph-mon[27972]: pgmap v220: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.5 MiB/s wr, 1.62k op/s 2024-04-22T00:09:18.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:18 smithi203 ceph-mon[31431]: pgmap v220: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.5 MiB/s wr, 1.62k op/s 2024-04-22T00:09:18.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:18 smithi111 ceph-mon[31543]: pgmap v220: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 6.5 MiB/s wr, 1.62k op/s 2024-04-22T00:09:20.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:20 smithi038 ceph-mon[27972]: pgmap v221: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.6 MiB/s wr, 1.16k op/s 2024-04-22T00:09:20.532 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:20 smithi203 ceph-mon[31431]: pgmap v221: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.6 MiB/s wr, 1.16k op/s 2024-04-22T00:09:20.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:20 smithi111 ceph-mon[31543]: pgmap v221: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.6 MiB/s wr, 1.16k op/s 2024-04-22T00:09:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 -- 172.21.15.38:0/3792467256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f873406a930 msgr2=0x7f873410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 --2- 172.21.15.38:0/3792467256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f873406a930 0x7f873410e5e0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f872c009ca0 tx=0x7f872c02c180 comp rx=0 tx=0).stop 2024-04-22T00:09:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 -- 172.21.15.38:0/3792467256 shutdown_connections 2024-04-22T00:09:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 --2- 172.21.15.38:0/3792467256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f873406a930 0x7f873410e5e0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 --2- 172.21.15.38:0/3792467256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8734069f70 0x7f873406a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 --2- 172.21.15.38:0/3792467256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734107940 0x7f8734107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 -- 172.21.15.38:0/3792467256 >> 172.21.15.38:0/3792467256 conn(0x7f8734076440 msgr2=0x7f8734076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 -- 172.21.15.38:0/3792467256 shutdown_connections 2024-04-22T00:09:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.593+0000 7f87398f4640 1 -- 172.21.15.38:0/3792467256 wait complete. 2024-04-22T00:09:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 Processor -- start 2024-04-22T00:09:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 -- start start 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 0x7f8734100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f873406a930 0x7f8734100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8734111b60 con 0x7f8734107940 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f87341119e0 con 0x7f8734069f70 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87398f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8734111ce0 con 0x7f873406a930 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87337fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53618/0 (socket says 172.21.15.38:53618) 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87337fe640 1 -- 172.21.15.38:0/3472862439 learned_addr learned my addr 172.21.15.38:0/3472862439 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.594+0000 7f87327fc640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f873406a930 0x7f8734100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 0x7f8734100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 -- 172.21.15.38:0/3472862439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f873406a930 msgr2=0x7f8734100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f873406a930 0x7f8734100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 -- 172.21.15.38:0/3472862439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 msgr2=0x7f8734100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 0x7f8734100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 -- 172.21.15.38:0/3472862439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341aa820 con 0x7f8734107940 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 0x7f8734100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87337fe640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f872c02c790 tx=0x7f872c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87388f2640 1 -- 172.21.15.38:0/3472862439 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c039050 con 0x7f8734107940 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87388f2640 1 -- 172.21.15.38:0/3472862439 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c045070 con 0x7f8734107940 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87341aa9f0 con 0x7f8734107940 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87388f2640 1 -- 172.21.15.38:0/3472862439 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c0044e0 con 0x7f8734107940 2024-04-22T00:09:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.595+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87341aaee0 con 0x7f8734107940 2024-04-22T00:09:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.598+0000 7f873c5ba740 1 Processor -- start 2024-04-22T00:09:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.598+0000 7f87388f2640 1 -- 172.21.15.38:0/3472862439 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f872c004680 con 0x7f8734107940 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 -- start start 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55790d948b70 con 0x55790d9ee210 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55790d8f1470 con 0x55790d9e5440 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f873c5ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55790d944b30 con 0x55790d9e4b20 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53634/0 (socket says 172.21.15.38:53634) 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 -- 172.21.15.38:0/169805739 learned_addr learned my addr 172.21.15.38:0/169805739 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f8732ffd640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87388f2640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 0x7f871c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 -- 172.21.15.38:0/169805739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 msgr2=0x55790d9e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 -- 172.21.15.38:0/169805739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 msgr2=0x55790d9edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f87327fc640 1 -- 172.21.15.38:0/169805739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790d65d210 con 0x55790d9ee210 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.599+0000 7f8732ffd640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f87337fe640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 0x7f871c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f87327fc640 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f87280058c0 tx=0x7f8728005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71a6642aff94580 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f87388f2640 1 -- 172.21.15.38:0/3472862439 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f872c0f1e20 con 0x7f8734107940 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f8719ffb640 1 -- 172.21.15.38:0/169805739 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728004030 con 0x55790d9ee210 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f8719ffb640 1 -- 172.21.15.38:0/169805739 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8728005500 con 0x55790d9ee210 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.600+0000 7f8719ffb640 1 -- 172.21.15.38:0/169805739 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728004d00 con 0x55790d9ee210 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- 172.21.15.38:0/169805739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 msgr2=0x55790d9f0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.604 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f87337fe640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 0x7f871c0a5260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ef780 tx=0x7f872c02c180 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f87280058c0 tx=0x7f8728005850 comp rx=0 tx=0).stop 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- 172.21.15.38:0/169805739 shutdown_connections 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9f0600 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 --2- 172.21.15.38:0/169805739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- 172.21.15.38:0/169805739 >> 172.21.15.38:0/169805739 conn(0x55790d9df440 msgr2=0x55790d9df820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- 172.21.15.38:0/169805739 shutdown_connections 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- 172.21.15.38:0/169805739 wait complete. 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 -- start start 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.601+0000 7f873c5ba740 1 Processor -- start 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e4b20 0x55790d9eb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55790d9f0e40 con 0x55790d9ee210 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55790d904a10 con 0x55790d9e4b20 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55790d9f0fc0 con 0x55790d9e5440 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f873c5ba740 10 client.0 _ll_register_callbacks cb 0x55790d93fb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53644/0 (socket says 172.21.15.38:53644) 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f87327fc640 1 -- 172.21.15.38:0/3326852053 learned_addr learned my addr 172.21.15.38:0/3326852053 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f87327fc640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e4b20 0x55790d9eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 -- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e4b20 msgr2=0x55790d9eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.606 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e4b20 0x55790d9eb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 -- 172.21.15.38:0/3326852053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 msgr2=0x55790d9ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 -- 172.21.15.38:0/3326852053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790d729620 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f87327fc640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8732ffd640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9eb7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8724012700 tx=0x7f8724012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d4a55f06179457e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.602+0000 7f8712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724011c70 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8728005500 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f8724011e10 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8724004020 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 10 client.27235 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55790d9fd150 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8728005500 con 0x55790d9e5440 2024-04-22T00:09:20.607 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f872401c470 con 0x55790d9e5440 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 10 client.27235 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 10 client.27235 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 20 client.27235 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790d9fe610 0x55790da00a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 4 client.27235 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.603+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55790d9560b0 con 0x55790d9fe610 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.604+0000 7f87327fc640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790d9fe610 0x55790da00a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.604+0000 7f87327fc640 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790d9fe610 0x55790da00a20 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd26651652d68b37 server_cookie=3450149619af6e51 in_seq=0 out_seq=0 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.604+0000 7f8712ffd640 10 client.27235 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f8712ffd640 1 -- 172.21.15.38:0/3326852053 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55790d9560b0 con 0x55790d9fe610 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f8712ffd640 10 client.27235 handle_command_reply: tid=0 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 client.27235 shutdown 2024-04-22T00:09:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 msgr2=0x55790d9eb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9eb7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8724012700 tx=0x7f8724012bc0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 shutdown_connections 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790d9fe610 0x55790da00a20 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9eb7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f8712ffd640 0 client.27235 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 --2- 172.21.15.38:0/3326852053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e4b20 0x55790d9eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 >> 172.21.15.38:0/3326852053 conn(0x55790d9df440 msgr2=0x55790d9e1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 shutdown_connections 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 1 -- 172.21.15.38:0/3326852053 wait complete. 2024-04-22T00:09:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.605+0000 7f873c5ba740 20 client.27235 trim_cache size 0 max 16384 2024-04-22T00:09:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.620+0000 7f873c5ba740 1 Processor -- start 2024-04-22T00:09:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.620+0000 7f873c5ba740 1 -- start start 2024-04-22T00:09:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9ed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9f9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b5220 con 0x55790d9ee210 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b6d60 con 0x55790d9e5440 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f873c5ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b58f0 con 0x55790d9e4b20 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53658/0 (socket says 172.21.15.38:53658) 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 -- 172.21.15.38:0/815724833 learned_addr learned my addr 172.21.15.38:0/815724833 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f8732ffd640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9f9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 -- 172.21.15.38:0/815724833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 msgr2=0x55790d9ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 -- 172.21.15.38:0/815724833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 msgr2=0x55790d9f9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9f9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 -- 172.21.15.38:0/815724833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790da010a0 con 0x55790d9ee210 2024-04-22T00:09:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f8732ffd640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9f9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.621+0000 7f87327fc640 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8728004990 tx=0x7f87280048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad40fc13d5c9d8b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f87197fa640 1 -- 172.21.15.38:0/815724833 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728011070 con 0x55790d9ee210 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f87197fa640 1 -- 172.21.15.38:0/815724833 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8728011210 con 0x55790d9ee210 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f87197fa640 1 -- 172.21.15.38:0/815724833 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728031940 con 0x55790d9ee210 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- 172.21.15.38:0/815724833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 msgr2=0x55790d9da450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8728004990 tx=0x7f87280048e0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- 172.21.15.38:0/815724833 shutdown_connections 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9ee210 0x55790d9da450 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9e5440 0x55790d9f9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 --2- 172.21.15.38:0/815724833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e4b20 0x55790d9ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- 172.21.15.38:0/815724833 >> 172.21.15.38:0/815724833 conn(0x55790d9df440 msgr2=0x55790d9eb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- 172.21.15.38:0/815724833 shutdown_connections 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- 172.21.15.38:0/815724833 wait complete. 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.622+0000 7f873c5ba740 1 -- start start 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 Processor -- start 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9ee210 0x55790d9e8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b6690 con 0x55790d9e4b20 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b5fc0 con 0x55790d9ee210 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87341b3220 con 0x55790d9e5440 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f873c5ba740 10 client.0 _ll_register_callbacks cb 0x7f87240097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53664/0 (socket says 172.21.15.38:53664) 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 -- 172.21.15.38:0/1466688739 learned_addr learned my addr 172.21.15.38:0/1466688739 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f8732ffd640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 -- 172.21.15.38:0/1466688739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 msgr2=0x55790d9e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 -- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9ee210 msgr2=0x55790d9e8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9ee210 0x55790d9e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 -- 172.21.15.38:0/1466688739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790da63ba0 con 0x55790d9e4b20 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f87327fc640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f8728011770 tx=0x7f87280319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff0e10bfecda7aa server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.623+0000 7f8712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87280314d0 con 0x55790d9e4b20 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9526+0+0 (secure 0 0 0) 0x7f8728019070 con 0x55790d9e4b20 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8728011210 con 0x55790d9e4b20 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8728031670 con 0x55790d9e4b20 2024-04-22T00:09:20.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 10 client.18078 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55790da64cd0 con 0x55790d9e4b20 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8728011210 con 0x55790d9e4b20 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f872801f9b0 con 0x55790d9e4b20 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 10 client.18078 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 10 client.18078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 20 client.18078 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790da651d0 0x55790da675f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 4 client.18078 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55790d950400 con 0x55790da651d0 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.624+0000 7f8732ffd640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790da651d0 0x55790da675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.625+0000 7f8732ffd640 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790da651d0 0x55790da675f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f14b5da950f595 server_cookie=652f088f50fb565e in_seq=0 out_seq=0 2024-04-22T00:09:20.629 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.625+0000 7f8712ffd640 10 client.18078 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f8712ffd640 1 -- 172.21.15.38:0/1466688739 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55790d950400 con 0x55790da651d0 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f8712ffd640 10 client.18078 handle_command_reply: tid=0 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 client.18078 shutdown 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 msgr2=0x55790d9f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f8728011770 tx=0x7f87280319b0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 shutdown_connections 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55790d9ee210 0x55790d9e8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55790d9e5440 0x55790d9e7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55790da651d0 0x55790da675f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 --2- 172.21.15.38:0/1466688739 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55790d9e4b20 0x55790d9f6760 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 >> 172.21.15.38:0/1466688739 conn(0x55790d9df440 msgr2=0x55790d9ebab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f8712ffd640 0 client.18078 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 shutdown_connections 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 1 -- 172.21.15.38:0/1466688739 wait complete. 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.647+0000 7f873c5ba740 20 client.18078 trim_cache size 0 max 16384 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:09:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:09:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 msgr2=0x7f871c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 0x7f871c0a5260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f872c0ef780 tx=0x7f872c02c180 comp rx=0 tx=0).stop 2024-04-22T00:09:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 msgr2=0x7f87341014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f872c02c790 tx=0x7f872c0346d0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 shutdown_connections 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f871c0a2da0 0x7f871c0a5260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8734107940 0x7f87341014e0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f873406a930 0x7f8734100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 --2- 172.21.15.38:0/3472862439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8734069f70 0x7f8734100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 >> 172.21.15.38:0/3472862439 conn(0x7f8734076440 msgr2=0x7f87340ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 shutdown_connections 2024-04-22T00:09:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:20.648+0000 7f87398f4640 1 -- 172.21.15.38:0/3472862439 wait complete. 2024-04-22T00:09:22.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:22 smithi038 ceph-mon[27972]: pgmap v222: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.2 MiB/s wr, 1.07k op/s 2024-04-22T00:09:22.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:22 smithi038 ceph-mon[27972]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T00:09:22.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:22 smithi203 ceph-mon[31431]: pgmap v222: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.2 MiB/s wr, 1.07k op/s 2024-04-22T00:09:22.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:22 smithi203 ceph-mon[31431]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T00:09:22.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:22 smithi111 ceph-mon[31543]: pgmap v222: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 4.2 MiB/s wr, 1.07k op/s 2024-04-22T00:09:22.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:22 smithi111 ceph-mon[31543]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T00:09:24.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:24 smithi038 ceph-mon[27972]: pgmap v224: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.0 MiB/s wr, 546 op/s 2024-04-22T00:09:24.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:24 smithi203 ceph-mon[31431]: pgmap v224: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.0 MiB/s wr, 546 op/s 2024-04-22T00:09:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:24 smithi111 ceph-mon[31543]: pgmap v224: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.0 MiB/s wr, 546 op/s 2024-04-22T00:09:26.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:26 smithi038 ceph-mon[27972]: pgmap v225: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.3 MiB/s wr, 381 op/s 2024-04-22T00:09:26.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:26 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:26.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:26 smithi203 ceph-mon[31431]: pgmap v225: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.3 MiB/s wr, 381 op/s 2024-04-22T00:09:26.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:26 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:26.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:26 smithi111 ceph-mon[31543]: pgmap v225: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 1.3 MiB/s wr, 381 op/s 2024-04-22T00:09:26.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:26 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:27.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:27 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:27.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:27 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:27.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:27 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:27.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:27 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:27.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:27 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:27.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:27 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:27.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:27 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:27.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:27 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:27 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:27 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:27 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:27.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:27 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:28.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:28 smithi038 ceph-mon[27972]: pgmap v226: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 891 KiB/s wr, 256 op/s 2024-04-22T00:09:28.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:28 smithi203 ceph-mon[31431]: pgmap v226: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 891 KiB/s wr, 256 op/s 2024-04-22T00:09:28.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:28 smithi111 ceph-mon[31543]: pgmap v226: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 891 KiB/s wr, 256 op/s 2024-04-22T00:09:29.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:29 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:09:29.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:29 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:09:29.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:29 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:09:30.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:30 smithi038 ceph-mon[27972]: pgmap v227: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 641 KiB/s wr, 172 op/s 2024-04-22T00:09:30.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:30 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:09:30.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:30 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:09:30.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:30 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:09:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:30 smithi203 ceph-mon[31431]: pgmap v227: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 641 KiB/s wr, 172 op/s 2024-04-22T00:09:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:30 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:09:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:30 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:09:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:30 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:09:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:30 smithi111 ceph-mon[31543]: pgmap v227: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 641 KiB/s wr, 172 op/s 2024-04-22T00:09:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:30 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:09:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:30 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:09:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:30 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:09:32.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:32 smithi038 ceph-mon[27972]: pgmap v228: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 525 KiB/s wr, 140 op/s 2024-04-22T00:09:32.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:32 smithi203 ceph-mon[31431]: pgmap v228: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 525 KiB/s wr, 140 op/s 2024-04-22T00:09:32.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:32 smithi111 ceph-mon[31543]: pgmap v228: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 525 KiB/s wr, 140 op/s 2024-04-22T00:09:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.747+0000 7fad44f8c640 1 -- 172.21.15.38:0/821070969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40102100 msgr2=0x7fad4010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.747+0000 7fad44f8c640 1 --2- 172.21.15.38:0/821070969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40102100 0x7fad4010e610 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fad38009cd0 tx=0x7fad3802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 -- 172.21.15.38:0/821070969 shutdown_connections 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 --2- 172.21.15.38:0/821070969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40102100 0x7fad4010e610 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 --2- 172.21.15.38:0/821070969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40101740 0x7fad40101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 --2- 172.21.15.38:0/821070969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 0x7fad40107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 -- 172.21.15.38:0/821070969 >> 172.21.15.38:0/821070969 conn(0x7fad400fd210 msgr2=0x7fad400ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 -- 172.21.15.38:0/821070969 shutdown_connections 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 -- 172.21.15.38:0/821070969 wait complete. 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 Processor -- start 2024-04-22T00:09:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.748+0000 7fad44f8c640 1 -- start start 2024-04-22T00:09:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 0x7fad40073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 0x7fad400744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad40111b60 con 0x7fad40101740 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad401119e0 con 0x7fad40102100 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad44f8c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad40111ce0 con 0x7fad40107750 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50122/0 (socket says 172.21.15.38:50122) 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 -- 172.21.15.38:0/2597653459 learned_addr learned my addr 172.21.15.38:0/2597653459 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad37fff640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 0x7fad40073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 0x7fad400744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 -- 172.21.15.38:0/2597653459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 msgr2=0x7fad400744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 0x7fad400744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 -- 172.21.15.38:0/2597653459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 msgr2=0x7fad40073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 0x7fad40073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 -- 172.21.15.38:0/2597653459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad401a6440 con 0x7fad40101740 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad37fff640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 0x7fad40073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.749+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fad3000d770 tx=0x7fad3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.750+0000 7fad377fe640 1 -- 172.21.15.38:0/2597653459 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30017070 con 0x7fad40101740 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.750+0000 7fad377fe640 1 -- 172.21.15.38:0/2597653459 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad30005500 con 0x7fad40101740 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.750+0000 7fad377fe640 1 -- 172.21.15.38:0/2597653459 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30004e00 con 0x7fad40101740 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.750+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad401a6730 con 0x7fad40101740 2024-04-22T00:09:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.750+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad401a6c80 con 0x7fad40101740 2024-04-22T00:09:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.752+0000 7fad377fe640 1 -- 172.21.15.38:0/2597653459 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fad30004fa0 con 0x7fad40101740 2024-04-22T00:09:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.752+0000 7fad47b78740 1 Processor -- start 2024-04-22T00:09:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 -- start start 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba83cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83d440 0x5622ba845cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5622ba7a0b70 con 0x5622ba846210 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622ba749470 con 0x5622ba83cb20 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad47b78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5622ba79cb30 con 0x5622ba83d440 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad377fe640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 0x7fad2c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50134/0 (socket says 172.21.15.38:50134) 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad37fff640 1 -- 172.21.15.38:0/1077224968 learned_addr learned my addr 172.21.15.38:0/1077224968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba83cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad3e59c640 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83d440 0x5622ba845cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 0x7fad2c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad377fe640 1 -- 172.21.15.38:0/2597653459 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fad300cf230 con 0x7fad40101740 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad37fff640 1 -- 172.21.15.38:0/1077224968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83d440 msgr2=0x5622ba845cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.753+0000 7fad37fff640 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83d440 0x5622ba845cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad37fff640 1 -- 172.21.15.38:0/1077224968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 msgr2=0x5622ba83cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad37fff640 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba83cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad37fff640 1 -- 172.21.15.38:0/1077224968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ba4b5210 con 0x5622ba846210 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad37fff640 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fad2800bcb0 tx=0x7fad2800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfffff5d7fdad24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 0x7fad2c0a52e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad3802c630 tx=0x7fad380095c0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad34ff9640 1 -- 172.21.15.38:0/1077224968 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28011440 con 0x5622ba846210 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad34ff9640 1 -- 172.21.15.38:0/1077224968 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad28011ac0 con 0x5622ba846210 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.754+0000 7fad34ff9640 1 -- 172.21.15.38:0/1077224968 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2801bb50 con 0x5622ba846210 2024-04-22T00:09:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- 172.21.15.38:0/1077224968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 msgr2=0x5622ba848600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fad2800bcb0 tx=0x7fad2800f730 comp rx=0 tx=0).stop 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- 172.21.15.38:0/1077224968 shutdown_connections 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba848600 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83d440 0x5622ba845cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 --2- 172.21.15.38:0/1077224968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba83cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- 172.21.15.38:0/1077224968 >> 172.21.15.38:0/1077224968 conn(0x5622ba837440 msgr2=0x5622ba837820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- 172.21.15.38:0/1077224968 shutdown_connections 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- 172.21.15.38:0/1077224968 wait complete. 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 -- start start 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.755+0000 7fad47b78740 1 Processor -- start 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba8432c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba846210 0x5622ba843d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5622ba848e40 con 0x5622ba83d440 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5622ba75ca10 con 0x5622ba83cb20 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5622ba848fc0 con 0x5622ba846210 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 10 client.0 _ll_register_callbacks cb 0x5622ba797b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50140/0 (socket says 172.21.15.38:50140) 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 -- 172.21.15.38:0/2719936296 learned_addr learned my addr 172.21.15.38:0/2719936296 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba846210 0x5622ba843d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba8432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 -- 172.21.15.38:0/2719936296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba846210 msgr2=0x5622ba843d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba846210 0x5622ba843d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 -- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 msgr2=0x5622ba8432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba8432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 -- 172.21.15.38:0/2719936296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ba581620 con 0x5622ba83d440 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba8432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad37fff640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fad2800fc30 tx=0x7fad2800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2100f0a38569deea server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2802a060 con 0x5622ba83d440 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fad28004250 con 0x5622ba83d440 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.756+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad28011ac0 con 0x5622ba83d440 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fad28031050 con 0x5622ba83d440 2024-04-22T00:09:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 10 client.18090 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5622ba855130 con 0x5622ba83d440 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad28011ac0 con 0x5622ba83d440 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fad2801bbd0 con 0x5622ba83d440 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 10 client.18090 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 10 client.18090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 20 client.18090 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba856530 0x5622ba858920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 4 client.18090 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622ba7ae0b0 con 0x5622ba856530 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.757+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba856530 0x5622ba858920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.758+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba856530 0x5622ba858920 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd910e31a634d228 server_cookie=c426019826035f in_seq=0 out_seq=0 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.758+0000 7fad25ffb640 10 client.18090 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad25ffb640 1 -- 172.21.15.38:0/2719936296 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622ba7ae0b0 con 0x5622ba856530 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad25ffb640 10 client.18090 handle_command_reply: tid=0 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 client.18090 shutdown 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 msgr2=0x5622ba843800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fad2800fc30 tx=0x7fad2800bfd0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 shutdown_connections 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba846210 0x5622ba843d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba856530 0x5622ba858920 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba843800 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 --2- 172.21.15.38:0/2719936296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83cb20 0x5622ba8432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad25ffb640 0 client.18090 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 >> 172.21.15.38:0/2719936296 conn(0x5622ba837440 msgr2=0x5622ba839190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 shutdown_connections 2024-04-22T00:09:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 1 -- 172.21.15.38:0/2719936296 wait complete. 2024-04-22T00:09:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.759+0000 7fad47b78740 20 client.18090 trim_cache size 0 max 16384 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 Processor -- start 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 -- start start 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba8459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba846210 0x5622ba840520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad401b0d60 con 0x5622ba83d440 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad401b28a0 con 0x5622ba846210 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad47b78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad401b1430 con 0x5622ba83cb20 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.774+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50154/0 (socket says 172.21.15.38:50154) 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 -- 172.21.15.38:0/2720054283 learned_addr learned my addr 172.21.15.38:0/2720054283 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad3e59c640 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba8459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba846210 0x5622ba840520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 -- 172.21.15.38:0/2720054283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 msgr2=0x5622ba8459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba8459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 -- 172.21.15.38:0/2720054283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba846210 msgr2=0x5622ba840520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba846210 0x5622ba840520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 -- 172.21.15.38:0/2720054283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ba841810 con 0x5622ba83d440 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad37fff640 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fad280059f0 tx=0x7fad2800bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f2bba09a3119d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad27fff640 1 -- 172.21.15.38:0/2720054283 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2802a040 con 0x5622ba83d440 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad27fff640 1 -- 172.21.15.38:0/2720054283 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad2800fc60 con 0x5622ba83d440 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.775+0000 7fad27fff640 1 -- 172.21.15.38:0/2720054283 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28011880 con 0x5622ba83d440 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- 172.21.15.38:0/2720054283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 msgr2=0x5622ba832220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fad280059f0 tx=0x7fad2800bb80 comp rx=0 tx=0).stop 2024-04-22T00:09:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- 172.21.15.38:0/2720054283 shutdown_connections 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba846210 0x5622ba840520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba83d440 0x5622ba832220 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- 172.21.15.38:0/2720054283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba8459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- 172.21.15.38:0/2720054283 >> 172.21.15.38:0/2720054283 conn(0x5622ba837440 msgr2=0x5622ba83dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- 172.21.15.38:0/2720054283 shutdown_connections 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- 172.21.15.38:0/2720054283 wait complete. 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- start start 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 Processor -- start 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba85b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83d440 0x5622ba858d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fad401b21d0 con 0x5622ba846210 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fad401b1b00 con 0x5622ba83d440 2024-04-22T00:09:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.776+0000 7fad47b78740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad401aed60 con 0x5622ba83cb20 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad47b78740 10 client.0 _ll_register_callbacks cb 0x7fad28015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50166/0 (socket says 172.21.15.38:50166) 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 -- 172.21.15.38:0/774303494 learned_addr learned my addr 172.21.15.38:0/774303494 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad37fff640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83d440 0x5622ba858d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3e59c640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba85b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 -- 172.21.15.38:0/774303494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 msgr2=0x5622ba85b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba85b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 -- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83d440 msgr2=0x5622ba858d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83d440 0x5622ba858d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 -- 172.21.15.38:0/774303494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ba8597d0 con 0x5622ba846210 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad3ed9d640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fad38009cd0 tx=0x7fad3802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11b93ff2eb7fdc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38056060 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fad38004240 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.777+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad2800fc60 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fad38039070 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 10 client.18096 fetch_fsmap learned FSMap version 35 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5622ba8bc3d0 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad2800fc60 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fad38034770 con 0x5622ba846210 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 10 client.18096 fetch_fsmap finished waiting for FSMap version 35 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 10 client.18096 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 20 client.18096 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba8bcb80 0x5622ba8befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 4 client.18096 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622ba7a8400 con 0x5622ba8bcb80 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.778+0000 7fad3e59c640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba8bcb80 0x5622ba8befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.779+0000 7fad25ffb640 10 client.18096 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.779+0000 7fad3e59c640 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba8bcb80 0x5622ba8befa0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24959dcb22ff2681 server_cookie=5bcfc5a7dc357f56 in_seq=0 out_seq=0 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad25ffb640 1 -- 172.21.15.38:0/774303494 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622ba7a8400 con 0x5622ba8bcb80 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad25ffb640 10 client.18096 handle_command_reply: tid=0 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 client.18096 shutdown 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 msgr2=0x5622ba859290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fad38009cd0 tx=0x7fad3802e040 comp rx=0 tx=0).stop 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 shutdown_connections 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5622ba8bcb80 0x5622ba8befa0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5622ba846210 0x5622ba859290 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5622ba83d440 0x5622ba858d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad25ffb640 0 client.18096 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 --2- 172.21.15.38:0/774303494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5622ba83cb20 0x5622ba85b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 >> 172.21.15.38:0/774303494 conn(0x5622ba837440 msgr2=0x5622ba83da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 shutdown_connections 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 1 -- 172.21.15.38:0/774303494 wait complete. 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.794+0000 7fad47b78740 20 client.18096 trim_cache size 0 max 16384 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-04-22T00:09:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 msgr2=0x7fad2c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 0x7fad2c0a52e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad3802c630 tx=0x7fad380095c0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 msgr2=0x7fad40073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fad3000d770 tx=0x7fad3000dc30 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 shutdown_connections 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fad2c0a2e20 0x7fad2c0a52e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad40107750 0x7fad400744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad40102100 0x7fad40073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 --2- 172.21.15.38:0/2597653459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fad40101740 0x7fad40073a40 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 >> 172.21.15.38:0/2597653459 conn(0x7fad400fd210 msgr2=0x7fad400fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 shutdown_connections 2024-04-22T00:09:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-04-22T00:09:33.795+0000 7fad44f8c640 1 -- 172.21.15.38:0/2597653459 wait complete. 2024-04-22T00:09:34.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:34 smithi038 ceph-mon[27972]: pgmap v229: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 435 KiB/s wr, 118 op/s 2024-04-22T00:09:34.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:34 smithi203 ceph-mon[31431]: pgmap v229: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 435 KiB/s wr, 118 op/s 2024-04-22T00:09:34.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:34 smithi111 ceph-mon[31543]: pgmap v229: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 435 KiB/s wr, 118 op/s 2024-04-22T00:09:35.808 INFO:tasks.workunit.client.0.smithi038.stdout:Syncing()...23 sec 2024-04-22T00:09:35.808 INFO:tasks.workunit.client.0.smithi038.stdout:FFSB benchmark finished at: Mon Apr 22 00:09:35 2024 2024-04-22T00:09:35.808 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.808 INFO:tasks.workunit.client.0.smithi038.stdout:Results: 2024-04-22T00:09:35.808 INFO:tasks.workunit.client.0.smithi038.stdout:Benchmark took 336.23 sec 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:Total Results 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:=============== 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout: ======= ============ ========= ======= =========== ========== 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout: write : 2567680 7636.60 100.000% 100.000% 29.8MB/sec 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:- 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:7636.60 Transactions per Second 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:Throughput Results 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:=================== 2024-04-22T00:09:35.809 INFO:tasks.workunit.client.0.smithi038.stdout:Write Throughput: 29.8MB/sec 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout:System Call Latency statistics in millisecs 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout:===== 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout: Min Avg Max Total Calls 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout: ======== ======== ======== ============ 2024-04-22T00:09:35.822 INFO:tasks.workunit.client.0.smithi038.stdout:[ open] 0.006000 0.083117 30.780001 2006 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[7] 20.000000 - 50.000000 : 1 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[8] 10.000000 - 20.000000 : 2 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[9] 5.000000 - 10.000000 : 6 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[10] 2.000000 - 5.000000 : 6 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[11] 1.000000 - 2.000000 : 2 2024-04-22T00:09:35.823 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[12] 0.500000 - 1.000000 : 5 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[14] 0.100000 - 0.200000 : 6 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[15] 0.050000 - 0.100000 : 45 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[16] 0.020000 - 0.050000 : 438 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[17] 0.010000 - 0.020000 : 1130 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[18] 0.000000 - 0.010000 : 364 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout:[ write] 0.001000 3.804094 52453.414062 2567680 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[0] 5000.000000 - 10000.000000 : 354 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[1] 2000.000000 - 5000.000000 : 365 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[2] 1000.000000 - 2000.000000 : 270 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[3] 500.000000 - 1000.000000 : 278 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[4] 200.000000 - 500.000000 : 355 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[5] 100.000000 - 200.000000 : 273 2024-04-22T00:09:35.824 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[6] 50.000000 - 100.000000 : 292 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[7] 20.000000 - 50.000000 : 731 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[8] 10.000000 - 20.000000 : 523 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[9] 5.000000 - 10.000000 : 553 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[10] 2.000000 - 5.000000 : 522 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[11] 1.000000 - 2.000000 : 246 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[12] 0.500000 - 1.000000 : 201 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[13] 0.200000 - 0.500000 : 197 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[14] 0.100000 - 0.200000 : 211 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[15] 0.050000 - 0.100000 : 564 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[16] 0.020000 - 0.050000 : 3931 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[17] 0.010000 - 0.020000 : 13316 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[18] 0.000000 - 0.010000 : 2544219 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout:[ lseek] 0.000000 0.001574 60.741001 2567680 2024-04-22T00:09:35.825 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[7] 20.000000 - 50.000000 : 42 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[8] 10.000000 - 20.000000 : 43 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[9] 5.000000 - 10.000000 : 38 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[10] 2.000000 - 5.000000 : 45 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[11] 1.000000 - 2.000000 : 22 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[12] 0.500000 - 1.000000 : 11 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[13] 0.200000 - 0.500000 : 23 2024-04-22T00:09:35.826 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[14] 0.100000 - 0.200000 : 18 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[15] 0.050000 - 0.100000 : 60 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[16] 0.020000 - 0.050000 : 359 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[17] 0.010000 - 0.020000 : 753 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[18] 0.000000 - 0.010000 : 2566264 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout:[ close] 0.001000 0.039734 68.203003 2006 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-22T00:09:35.827 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[11] 1.000000 - 2.000000 : 1 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[15] 0.050000 - 0.100000 : 1 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[16] 0.020000 - 0.050000 : 7 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[17] 0.010000 - 0.020000 : 14 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: msec_range[18] 0.000000 - 0.010000 : 1982 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-04-22T00:09:35.828 INFO:tasks.workunit.client.0.smithi038.stdout:0.7% User Time 2024-04-22T00:09:35.829 INFO:tasks.workunit.client.0.smithi038.stdout:2.5% System Time 2024-04-22T00:09:35.829 INFO:tasks.workunit.client.0.smithi038.stdout:3.3% CPU Utilization 2024-04-22T00:09:35.849 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd .. 2024-04-22T00:09:35.851 INFO:tasks.workunit.client.0.smithi038.stderr:+ rm -r tmp ffsb 2024-04-22T00:09:36.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:36 smithi038 ceph-mon[27972]: pgmap v230: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-04-22T00:09:36.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:36 smithi203 ceph-mon[31431]: pgmap v230: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-04-22T00:09:36.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:36 smithi111 ceph-mon[31543]: pgmap v230: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-04-22T00:09:38.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:38 smithi038 ceph-mon[27972]: pgmap v231: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-22T00:09:38.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:38 smithi203 ceph-mon[31431]: pgmap v231: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-22T00:09:38.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:38 smithi111 ceph-mon[31543]: pgmap v231: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 14 op/s 2024-04-22T00:09:38.905 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T00:09:38.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T00:09:38.980 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-22T00:09:38.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T00:09:39.499 DEBUG:teuthology.parallel:result is None 2024-04-22T00:09:39.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T00:09:39.547 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T00:09:39.547 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T00:09:39.559 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:09:39.570 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-22T00:09:39.580 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-22T00:09:39.580 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T00:09:39.580 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:09:39.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-22T00:09:39.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-22T00:09:40.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:40 smithi038 ceph-mon[27972]: pgmap v232: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 148 KiB/s wr, 21 op/s 2024-04-22T00:09:40.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:40 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:40.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:40 smithi203 ceph-mon[31431]: pgmap v232: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 148 KiB/s wr, 21 op/s 2024-04-22T00:09:40.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:40 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:40.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:40 smithi111 ceph-mon[31543]: pgmap v232: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 148 KiB/s wr, 21 op/s 2024-04-22T00:09:40.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:40 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:41.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:41 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:41.782 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:41 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:41.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:41 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:42.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:42 smithi203 ceph-mon[31431]: pgmap v233: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 149 KiB/s wr, 25 op/s 2024-04-22T00:09:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:42 smithi111 ceph-mon[31543]: pgmap v233: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 149 KiB/s wr, 25 op/s 2024-04-22T00:09:42.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:42 smithi038 ceph-mon[27972]: pgmap v233: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 149 KiB/s wr, 25 op/s 2024-04-22T00:09:43.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:43 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:43.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:43 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:43.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:43 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:09:44.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:44 smithi203 ceph-mon[31431]: pgmap v234: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 271 KiB/s wr, 99 op/s 2024-04-22T00:09:44.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:44 smithi111 ceph-mon[31543]: pgmap v234: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 271 KiB/s wr, 99 op/s 2024-04-22T00:09:44.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:44 smithi038 ceph-mon[27972]: pgmap v234: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 271 KiB/s wr, 99 op/s 2024-04-22T00:09:45.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:09:45.250 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:09:45.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.443+0000 7f7fe5597640 1 -- 172.21.15.38:0/3409819021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0108950 msgr2=0x7f7fe0108d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.443+0000 7f7fe5597640 1 --2- 172.21.15.38:0/3409819021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0108950 0x7f7fe0108d30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009900 tx=0x7f7fc802bc50 comp rx=0 tx=0).stop 2024-04-22T00:09:45.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.447+0000 7f7fe5597640 1 -- 172.21.15.38:0/3409819021 shutdown_connections 2024-04-22T00:09:45.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.447+0000 7f7fe5597640 1 --2- 172.21.15.38:0/3409819021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0114890 0x7f7fe0116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.447+0000 7f7fe5597640 1 --2- 172.21.15.38:0/3409819021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fe0109270 0x7f7fe0114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.447+0000 7f7fe5597640 1 --2- 172.21.15.38:0/3409819021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0108950 0x7f7fe0108d30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.447+0000 7f7fe5597640 1 -- 172.21.15.38:0/3409819021 >> 172.21.15.38:0/3409819021 conn(0x7f7fe006c850 msgr2=0x7f7fe006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.452+0000 7f7fe5597640 1 -- 172.21.15.38:0/3409819021 shutdown_connections 2024-04-22T00:09:45.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.455+0000 7f7fe5597640 1 -- 172.21.15.38:0/3409819021 wait complete. 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.456+0000 7f7fe5597640 1 Processor -- start 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.456+0000 7f7fe5597640 1 -- start start 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 0x7f7fe0113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fe0109270 0x7f7fe0114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 0x7f7fe010adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe011a380 con 0x7f7fe0114890 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe011a200 con 0x7f7fe0108950 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fe5597640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe011a500 con 0x7f7fe0109270 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fdf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 0x7f7fe010adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fdf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 0x7f7fe010adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38536/0 (socket says 172.21.15.38:38536) 2024-04-22T00:09:45.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.457+0000 7f7fdf7fe640 1 -- 172.21.15.38:0/2500243846 learned_addr learned my addr 172.21.15.38:0/2500243846 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.459+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 0x7f7fe0113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.460+0000 7f7fdeffd640 1 -- 172.21.15.38:0/2500243846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fe0109270 msgr2=0x7f7fe0114240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:09:45.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.460+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fe0109270 0x7f7fe0114240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.460+0000 7f7fdeffd640 1 -- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 msgr2=0x7f7fe010adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.460+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 0x7f7fe010adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.460+0000 7f7fdeffd640 1 -- 172.21.15.38:0/2500243846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe010b670 con 0x7f7fe0108950 2024-04-22T00:09:45.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.461+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 0x7f7fe0113d00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009230 tx=0x7f7fc8005310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.462+0000 7f7fdcff9640 1 -- 172.21.15.38:0/2500243846 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8041020 con 0x7f7fe0108950 2024-04-22T00:09:45.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.462+0000 7f7fdcff9640 1 -- 172.21.15.38:0/2500243846 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fc802cad0 con 0x7f7fe0108950 2024-04-22T00:09:45.458 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.462+0000 7f7fdcff9640 1 -- 172.21.15.38:0/2500243846 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8004230 con 0x7f7fe0108950 2024-04-22T00:09:45.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.465+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe010b900 con 0x7f7fe0108950 2024-04-22T00:09:45.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.465+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe010f920 con 0x7f7fe0108950 2024-04-22T00:09:45.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.472+0000 7f7fe8180740 1 Processor -- start 2024-04-22T00:09:45.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.476+0000 7f7fdcff9640 1 -- 172.21.15.38:0/2500243846 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7fc8044020 con 0x7f7fe0108950 2024-04-22T00:09:45.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.477+0000 7f7fdcff9640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 0x7f7fcc0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.478+0000 7f7fdcff9640 1 -- 172.21.15.38:0/2500243846 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f7fc80f0740 con 0x7f7fe0108950 2024-04-22T00:09:45.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.478+0000 7f7fe8180740 1 -- start start 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.480+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552c97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.480+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552ca2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fe8180740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563552bfb560 con 0x563552c97850 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fe8180740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563552ba3730 con 0x563552ca0430 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fe8180740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563552bf7520 con 0x563552c96f30 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38544/0 (socket says 172.21.15.38:38544) 2024-04-22T00:09:45.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 -- 172.21.15.38:0/2256542413 learned_addr learned my addr 172.21.15.38:0/2256542413 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 -- 172.21.15.38:0/2256542413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 msgr2=0x563552c97310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:09:45.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552c97310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 -- 172.21.15.38:0/2256542413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 msgr2=0x563552ca2820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:09:45.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552ca2820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.481+0000 7f7fd7fff640 1 -- 172.21.15.38:0/2256542413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635529d97d0 con 0x563552c97850 2024-04-22T00:09:45.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.482+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000a920 tx=0x7f7fd000ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606cb46c7fc15804 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.482+0000 7f7fd57fa640 1 -- 172.21.15.38:0/2256542413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000cd80 con 0x563552c97850 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.482+0000 7f7fd57fa640 1 -- 172.21.15.38:0/2256542413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd000f930 con 0x563552c97850 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.482+0000 7f7fd57fa640 1 -- 172.21.15.38:0/2256542413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0018a10 con 0x563552c97850 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.483+0000 7f7fe8180740 1 -- 172.21.15.38:0/2256542413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 msgr2=0x563552c9fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.483+0000 7f7fe8180740 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000a920 tx=0x7f7fd000ade0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.484+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 0x7f7fcc0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.486+0000 7f7fe8180740 1 -- 172.21.15.38:0/2256542413 shutdown_connections 2024-04-22T00:09:45.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.486+0000 7f7fe8180740 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552ca2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.486+0000 7f7fe8180740 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c97850 0x563552c9fef0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.486+0000 7f7fe8180740 1 --2- 172.21.15.38:0/2256542413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552c97310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.486+0000 7f7fe8180740 1 -- 172.21.15.38:0/2256542413 >> 172.21.15.38:0/2256542413 conn(0x563552c91660 msgr2=0x563552c91a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.487+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 0x7f7fcc0a5390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x563552c927f0 tx=0x7f7fd80094d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.490+0000 7f7fe8180740 1 -- 172.21.15.38:0/2256542413 shutdown_connections 2024-04-22T00:09:45.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.492+0000 7f7fe8180740 1 -- 172.21.15.38:0/2256542413 wait complete. 2024-04-22T00:09:45.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.492+0000 7f7fe8180740 1 -- start start 2024-04-22T00:09:45.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.493+0000 7f7fe8180740 1 Processor -- start 2024-04-22T00:09:45.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552c9daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552ca0430 0x563552c9e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563552ca3220 con 0x563552c96f30 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x563552bb6e00 con 0x563552c97850 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563552ca33a0 con 0x563552ca0430 2024-04-22T00:09:45.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.495+0000 7f7fe8180740 10 client.0 _ll_register_callbacks cb 0x563552bf1eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552c9daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38560/0 (socket says 172.21.15.38:38560) 2024-04-22T00:09:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1606381221 learned_addr learned my addr 172.21.15.38:0/1606381221 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552ca0430 0x563552c9e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1606381221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552ca0430 msgr2=0x563552c9e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552ca0430 0x563552c9e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 msgr2=0x563552c9daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.497+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552c9daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.498+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1606381221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56355290f3f0 con 0x563552c96f30 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.498+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80b9350 tx=0x7f7fc8004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca56b88cdd84dfcc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.498+0000 7f7fc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:45.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.505+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80d5060 con 0x563552c96f30 2024-04-22T00:09:45.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.505+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd000f930 con 0x563552c96f30 2024-04-22T00:09:45.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.505+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f7fc80edd20 con 0x563552c96f30 2024-04-22T00:09:45.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.506+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7fc80da050 con 0x563552c96f30 2024-04-22T00:09:45.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.506+0000 7f7fe8180740 10 client.18108 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:45.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.506+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5635528cf0f0 con 0x563552c96f30 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd000f930 con 0x563552c96f30 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f7fc80f02d0 con 0x563552c96f30 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 10 client.18108 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 10 client.18108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 20 client.18108 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552cb09a0 0x563552cb2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 4 client.18108 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563552c08aa0 con 0x563552cb09a0 2024-04-22T00:09:45.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.508+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552cb09a0 0x563552cb2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.519+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552cb09a0 0x563552cb2d90 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30f3805d4113a90a server_cookie=fdd2585a02076e82 in_seq=0 out_seq=0 2024-04-22T00:09:45.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.519+0000 7f7fc67fc640 10 client.18108 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:45.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.523+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1606381221 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563552c08aa0 con 0x563552cb09a0 2024-04-22T00:09:45.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.523+0000 7f7fc67fc640 10 client.18108 handle_command_reply: tid=0 2024-04-22T00:09:45.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.525+0000 7f7fe8180740 1 client.18108 shutdown 2024-04-22T00:09:45.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.527+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 msgr2=0x563552c9d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.527+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80b9350 tx=0x7f7fc8004bb0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 shutdown_connections 2024-04-22T00:09:45.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552ca0430 0x563552c9e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552c9daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552cb09a0 0x563552cb2d90 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1606381221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9d5b0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.532+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 >> 172.21.15.38:0/1606381221 conn(0x563552c91660 msgr2=0x563552c933b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.534+0000 7f7fc67fc640 0 client.18108 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:45.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.534+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 shutdown_connections 2024-04-22T00:09:45.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.534+0000 7f7fe8180740 1 -- 172.21.15.38:0/1606381221 wait complete. 2024-04-22T00:09:45.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.535+0000 7f7fe8180740 20 client.18108 trim_cache size 0 max 16384 2024-04-22T00:09:45.567 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.570+0000 7f7fe8180740 1 Processor -- start 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.570+0000 7f7fe8180740 1 -- start start 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.570+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 0x563552cac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552c9c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fe8180740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c61a0 con 0x563552c96f30 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fe8180740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c7da0 con 0x563552ca0430 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fe8180740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c68a0 con 0x563552c97850 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38576/0 (socket says 172.21.15.38:38576) 2024-04-22T00:09:45.568 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.571+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1182286891 learned_addr learned my addr 172.21.15.38:0/1182286891 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.573+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 0x563552cac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.569 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.573+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552c9c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1182286891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 msgr2=0x563552c9c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552c9c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1182286891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 msgr2=0x563552c9fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1182286891 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563552c96020 con 0x563552c97850 2024-04-22T00:09:45.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552c9c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:45.575 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.576+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:45.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.580+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 0x563552cac0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00058c0 tx=0x7f7fd0018510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7722d684c3d9d3e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.585+0000 7f7fd4ff9640 1 -- 172.21.15.38:0/1182286891 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000cca0 con 0x563552c97850 2024-04-22T00:09:45.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.585+0000 7f7fd4ff9640 1 -- 172.21.15.38:0/1182286891 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd002c020 con 0x563552c97850 2024-04-22T00:09:45.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.585+0000 7f7fd4ff9640 1 -- 172.21.15.38:0/1182286891 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0028820 con 0x563552c97850 2024-04-22T00:09:45.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 -- 172.21.15.38:0/1182286891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 msgr2=0x563552cac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 0x563552cac0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00058c0 tx=0x7f7fd0018510 comp rx=0 tx=0).stop 2024-04-22T00:09:45.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 -- 172.21.15.38:0/1182286891 shutdown_connections 2024-04-22T00:09:45.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552ca0430 0x563552c9c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c97850 0x563552cac0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1182286891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552c96f30 0x563552c9fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 -- 172.21.15.38:0/1182286891 >> 172.21.15.38:0/1182286891 conn(0x563552c91660 msgr2=0x563552c8fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.586+0000 7f7fe8180740 1 -- 172.21.15.38:0/1182286891 shutdown_connections 2024-04-22T00:09:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 -- 172.21.15.38:0/1182286891 wait complete. 2024-04-22T00:09:45.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 -- start start 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 Processor -- start 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552ca88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552ca3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 0x563552ca4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c76d0 con 0x563552ca0430 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c6fa0 con 0x563552c97850 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe01c41a0 con 0x563552c96f30 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.587+0000 7f7fe8180740 10 client.0 _ll_register_callbacks cb 0x7f7fc8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552ca88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552ca3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fdeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552ca88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:58336/0 (socket says 172.21.15.38:58336) 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fdf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 0x563552ca4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fdeffd640 1 -- 172.21.15.38:0/1385053742 learned_addr learned my addr 172.21.15.38:0/1385053742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1385053742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 msgr2=0x563552ca88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552ca88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1385053742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 msgr2=0x563552ca4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 0x563552ca4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 -- 172.21.15.38:0/1385053742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563552ca4860 con 0x563552c97850 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fdf7fe640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 0x563552ca4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fd7fff640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552ca3c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0018580 tx=0x7f7fd000cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7db0d683d3a1d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.588+0000 7f7fc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00285a0 con 0x563552c97850 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd002c020 con 0x563552c97850 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f7fd0028740 con 0x563552c97850 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7fd0005040 con 0x563552c97850 2024-04-22T00:09:45.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fe8180740 10 client.27416 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:45.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.589+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563552d16de0 con 0x563552c97850 2024-04-22T00:09:45.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd002c020 con 0x563552c97850 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f7fd0020040 con 0x563552c97850 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 10 client.27416 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 10 client.27416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 20 client.27416 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552d17310 0x563552d19730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 4 client.27416 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.590+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563552c02df0 con 0x563552d17310 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552d17310 0x563552d19730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fc67fc640 10 client.27416 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fdeffd640 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552d17310 0x563552d19730 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d393173f6d87039 server_cookie=76bb8eb44c3f5fb6 in_seq=0 out_seq=0 2024-04-22T00:09:45.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fc67fc640 1 -- 172.21.15.38:0/1385053742 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563552c02df0 con 0x563552d17310 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fc67fc640 10 client.27416 handle_command_reply: tid=0 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fe8180740 1 client.27416 shutdown 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 msgr2=0x563552ca3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.591+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552ca3c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0018580 tx=0x7f7fd000cca0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 shutdown_connections 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x563552d17310 0x563552d19730 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563552ca0430 0x563552ca4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fc67fc640 0 client.27416 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563552c97850 0x563552ca3c30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 --2- 172.21.15.38:0/1385053742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563552c96f30 0x563552ca88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 >> 172.21.15.38:0/1385053742 conn(0x563552c91660 msgr2=0x563552c8fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 shutdown_connections 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 1 -- 172.21.15.38:0/1385053742 wait complete. 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe8180740 20 client.27416 trim_cache size 0 max 16384 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:09:45.592 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 msgr2=0x7f7fcc0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 0x7f7fcc0a5390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x563552c927f0 tx=0x7f7fd80094d0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 msgr2=0x7f7fe0113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.592+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 0x7f7fe0113d00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009230 tx=0x7f7fc8005310 comp rx=0 tx=0).stop 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 shutdown_connections 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7fcc0a2ed0 0x7f7fcc0a5390 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fe0114890 0x7f7fe010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fe0109270 0x7f7fe0114240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 --2- 172.21.15.38:0/2500243846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fe0108950 0x7f7fe0113d00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 >> 172.21.15.38:0/2500243846 conn(0x7f7fe006c850 msgr2=0x7f7fe00703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 shutdown_connections 2024-04-22T00:09:45.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.593+0000 7f7fe5597640 1 -- 172.21.15.38:0/2500243846 wait complete. 2024-04-22T00:09:45.604 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:09:45.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:09:45.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 -- 172.21.15.38:0/2070559815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1810fea0 msgr2=0x7fed1810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 --2- 172.21.15.38:0/2070559815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1810fea0 0x7fed1810da80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009900 tx=0x7fed0c02bc60 comp rx=0 tx=0).stop 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 -- 172.21.15.38:0/2070559815 shutdown_connections 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 --2- 172.21.15.38:0/2070559815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed1810e6a0 0x7fed1811cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 --2- 172.21.15.38:0/2070559815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed18110310 0x7fed1810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 --2- 172.21.15.38:0/2070559815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed1810fea0 0x7fed1810da80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.799+0000 7fed1d33e640 1 -- 172.21.15.38:0/2070559815 >> 172.21.15.38:0/2070559815 conn(0x7fed1806c860 msgr2=0x7fed1806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:45.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.800+0000 7fed1d33e640 1 -- 172.21.15.38:0/2070559815 shutdown_connections 2024-04-22T00:09:45.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.800+0000 7fed1d33e640 1 -- 172.21.15.38:0/2070559815 wait complete. 2024-04-22T00:09:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.804+0000 7fed1d33e640 1 Processor -- start 2024-04-22T00:09:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 -- start start 2024-04-22T00:09:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed1810fea0 0x7fed181a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 0x7fed181ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 0x7fed181ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fed18120920 con 0x7fed1810fea0 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fed181207a0 con 0x7fed181abce0 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.805+0000 7fed1d33e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed18120aa0 con 0x7fed181a7420 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.806+0000 7fed167fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 0x7fed181ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.806+0000 7fed177fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 0x7fed181ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.806+0000 7fed177fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 0x7fed181ac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46496/0 (socket says 172.21.15.38:46496) 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.806+0000 7fed177fe640 1 -- 172.21.15.38:0/1820907524 learned_addr learned my addr 172.21.15.38:0/1820907524 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 -- 172.21.15.38:0/1820907524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 msgr2=0x7fed181ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 0x7fed181ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 -- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed1810fea0 msgr2=0x7fed181a6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:45.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed1810fea0 0x7fed181a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:45.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 -- 172.21.15.38:0/1820907524 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18110700 con 0x7fed181a7420 2024-04-22T00:09:45.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.807+0000 7fed167fc640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 0x7fed181ab7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d700 tx=0x7fed0800dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.809+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08017070 con 0x7fed181a7420 2024-04-22T00:09:45.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.810+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed181109f0 con 0x7fed181a7420 2024-04-22T00:09:45.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.810+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed18110e80 con 0x7fed181a7420 2024-04-22T00:09:45.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.810+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed08009820 con 0x7fed181a7420 2024-04-22T00:09:45.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.810+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08004d00 con 0x7fed181a7420 2024-04-22T00:09:45.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.812+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed04005290 con 0x7fed181a7420 2024-04-22T00:09:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.817+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fed08005220 con 0x7fed181a7420 2024-04-22T00:09:45.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.818+0000 7fecfffff640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 0x7fed000a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.819+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fed080ceec0 con 0x7fed181a7420 2024-04-22T00:09:45.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.819+0000 7fed16ffd640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 0x7fed000a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.819+0000 7fed16ffd640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 0x7fed000a53c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0057c0 tx=0x7fed0c000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:45.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:45.824+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fed0809b6f0 con 0x7fed181a7420 2024-04-22T00:09:46.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.273+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed04005850 con 0x7fed181a7420 2024-04-22T00:09:46.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.274+0000 7fecfffff640 1 -- 172.21.15.38:0/1820907524 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11941 (secure 0 0 0) 0x7fed0809aff0 con 0x7fed181a7420 2024-04-22T00:09:46.270 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:09:46.270 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:09:42.375340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:09:46.271 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 38 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 msgr2=0x7fed000a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 0x7fed000a53c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0057c0 tx=0x7fed0c000bc0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 msgr2=0x7fed181ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 0x7fed181ab7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fed0800d700 tx=0x7fed0800dbc0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 shutdown_connections 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fed000a2f00 0x7fed000a53c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed181abce0 0x7fed181ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed181a7420 0x7fed181ab7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 --2- 172.21.15.38:0/1820907524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed1810fea0 0x7fed181a6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 >> 172.21.15.38:0/1820907524 conn(0x7fed1806c860 msgr2=0x7fed18072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 shutdown_connections 2024-04-22T00:09:46.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.278+0000 7fed1d33e640 1 -- 172.21.15.38:0/1820907524 wait complete. 2024-04-22T00:09:46.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 -- 172.21.15.38:0/373984955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1341066a0 msgr2=0x7ff134106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 --2- 172.21.15.38:0/373984955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1341066a0 0x7ff134106a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff124009930 tx=0x7ff12402bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 -- 172.21.15.38:0/373984955 shutdown_connections 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 --2- 172.21.15.38:0/373984955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1341014e0 0x7ff13410d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 --2- 172.21.15.38:0/373984955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff134100b20 0x7ff134100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 --2- 172.21.15.38:0/373984955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1341066a0 0x7ff134106a80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 -- 172.21.15.38:0/373984955 >> 172.21.15.38:0/373984955 conn(0x7ff1340761c0 msgr2=0x7ff1340765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.482+0000 7ff13c488640 1 -- 172.21.15.38:0/373984955 shutdown_connections 2024-04-22T00:09:46.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.483+0000 7ff13c488640 1 -- 172.21.15.38:0/373984955 wait complete. 2024-04-22T00:09:46.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.483+0000 7ff13c488640 1 Processor -- start 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.484+0000 7ff13c488640 1 -- start start 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.484+0000 7ff13c488640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff134100b20 0x7ff13410d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13c488640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1341014e0 0x7ff13410a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13c488640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13c488640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff13410f910 con 0x7ff1341014e0 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13c488640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff13410f790 con 0x7ff1341066a0 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13c488640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff13410fa90 con 0x7ff134100b20 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46528/0 (socket says 172.21.15.38:46528) 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 -- 172.21.15.38:0/1901034284 learned_addr learned my addr 172.21.15.38:0/1901034284 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 -- 172.21.15.38:0/1901034284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff134100b20 msgr2=0x7ff13410d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff134100b20 0x7ff13410d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 -- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1341014e0 msgr2=0x7ff13410a510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1341014e0 0x7ff13410a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.485+0000 7ff13aa25640 1 -- 172.21.15.38:0/1901034284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13406b160 con 0x7ff1341066a0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.486+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00eb10 tx=0x7ff12c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.487+0000 7ff12b7fe640 1 -- 172.21.15.38:0/1901034284 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c004b20 con 0x7ff1341066a0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.487+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13406b450 con 0x7ff1341066a0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.487+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13406b9a0 con 0x7ff1341066a0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.487+0000 7ff12b7fe640 1 -- 172.21.15.38:0/1901034284 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c004cc0 con 0x7ff1341066a0 2024-04-22T00:09:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.487+0000 7ff12b7fe640 1 -- 172.21.15.38:0/1901034284 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c010670 con 0x7ff1341066a0 2024-04-22T00:09:46.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.491+0000 7ff12b7fe640 1 -- 172.21.15.38:0/1901034284 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff12c005000 con 0x7ff1341066a0 2024-04-22T00:09:46.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.492+0000 7ff12b7fe640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 0x7ff1200a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.493+0000 7ff12b7fe640 1 -- 172.21.15.38:0/1901034284 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff12c0cfd90 con 0x7ff1341066a0 2024-04-22T00:09:46.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.493+0000 7ff13a224640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 0x7ff1200a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.494+0000 7ff13a224640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 0x7ff1200a52c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff1240057c0 tx=0x7ff124000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.496+0000 7ff13f07a740 1 Processor -- start 2024-04-22T00:09:46.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.496+0000 7ff13f07a740 1 -- start start 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fdcb50530 con 0x558fdcbec940 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558fdcaf8830 con 0x558fdcbf5520 2024-04-22T00:09:46.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.497+0000 7ff13f07a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558fdcb4c4f0 con 0x558fdcbec020 2024-04-22T00:09:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff139a23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff139a23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38612/0 (socket says 172.21.15.38:38612) 2024-04-22T00:09:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff139a23640 1 -- 172.21.15.38:0/2465779638 learned_addr learned my addr 172.21.15.38:0/2465779638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13a224640 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13aa25640 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13aa25640 1 -- 172.21.15.38:0/2465779638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 msgr2=0x558fdcbec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13aa25640 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13aa25640 1 -- 172.21.15.38:0/2465779638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 msgr2=0x558fdcbf4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.498+0000 7ff13aa25640 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.499+0000 7ff13aa25640 1 -- 172.21.15.38:0/2465779638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdcb1c2c0 con 0x558fdcbf5520 2024-04-22T00:09:46.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.499+0000 7ff13aa25640 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf7910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0cd0f0 tx=0x7ff12c09bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=899b4d313f66bbdb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.499+0000 7ff128bfe640 1 -- 172.21.15.38:0/2465779638 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c09bec0 con 0x558fdcbf5520 2024-04-22T00:09:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.500+0000 7ff128bfe640 1 -- 172.21.15.38:0/2465779638 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c0a0270 con 0x558fdcbf5520 2024-04-22T00:09:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.500+0000 7ff128bfe640 1 -- 172.21.15.38:0/2465779638 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0a46a0 con 0x558fdcbf5520 2024-04-22T00:09:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 -- 172.21.15.38:0/2465779638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 msgr2=0x558fdcbf7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf7910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0cd0f0 tx=0x7ff12c09bac0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 -- 172.21.15.38:0/2465779638 shutdown_connections 2024-04-22T00:09:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf7910 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 --2- 172.21.15.38:0/2465779638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.501+0000 7ff13f07a740 1 -- 172.21.15.38:0/2465779638 >> 172.21.15.38:0/2465779638 conn(0x558fdcbe6750 msgr2=0x558fdcbe6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.503+0000 7ff13f07a740 1 -- 172.21.15.38:0/2465779638 shutdown_connections 2024-04-22T00:09:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.505+0000 7ff13f07a740 1 -- 172.21.15.38:0/2465779638 wait complete. 2024-04-22T00:09:46.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.505+0000 7ff13f07a740 1 -- start start 2024-04-22T00:09:46.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.506+0000 7ff13f07a740 1 Processor -- start 2024-04-22T00:09:46.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.506+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbf4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.506+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.507+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.507+0000 7ff13f07a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fdcbf8310 con 0x558fdcbec940 2024-04-22T00:09:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.507+0000 7ff13f07a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558fdcb0bf10 con 0x558fdcbf5520 2024-04-22T00:09:46.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.507+0000 7ff13f07a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558fdcbf8490 con 0x558fdcbec020 2024-04-22T00:09:46.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.507+0000 7ff13f07a740 10 client.0 _ll_register_callbacks cb 0x558fdc863be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:46.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.508+0000 7ff139a23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.508+0000 7ff13a224640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbf4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.508+0000 7ff139a23640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38618/0 (socket says 172.21.15.38:38618) 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.508+0000 7ff139a23640 1 -- 172.21.15.38:0/3218782855 learned_addr learned my addr 172.21.15.38:0/3218782855 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 -- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 msgr2=0x558fdcbf4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbf4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 -- 172.21.15.38:0/3218782855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 msgr2=0x558fdcbf27b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 -- 172.21.15.38:0/3218782855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdcaddc80 con 0x558fdcbec940 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff139a23640 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7ff130004620 tx=0x7ff130011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12be29ad3a4c2988 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.509+0000 7ff11dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:46.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.515+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13001e060 con 0x558fdcbec940 2024-04-22T00:09:46.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.515+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff13000f750 con 0x558fdcbec940 2024-04-22T00:09:46.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.515+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12c0a0270 con 0x558fdcbec940 2024-04-22T00:09:46.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.521+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff13000f950 con 0x558fdcbec940 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.523+0000 7ff13f07a740 10 client.18126 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.523+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558fdc8643f0 con 0x558fdcbec940 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c0a0270 con 0x558fdcbec940 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff1300183b0 con 0x558fdcbec940 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 10 client.18126 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 10 client.18126 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 20 client.18126 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc05410 0x558fdcc07820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 4 client.18126 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:46.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.524+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fdcb5da70 con 0x558fdcc05410 2024-04-22T00:09:46.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.533+0000 7ff13a224640 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc05410 0x558fdcc07820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.537+0000 7ff13a224640 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc05410 0x558fdcc07820 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d176d535db77aaf5 server_cookie=e6cc04021d09ed74 in_seq=0 out_seq=0 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.539+0000 7ff11dffb640 10 client.18126 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.540+0000 7ff11dffb640 1 -- 172.21.15.38:0/3218782855 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fdcb5da70 con 0x558fdcc05410 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.540+0000 7ff11dffb640 10 client.18126 handle_command_reply: tid=0 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.540+0000 7ff13f07a740 1 client.18126 shutdown 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.541+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 msgr2=0x558fdcbf2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.541+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7ff130004620 tx=0x7ff130011920 comp rx=0 tx=0).stop 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 shutdown_connections 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc05410 0x558fdcc07820 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbf5520 0x558fdcbf27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec940 0x558fdcbf2230 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3218782855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbf4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 >> 172.21.15.38:0/3218782855 conn(0x558fdcbe6750 msgr2=0x558fdcbe84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff11dffb640 0 client.18126 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:46.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 shutdown_connections 2024-04-22T00:09:46.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.545+0000 7ff13f07a740 1 -- 172.21.15.38:0/3218782855 wait complete. 2024-04-22T00:09:46.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.546+0000 7ff13f07a740 20 client.18126 trim_cache size 0 max 16384 2024-04-22T00:09:46.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.576+0000 7ff13f07a740 1 Processor -- start 2024-04-22T00:09:46.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 -- start start 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 0x558fdcc01230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbed690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 0x558fdcbf4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff11800ffb0 con 0x558fdcbec020 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff118011bb0 con 0x558fdcbec940 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.577+0000 7ff13f07a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1180106b0 con 0x558fdcbf5520 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13a224640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 0x558fdcc01230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff139a23640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13a224640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 0x558fdcc01230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38620/0 (socket says 172.21.15.38:38620) 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13a224640 1 -- 172.21.15.38:0/1968094409 learned_addr learned my addr 172.21.15.38:0/1968094409 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 0x558fdcbf4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13aa25640 1 -- 172.21.15.38:0/1968094409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 msgr2=0x558fdcbed690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbed690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13aa25640 1 -- 172.21.15.38:0/1968094409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 msgr2=0x558fdcc01230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.578+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 0x558fdcc01230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.579+0000 7ff13aa25640 1 -- 172.21.15.38:0/1968094409 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fdcbf2b20 con 0x558fdcbf5520 2024-04-22T00:09:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.579+0000 7ff13aa25640 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 0x558fdcbf4960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x558fdcbe78e0 tx=0x7ff12c00efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec6f5d5b54fa939c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.580+0000 7ff11ffff640 1 -- 172.21.15.38:0/1968094409 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c09bec0 con 0x558fdcbf5520 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.581+0000 7ff11ffff640 1 -- 172.21.15.38:0/1968094409 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c0a0270 con 0x558fdcbf5520 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff11ffff640 1 -- 172.21.15.38:0/1968094409 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0cd570 con 0x558fdcbf5520 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 -- 172.21.15.38:0/1968094409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 msgr2=0x558fdcbf4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 0x558fdcbf4960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x558fdcbe78e0 tx=0x7ff12c00efd0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 -- 172.21.15.38:0/1968094409 shutdown_connections 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbf5520 0x558fdcbf4960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbed690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 --2- 172.21.15.38:0/1968094409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbec020 0x558fdcc01230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.582+0000 7ff13f07a740 1 -- 172.21.15.38:0/1968094409 >> 172.21.15.38:0/1968094409 conn(0x558fdcbe6750 msgr2=0x558fdcbecf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.590+0000 7ff13f07a740 1 -- 172.21.15.38:0/1968094409 shutdown_connections 2024-04-22T00:09:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.590+0000 7ff13f07a740 1 -- 172.21.15.38:0/1968094409 wait complete. 2024-04-22T00:09:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 -- start start 2024-04-22T00:09:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 Processor -- start 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbfdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbf5520 0x558fdcbef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1180114e0 con 0x558fdcbf5520 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff118010db0 con 0x558fdcbec940 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff11800dfb0 con 0x558fdcbec020 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.591+0000 7ff13f07a740 10 client.0 _ll_register_callbacks cb 0x7ff130009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.592+0000 7ff139a23640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.592+0000 7ff139a23640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46542/0 (socket says 172.21.15.38:46542) 2024-04-22T00:09:46.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.592+0000 7ff139a23640 1 -- 172.21.15.38:0/3197724637 learned_addr learned my addr 172.21.15.38:0/3197724637 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 -- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 msgr2=0x558fdcbfdc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:09:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbfdc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 -- 172.21.15.38:0/3197724637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbf5520 msgr2=0x558fdcbef1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbf5520 0x558fdcbef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 -- 172.21.15.38:0/3197724637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fdcbec770 con 0x558fdcbec940 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff139a23640 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x558fdcbf3800 tx=0x7ff13000fd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33cc2a4a72a91177 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff11dffb640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.593+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130028080 con 0x558fdcbec940 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.594+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12c0a0270 con 0x558fdcbec940 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.594+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff130005250 con 0x558fdcbec940 2024-04-22T00:09:46.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.594+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff13003a050 con 0x558fdcbec940 2024-04-22T00:09:46.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.597+0000 7ff13f07a740 10 client.27437 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:46.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.597+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558fdc92e7d0 con 0x558fdcbec940 2024-04-22T00:09:46.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.599+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c0a0270 con 0x558fdcbec940 2024-04-22T00:09:46.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.599+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff1300049d0 con 0x558fdcbec940 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 10 client.27437 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 10 client.27437 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 20 client.27437 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc6be70 0x558fdcc6e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 4 client.27437 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:46.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.600+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558fdcb57dc0 con 0x558fdcc6be70 2024-04-22T00:09:46.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.603+0000 7ff13a224640 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc6be70 0x558fdcc6e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.609+0000 7ff11dffb640 10 client.27437 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:46.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.609+0000 7ff13a224640 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc6be70 0x558fdcc6e290 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d07115a99ec965cb server_cookie=88e0ef6b2b8b4de0 in_seq=0 out_seq=0 2024-04-22T00:09:46.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.609+0000 7ff11dffb640 1 -- 172.21.15.38:0/3197724637 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558fdcb57dc0 con 0x558fdcc6be70 2024-04-22T00:09:46.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.609+0000 7ff11dffb640 10 client.27437 handle_command_reply: tid=0 2024-04-22T00:09:46.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.610+0000 7ff13f07a740 1 client.27437 shutdown 2024-04-22T00:09:46.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.611+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 msgr2=0x558fdcbeec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.611+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x558fdcbf3800 tx=0x7ff13000fd20 comp rx=0 tx=0).stop 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 shutdown_connections 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x558fdcc6be70 0x558fdcc6e290 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fdcbf5520 0x558fdcbef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558fdcbec940 0x558fdcbeec50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 --2- 172.21.15.38:0/3197724637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558fdcbec020 0x558fdcbfdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 >> 172.21.15.38:0/3197724637 conn(0x558fdcbe6750 msgr2=0x558fdcbe5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff11dffb640 0 client.27437 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:09:46.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.615+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 shutdown_connections 2024-04-22T00:09:46.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.616+0000 7ff13f07a740 1 -- 172.21.15.38:0/3197724637 wait complete. 2024-04-22T00:09:46.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.616+0000 7ff13f07a740 20 client.27437 trim_cache size 0 max 16384 2024-04-22T00:09:46.615 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.621+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 msgr2=0x7ff1200a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.621+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 0x7ff1200a52c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff1240057c0 tx=0x7ff124000bc0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.621+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 msgr2=0x7ff13410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.621+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00eb10 tx=0x7ff12c00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 shutdown_connections 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1200a2e00 0x7ff1200a52c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1341066a0 0x7ff13410ab00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1341014e0 0x7ff13410a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 --2- 172.21.15.38:0/1901034284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff134100b20 0x7ff13410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.626+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 >> 172.21.15.38:0/1901034284 conn(0x7ff1340761c0 msgr2=0x7ff1340ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.627+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 shutdown_connections 2024-04-22T00:09:46.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.627+0000 7ff13c488640 1 -- 172.21.15.38:0/1901034284 wait complete. 2024-04-22T00:09:46.642 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:46 smithi038 ceph-mon[27972]: pgmap v235: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 124 op/s 2024-04-22T00:09:46.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:46 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1820907524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:46.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:09:46.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:46 smithi203 ceph-mon[31431]: pgmap v235: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 124 op/s 2024-04-22T00:09:46.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:46 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1820907524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:46.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:46 smithi111 ceph-mon[31543]: pgmap v235: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 124 op/s 2024-04-22T00:09:46.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:46 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1820907524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 -- 172.21.15.38:0/3179908378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c114890 msgr2=0x7f242c116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 --2- 172.21.15.38:0/3179908378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c114890 0x7f242c116c80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2420009900 tx=0x7f242002bc80 comp rx=0 tx=0).stop 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 -- 172.21.15.38:0/3179908378 shutdown_connections 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 --2- 172.21.15.38:0/3179908378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c114890 0x7f242c116c80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 --2- 172.21.15.38:0/3179908378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c109270 0x7f242c114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 --2- 172.21.15.38:0/3179908378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c108950 0x7f242c108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.836+0000 7f2431888640 1 -- 172.21.15.38:0/3179908378 >> 172.21.15.38:0/3179908378 conn(0x7f242c06c850 msgr2=0x7f242c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.838+0000 7f2431888640 1 -- 172.21.15.38:0/3179908378 shutdown_connections 2024-04-22T00:09:46.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.839+0000 7f2431888640 1 -- 172.21.15.38:0/3179908378 wait complete. 2024-04-22T00:09:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.840+0000 7f2431888640 1 Processor -- start 2024-04-22T00:09:46.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 -- start start 2024-04-22T00:09:46.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c108950 0x7f242c1abb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c109270 0x7f242c1ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f242c11a470 con 0x7f242c114890 2024-04-22T00:09:46.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f242c11a2f0 con 0x7f242c109270 2024-04-22T00:09:46.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.841+0000 7f2431888640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f242c11a5f0 con 0x7f242c108950 2024-04-22T00:09:46.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.843+0000 7f242b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.843+0000 7f242b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38656/0 (socket says 172.21.15.38:38656) 2024-04-22T00:09:46.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.843+0000 7f242b7fe640 1 -- 172.21.15.38:0/2572077611 learned_addr learned my addr 172.21.15.38:0/2572077611 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.843+0000 7f242a7fc640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c109270 0x7f242c1ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.844+0000 7f242affd640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c108950 0x7f242c1abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 -- 172.21.15.38:0/2572077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c108950 msgr2=0x7f242c1abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c108950 0x7f242c1abb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 -- 172.21.15.38:0/2572077611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c109270 msgr2=0x7f242c1ac060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c109270 0x7f242c1ac060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 -- 172.21.15.38:0/2572077611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c113be0 con 0x7f242c114890 2024-04-22T00:09:46.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.845+0000 7f242b7fe640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f2420036040 tx=0x7f24200043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.846+0000 7f2430886640 1 -- 172.21.15.38:0/2572077611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420042020 con 0x7f242c114890 2024-04-22T00:09:46.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.846+0000 7f2430886640 1 -- 172.21.15.38:0/2572077611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242002cd10 con 0x7f242c114890 2024-04-22T00:09:46.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.846+0000 7f2430886640 1 -- 172.21.15.38:0/2572077611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420003e60 con 0x7f242c114890 2024-04-22T00:09:46.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.846+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242c113e70 con 0x7f242c114890 2024-04-22T00:09:46.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.846+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f242c1b3a40 con 0x7f242c114890 2024-04-22T00:09:46.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.850+0000 7f2434478740 1 Processor -- start 2024-04-22T00:09:46.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 -- start start 2024-04-22T00:09:46.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d41fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d42520 0x562630d44910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562630c9d530 con 0x562630d42520 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562630c45830 con 0x562630d39020 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2434478740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562630c994f0 con 0x562630d39940 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f2430886640 1 -- 172.21.15.38:0/2572077611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2420045020 con 0x7f242c114890 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f242a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f242a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46572/0 (socket says 172.21.15.38:46572) 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.851+0000 7f242a7fc640 1 -- 172.21.15.38:0/2965964045 learned_addr learned my addr 172.21.15.38:0/2965964045 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.852+0000 7f242a7fc640 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d42520 0x562630d44910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242affd640 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d41fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242a7fc640 1 -- 172.21.15.38:0/2965964045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 msgr2=0x562630d41fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242a7fc640 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d41fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242a7fc640 1 -- 172.21.15.38:0/2965964045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d42520 msgr2=0x562630d44910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242a7fc640 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d42520 0x562630d44910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.854+0000 7f242a7fc640 1 -- 172.21.15.38:0/2965964045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562630c692c0 con 0x562630d39020 2024-04-22T00:09:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.855+0000 7f242a7fc640 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f242400b9c0 tx=0x7f2424009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679536ea4c3d5022 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.855+0000 7f2430886640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 0x7f24140a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.855+0000 7f2430886640 1 -- 172.21.15.38:0/2572077611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f24200f1640 con 0x7f242c114890 2024-04-22T00:09:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.855+0000 7f2411ffb640 1 -- 172.21.15.38:0/2965964045 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424014040 con 0x562630d39020 2024-04-22T00:09:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.858+0000 7f2411ffb640 1 -- 172.21.15.38:0/2965964045 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24240193b0 con 0x562630d39020 2024-04-22T00:09:46.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.858+0000 7f2411ffb640 1 -- 172.21.15.38:0/2965964045 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24240185c0 con 0x562630d39020 2024-04-22T00:09:46.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.859+0000 7f242affd640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 0x7f24140a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.859+0000 7f2434478740 1 -- 172.21.15.38:0/2965964045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 msgr2=0x562630d39400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.859+0000 7f2434478740 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f242400b9c0 tx=0x7f2424009b60 comp rx=0 tx=0).stop 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 -- 172.21.15.38:0/2965964045 shutdown_connections 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d42520 0x562630d44910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d41fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 --2- 172.21.15.38:0/2965964045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d39400 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 -- 172.21.15.38:0/2965964045 >> 172.21.15.38:0/2965964045 conn(0x562630d33750 msgr2=0x562630d33b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f242affd640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 0x7f24140a52c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x562630d38af0 tx=0x7f241c00a2d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.863+0000 7f2434478740 1 -- 172.21.15.38:0/2965964045 shutdown_connections 2024-04-22T00:09:46.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.864+0000 7f2434478740 1 -- 172.21.15.38:0/2965964045 wait complete. 2024-04-22T00:09:46.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.864+0000 7f2434478740 1 -- start start 2024-04-22T00:09:46.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 Processor -- start 2024-04-22T00:09:46.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d3f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d42520 0x562630d3f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562630d45310 con 0x562630d39020 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x562630c58f10 con 0x562630d42520 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562630d45490 con 0x562630d39940 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.865+0000 7f2434478740 10 client.0 _ll_register_callbacks cb 0x5626309b0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.866+0000 7f242affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.866+0000 7f242affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38672/0 (socket says 172.21.15.38:38672) 2024-04-22T00:09:46.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.866+0000 7f242affd640 1 -- 172.21.15.38:0/3380768623 learned_addr learned my addr 172.21.15.38:0/3380768623 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.867+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d3f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242b7fe640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d42520 0x562630d3f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242affd640 1 -- 172.21.15.38:0/3380768623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 msgr2=0x562630d3f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242affd640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d3f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242affd640 1 -- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d42520 msgr2=0x562630d3f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242affd640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d42520 0x562630d3f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.869+0000 7f242affd640 1 -- 172.21.15.38:0/3380768623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562630c2ac80 con 0x562630d39020 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.870+0000 7f242affd640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f241c012a50 tx=0x7f241c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a495a28ca5719820 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.870+0000 7f240affd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.870+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241c00fd60 con 0x562630d39020 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.870+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24240193b0 con 0x562630d39020 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.870+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f241c01f420 con 0x562630d39020 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.871+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f241c030050 con 0x562630d39020 2024-04-22T00:09:46.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.872+0000 7f2434478740 10 client.18138 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:46.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.872+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5626309b13f0 con 0x562630d39020 2024-04-22T00:09:46.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24240193b0 con 0x562630d39020 2024-04-22T00:09:46.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f241c0277c0 con 0x562630d39020 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 10 client.18138 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 10 client.18138 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 20 client.18138 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d52650 0x562630d54a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 4 client.18138 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:46.874 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.877+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562630caaa70 con 0x562630d52650 2024-04-22T00:09:46.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.878+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d52650 0x562630d54a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.881+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d52650 0x562630d54a60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4fb680dc6eb36d9a server_cookie=ae0bedbf6a1e7793 in_seq=0 out_seq=0 2024-04-22T00:09:46.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.881+0000 7f240affd640 10 client.18138 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:46.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.882+0000 7f240affd640 1 -- 172.21.15.38:0/3380768623 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562630caaa70 con 0x562630d52650 2024-04-22T00:09:46.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.882+0000 7f240affd640 10 client.18138 handle_command_reply: tid=0 2024-04-22T00:09:46.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.882+0000 7f2434478740 1 client.18138 shutdown 2024-04-22T00:09:46.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.882+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 msgr2=0x562630d41e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.882+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f241c012a50 tx=0x7f241c0046c0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 shutdown_connections 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d42520 0x562630d3f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d3f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d52650 0x562630d54a60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 --2- 172.21.15.38:0/3380768623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39020 0x562630d41e20 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 >> 172.21.15.38:0/3380768623 conn(0x562630d33750 msgr2=0x562630d354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.886+0000 7f240affd640 0 client.18138 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:46.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.888+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 shutdown_connections 2024-04-22T00:09:46.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.889+0000 7f2434478740 1 -- 172.21.15.38:0/3380768623 wait complete. 2024-04-22T00:09:46.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.892+0000 7f2434478740 20 client.18138 trim_cache size 0 max 16384 2024-04-22T00:09:46.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.920+0000 7f2434478740 1 Processor -- start 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 -- start start 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d33750 0x562630d4e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d418e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f241800ffb0 con 0x562630d33750 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2418011bb0 con 0x562630d39020 2024-04-22T00:09:46.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f2434478740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24180106b0 con 0x562630d39940 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f242a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f242a7fc640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46584/0 (socket says 172.21.15.38:46584) 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f242a7fc640 1 -- 172.21.15.38:0/3465920205 learned_addr learned my addr 172.21.15.38:0/3465920205 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.921+0000 7f242b7fe640 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 -- 172.21.15.38:0/3465920205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 msgr2=0x562630d418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 -- 172.21.15.38:0/3465920205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d33750 msgr2=0x562630d4e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d33750 0x562630d4e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 -- 172.21.15.38:0/3465920205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562630a7b7d0 con 0x562630d39020 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f242a7fc640 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f242400b990 tx=0x7f2424009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cfd57cd852bf67a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f24117fa640 1 -- 172.21.15.38:0/3465920205 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242400b180 con 0x562630d39020 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f24117fa640 1 -- 172.21.15.38:0/3465920205 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24240087c0 con 0x562630d39020 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.922+0000 7f24117fa640 1 -- 172.21.15.38:0/3465920205 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24240294e0 con 0x562630d39020 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.923+0000 7f2434478740 1 -- 172.21.15.38:0/3465920205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 msgr2=0x562630d3f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.923+0000 7f2434478740 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f242400b990 tx=0x7f2424009770 comp rx=0 tx=0).stop 2024-04-22T00:09:46.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 -- 172.21.15.38:0/3465920205 shutdown_connections 2024-04-22T00:09:46.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39940 0x562630d418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d39020 0x562630d3f170 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 --2- 172.21.15.38:0/3465920205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d33750 0x562630d4e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 -- 172.21.15.38:0/3465920205 >> 172.21.15.38:0/3465920205 conn(0x562630d3f850 msgr2=0x562630d32850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 -- 172.21.15.38:0/3465920205 shutdown_connections 2024-04-22T00:09:46.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.929+0000 7f2434478740 1 -- 172.21.15.38:0/3465920205 wait complete. 2024-04-22T00:09:46.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.930+0000 7f2434478740 1 -- start start 2024-04-22T00:09:46.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.931+0000 7f2434478740 1 Processor -- start 2024-04-22T00:09:46.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.931+0000 7f2434478740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d33750 0x562630d3bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.931+0000 7f2434478740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 0x562630d3c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.931+0000 7f2434478740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f2434478740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f24180114e0 con 0x562630d39940 2024-04-22T00:09:46.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f2434478740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2418010db0 con 0x562630d33750 2024-04-22T00:09:46.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f2434478740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f241800dfb0 con 0x562630d39020 2024-04-22T00:09:46.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f2434478740 10 client.0 _ll_register_callbacks cb 0x7f241c00a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:46.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f242b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f242b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38694/0 (socket says 172.21.15.38:38694) 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f242b7fe640 1 -- 172.21.15.38:0/1862342446 learned_addr learned my addr 172.21.15.38:0/1862342446 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f242b7fe640 1 -- 172.21.15.38:0/1862342446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 msgr2=0x562630d3c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.932+0000 7f242a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 0x562630d3c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.933+0000 7f242b7fe640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 0x562630d3c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.933+0000 7f242b7fe640 1 -- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d33750 msgr2=0x562630d3bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.933+0000 7f242b7fe640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d33750 0x562630d3bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.933+0000 7f242b7fe640 1 -- 172.21.15.38:0/1862342446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562630db7ce0 con 0x562630d39940 2024-04-22T00:09:46.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.933+0000 7f242a7fc640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 0x562630d3c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:46.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.934+0000 7f242b7fe640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2420003ab0 tx=0x7f24200bd1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348135fccbfacb4a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:46.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.934+0000 7f240affd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:46.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.935+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24200d5020 con 0x562630d39940 2024-04-22T00:09:46.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.935+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f24200f11e0 con 0x562630d39940 2024-04-22T00:09:46.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.935+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24240087c0 con 0x562630d39940 2024-04-22T00:09:46.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.936+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2420003800 con 0x562630d39940 2024-04-22T00:09:46.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.936+0000 7f2434478740 10 client.18141 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:46.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.937+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x562630db9020 con 0x562630d39940 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.940+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24240087c0 con 0x562630d39940 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.940+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f24200bdae0 con 0x562630d39940 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 10 client.18141 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 10 client.18141 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 20 client.18141 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:46.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d42520 0x562630dbb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 4 client.18141 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.943+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562630ca4dc0 con 0x562630d42520 2024-04-22T00:09:46.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.944+0000 7f242affd640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d42520 0x562630dbb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:46.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.954+0000 7f242affd640 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d42520 0x562630dbb7f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=adb7c89342c5f2df server_cookie=961f171b4dd225fb in_seq=0 out_seq=0 2024-04-22T00:09:46.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.955+0000 7f240affd640 10 client.18141 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:46.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.958+0000 7f240affd640 1 -- 172.21.15.38:0/1862342446 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562630ca4dc0 con 0x562630d42520 2024-04-22T00:09:46.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.958+0000 7f240affd640 10 client.18141 handle_command_reply: tid=0 2024-04-22T00:09:46.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.958+0000 7f2434478740 1 client.18141 shutdown 2024-04-22T00:09:46.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.965+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 msgr2=0x562630d3c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.965+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2420003ab0 tx=0x7f24200bd1e0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.965+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 shutdown_connections 2024-04-22T00:09:46.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.965+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x562630d42520 0x562630dbb7f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.966+0000 7f240affd640 0 client.18141 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:09:46.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.966+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562630d39940 0x562630d3c9f0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.966+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562630d39020 0x562630d3c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.966+0000 7f2434478740 1 --2- 172.21.15.38:0/1862342446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562630d33750 0x562630d3bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.966+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 >> 172.21.15.38:0/1862342446 conn(0x562630d3f850 msgr2=0x562630d349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.971+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 shutdown_connections 2024-04-22T00:09:46.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.973+0000 7f2434478740 1 -- 172.21.15.38:0/1862342446 wait complete. 2024-04-22T00:09:46.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.973+0000 7f2434478740 20 client.18141 trim_cache size 0 max 16384 2024-04-22T00:09:46.970 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.975+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 msgr2=0x7f24140a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.975+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 0x7f24140a52c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x562630d38af0 tx=0x7f241c00a2d0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.975+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 msgr2=0x7f242c113460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.975+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f2420036040 tx=0x7f24200043d0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 shutdown_connections 2024-04-22T00:09:46.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f24140a2e00 0x7f24140a52c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f242c114890 0x7f242c113460 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c109270 0x7f242c1ac060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 --2- 172.21.15.38:0/2572077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f242c108950 0x7f242c1abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:46.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.976+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 >> 172.21.15.38:0/2572077611 conn(0x7f242c06c850 msgr2=0x7f242c10a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:46.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.978+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 shutdown_connections 2024-04-22T00:09:46.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:46.978+0000 7f2431888640 1 -- 172.21.15.38:0/2572077611 wait complete. 2024-04-22T00:09:46.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.207+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1293880162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 msgr2=0x7ff1f010e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.207+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1293880162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f010e440 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009930 tx=0x7ff1d802bc60 comp rx=0 tx=0).stop 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.208+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1293880162 shutdown_connections 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.208+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1293880162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1f011af20 0x7ff1f011d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.208+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1293880162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f010e440 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.208+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1293880162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f010ff90 0x7ff1f010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.208+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1293880162 >> 172.21.15.38:0/1293880162 conn(0x7ff1f006c850 msgr2=0x7ff1f006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.212+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1293880162 shutdown_connections 2024-04-22T00:09:47.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.214+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1293880162 wait complete. 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 Processor -- start 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 -- start start 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1f010ff90 0x7ff1f01136a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0120970 con 0x7ff1f010dfc0 2024-04-22T00:09:47.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01207f0 con 0x7ff1f010ff90 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.215+0000 7ff1f5fb1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0120af0 con 0x7ff1f011af20 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:58472/0 (socket says 172.21.15.38:58472) 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 -- 172.21.15.38:0/1292318126 learned_addr learned my addr 172.21.15.38:0/1292318126 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 -- 172.21.15.38:0/1292318126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1f010ff90 msgr2=0x7ff1f01136a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1f010ff90 0x7ff1f01136a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 -- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 msgr2=0x7ff1f011a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f011a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.216+0000 7ff1effff640 1 -- 172.21.15.38:0/1292318126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f0114220 con 0x7ff1f011af20 2024-04-22T00:09:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.217+0000 7ff1effff640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b3f0 tx=0x7ff1e000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.218+0000 7ff1ecff9640 1 -- 172.21.15.38:0/1292318126 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0015020 con 0x7ff1f011af20 2024-04-22T00:09:47.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.218+0000 7ff1ecff9640 1 -- 172.21.15.38:0/1292318126 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e0011d10 con 0x7ff1f011af20 2024-04-22T00:09:47.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.219+0000 7ff1ecff9640 1 -- 172.21.15.38:0/1292318126 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000f2e0 con 0x7ff1f011af20 2024-04-22T00:09:47.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.219+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f0118030 con 0x7ff1f011af20 2024-04-22T00:09:47.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.219+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f0118500 con 0x7ff1f011af20 2024-04-22T00:09:47.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.223+0000 7ff1f8b96740 1 Processor -- start 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 -- start start 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52ee5940 0x559b52eedfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52eee520 0x559b52ef0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559b52e49530 con 0x559b52eee520 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559b52df1830 con 0x559b52ee5020 2024-04-22T00:09:47.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b52e454f0 con 0x559b52ee5940 2024-04-22T00:09:47.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.228+0000 7ff1ecff9640 1 -- 172.21.15.38:0/1292318126 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff1e0018020 con 0x7ff1f011af20 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.229+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.229+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46616/0 (socket says 172.21.15.38:46616) 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.229+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2089822074 learned_addr learned my addr 172.21.15.38:0/2089822074 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.229+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52eee520 0x559b52ef0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.229+0000 7ff1ecff9640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 0x7ff1dc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.230+0000 7ff1ecff9640 1 -- 172.21.15.38:0/1292318126 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff1e00cedc0 con 0x7ff1f011af20 2024-04-22T00:09:47.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.230+0000 7ff1effff640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 0x7ff1dc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52ee5940 0x559b52eedfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1effff640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 0x7ff1dc0a5160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1e001f000 tx=0x7ff1e0010870 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2089822074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52ee5940 msgr2=0x559b52eedfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52ee5940 0x559b52eedfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2089822074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52eee520 msgr2=0x559b52ef0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52eee520 0x559b52ef0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.231+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2089822074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b52e152c0 con 0x559b52ee5020 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.232+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009850 tx=0x7ff1d802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc57d7dd6508e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.232+0000 7ff1e5ffb640 1 -- 172.21.15.38:0/2089822074 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8048070 con 0x559b52ee5020 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.232+0000 7ff1e5ffb640 1 -- 172.21.15.38:0/2089822074 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d803a040 con 0x559b52ee5020 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2089822074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 msgr2=0x559b52ee5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009850 tx=0x7ff1d802e040 comp rx=0 tx=0).stop 2024-04-22T00:09:47.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2089822074 shutdown_connections 2024-04-22T00:09:47.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52eee520 0x559b52ef0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52ee5940 0x559b52eedfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2089822074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5020 0x559b52ee5400 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.233+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2089822074 >> 172.21.15.38:0/2089822074 conn(0x559b52edf750 msgr2=0x559b52edfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.234+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2089822074 shutdown_connections 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.234+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2089822074 wait complete. 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.234+0000 7ff1f8b96740 1 -- start start 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 Processor -- start 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5940 0x559b52eeb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52eee520 0x559b52eeb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559b52ef1350 con 0x559b52ee5020 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559b52e04f10 con 0x559b52ee5940 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b52ef14d0 con 0x559b52eee520 2024-04-22T00:09:47.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.235+0000 7ff1f8b96740 10 client.0 _ll_register_callbacks cb 0x559b52b5cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52eee520 0x559b52eeb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38730/0 (socket says 172.21.15.38:38730) 2024-04-22T00:09:47.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2109721013 learned_addr learned my addr 172.21.15.38:0/2109721013 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5940 0x559b52eeb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52eee520 msgr2=0x559b52eeb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52eee520 0x559b52eeb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2109721013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5940 msgr2=0x559b52eeb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5940 0x559b52eeb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.238+0000 7ff1eeffd640 1 -- 172.21.15.38:0/2109721013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b52dd6c80 con 0x559b52ee5020 2024-04-22T00:09:47.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.243+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009900 tx=0x7ff1d8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f67503d9209d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.243+0000 7ff1ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:47.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.245+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8048040 con 0x559b52ee5020 2024-04-22T00:09:47.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.245+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d803a040 con 0x559b52ee5020 2024-04-22T00:09:47.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.245+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff1d80055f0 con 0x559b52ee5020 2024-04-22T00:09:47.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.246+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff1d805d050 con 0x559b52ee5020 2024-04-22T00:09:47.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.246+0000 7ff1f8b96740 10 client.18153 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.246+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b52b5d3f0 con 0x559b52ee5020 2024-04-22T00:09:47.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d803a040 con 0x559b52ee5020 2024-04-22T00:09:47.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff1d804e040 con 0x559b52ee5020 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 10 client.18153 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 10 client.18153 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 20 client.18153 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f20760 0x559b52f22ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 4 client.18153 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.247+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b52e56a70 con 0x559b52f20760 2024-04-22T00:09:47.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.249+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f20760 0x559b52f22ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.250+0000 7ff1ceffd640 10 client.18153 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:47.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.250+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f20760 0x559b52f22ba0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b79af75f5f6429a1 server_cookie=58672029d3462b6b in_seq=0 out_seq=0 2024-04-22T00:09:47.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.252+0000 7ff1ceffd640 1 -- 172.21.15.38:0/2109721013 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b52e56a70 con 0x559b52f20760 2024-04-22T00:09:47.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.253+0000 7ff1ceffd640 10 client.18153 handle_command_reply: tid=0 2024-04-22T00:09:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.254+0000 7ff1f8b96740 1 client.18153 shutdown 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 msgr2=0x559b52eeddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009900 tx=0x7ff1d8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 shutdown_connections 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f20760 0x559b52f22ba0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52eee520 0x559b52eeb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52ee5940 0x559b52eeb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.254 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/2109721013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5020 0x559b52eeddb0 secure :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009900 tx=0x7ff1d8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 >> 172.21.15.38:0/2109721013 conn(0x559b52edf750 msgr2=0x559b52ee14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.257+0000 7ff1ceffd640 0 client.18153 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:47.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.258+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 shutdown_connections 2024-04-22T00:09:47.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.258+0000 7ff1f8b96740 1 -- 172.21.15.38:0/2109721013 wait complete. 2024-04-22T00:09:47.255 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.258+0000 7ff1f8b96740 20 client.18153 trim_cache size 0 max 16384 2024-04-22T00:09:47.287 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 Processor -- start 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 -- start start 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52efa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52eed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012c6e0 con 0x559b52ee5940 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012a3e0 con 0x559b52eee520 2024-04-22T00:09:47.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.291+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012bfe0 con 0x559b52edf750 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.292+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.292+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46648/0 (socket says 172.21.15.38:46648) 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.292+0000 7ff1eeffd640 1 -- 172.21.15.38:0/458366780 learned_addr learned my addr 172.21.15.38:0/458366780 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.292+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52efa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.293+0000 7ff1eeffd640 1 -- 172.21.15.38:0/458366780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 msgr2=0x559b52efa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.293+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52efa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.293+0000 7ff1eeffd640 1 -- 172.21.15.38:0/458366780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 msgr2=0x559b52eed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.293+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52eed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.293+0000 7ff1eeffd640 1 -- 172.21.15.38:0/458366780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b52c277d0 con 0x559b52eee520 2024-04-22T00:09:47.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.297+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009850 tx=0x7ff1d8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e84053220d262e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.302+0000 7ff1e57fa640 1 -- 172.21.15.38:0/458366780 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8048070 con 0x559b52eee520 2024-04-22T00:09:47.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.302+0000 7ff1e57fa640 1 -- 172.21.15.38:0/458366780 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d80046c0 con 0x559b52eee520 2024-04-22T00:09:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.303+0000 7ff1f8b96740 1 -- 172.21.15.38:0/458366780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 msgr2=0x559b52eeddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.303+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009850 tx=0x7ff1d8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.304+0000 7ff1f8b96740 1 -- 172.21.15.38:0/458366780 shutdown_connections 2024-04-22T00:09:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.304+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52eee520 0x559b52eeddb0 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009850 tx=0x7ff1d8005fd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.304+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52eed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.304+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/458366780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52efa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.304+0000 7ff1f8b96740 1 -- 172.21.15.38:0/458366780 >> 172.21.15.38:0/458366780 conn(0x559b52eeb7e0 msgr2=0x559b52eddfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.309+0000 7ff1f8b96740 1 -- 172.21.15.38:0/458366780 shutdown_connections 2024-04-22T00:09:47.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.313+0000 7ff1f8b96740 1 -- 172.21.15.38:0/458366780 wait complete. 2024-04-22T00:09:47.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.313+0000 7ff1f8b96740 1 -- start start 2024-04-22T00:09:47.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.313+0000 7ff1f8b96740 1 Processor -- start 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52ef1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52f7ed10 0x559b52f7f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012b1e0 con 0x559b52ee5940 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012aae0 con 0x559b52f7ed10 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f012b910 con 0x559b52edf750 2024-04-22T00:09:47.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.314+0000 7ff1f8b96740 10 client.0 _ll_register_callbacks cb 0x7ff1d8046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.315+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.315+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52f7ed10 0x559b52f7f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.315+0000 7ff1eeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:58500/0 (socket says 172.21.15.38:58500) 2024-04-22T00:09:47.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.315+0000 7ff1eeffd640 1 -- 172.21.15.38:0/421256103 learned_addr learned my addr 172.21.15.38:0/421256103 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52ef1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1eeffd640 1 -- 172.21.15.38:0/421256103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52f7ed10 msgr2=0x559b52f7f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52f7ed10 0x559b52f7f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1eeffd640 1 -- 172.21.15.38:0/421256103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 msgr2=0x559b52ef1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52ef1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1eeffd640 1 -- 172.21.15.38:0/421256103 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b52f7f6d0 con 0x559b52edf750 2024-04-22T00:09:47.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.319+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52ef1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:47.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.323+0000 7ff1eeffd640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8050480 tx=0x7ff1d8050560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65877335291d1cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.323+0000 7ff1ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:09:47.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.323+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8048070 con 0x559b52edf750 2024-04-22T00:09:47.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.323+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7ff1d8034440 con 0x559b52edf750 2024-04-22T00:09:47.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.328+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b52e50dc0 con 0x559b52edf750 2024-04-22T00:09:47.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.333+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff1d802c2e0 con 0x559b52edf750 2024-04-22T00:09:47.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.334+0000 7ff1f8b96740 10 client.27307 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.334+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b52f86a50 con 0x559b52edf750 2024-04-22T00:09:47.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.338+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559b52e50dc0 con 0x559b52edf750 2024-04-22T00:09:47.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff1d804e070 con 0x559b52edf750 2024-04-22T00:09:47.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 10 client.27307 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 10 client.27307 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 20 client.27307 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f870b0 0x559b52f894a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 4 client.27307 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.339+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b52dce6e0 con 0x559b52f870b0 2024-04-22T00:09:47.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.340+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f870b0 0x559b52f894a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.341+0000 7ff1ef7fe640 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f870b0 0x559b52f894a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=736ebe5d096e5c88 server_cookie=3b48083c7f0a1718 in_seq=0 out_seq=0 2024-04-22T00:09:47.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.342+0000 7ff1ceffd640 10 client.27307 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:47.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1ceffd640 1 -- 172.21.15.38:0/421256103 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b52dce6e0 con 0x559b52f870b0 2024-04-22T00:09:47.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1ceffd640 10 client.27307 handle_command_reply: tid=0 2024-04-22T00:09:47.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 client.27307 shutdown 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 msgr2=0x559b52ef19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8050480 tx=0x7ff1d8050560 comp rx=0 tx=0).stop 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 shutdown_connections 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x559b52f870b0 0x559b52f894a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b52f7ed10 0x559b52f7f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b52ee5940 0x559b52ef1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 --2- 172.21.15.38:0/421256103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b52edf750 0x559b52ef19c0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8050480 tx=0x7ff1d8050560 comp rx=0 tx=0).stop 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 >> 172.21.15.38:0/421256103 conn(0x559b52eeb7e0 msgr2=0x559b52ede850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.343+0000 7ff1ceffd640 0 client.27307 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.344+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 shutdown_connections 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.344+0000 7ff1f8b96740 1 -- 172.21.15.38:0/421256103 wait complete. 2024-04-22T00:09:47.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.344+0000 7ff1f8b96740 20 client.27307 trim_cache size 0 max 16384 2024-04-22T00:09:47.343 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:47.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.352+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 msgr2=0x7ff1dc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.352+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 0x7ff1dc0a5160 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1e001f000 tx=0x7ff1e0010870 comp rx=0 tx=0).stop 2024-04-22T00:09:47.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.352+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 msgr2=0x7ff1f0113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.352+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000b3f0 tx=0x7ff1e000b8b0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 shutdown_connections 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff1dc0a2ca0 0x7ff1dc0a5160 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1f011af20 0x7ff1f0113be0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1f010ff90 0x7ff1f01136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 --2- 172.21.15.38:0/1292318126 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1f010dfc0 0x7ff1f011a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.354+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 >> 172.21.15.38:0/1292318126 conn(0x7ff1f006c850 msgr2=0x7ff1f0072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.355+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 shutdown_connections 2024-04-22T00:09:47.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.355+0000 7ff1f5fb1640 1 -- 172.21.15.38:0/1292318126 wait complete. 2024-04-22T00:09:47.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:09:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- 172.21.15.38:0/2873777716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 msgr2=0x7fb000110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 --2- 172.21.15.38:0/2873777716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb000110850 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7faff8009cd0 tx=0x7faff802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- 172.21.15.38:0/2873777716 shutdown_connections 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 --2- 172.21.15.38:0/2873777716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb000110850 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 --2- 172.21.15.38:0/2873777716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb000103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 --2- 172.21.15.38:0/2873777716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 0x7fb000109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- 172.21.15.38:0/2873777716 >> 172.21.15.38:0/2873777716 conn(0x7fb0000ff4d0 msgr2=0x7fb0001018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- 172.21.15.38:0/2873777716 shutdown_connections 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- 172.21.15.38:0/2873777716 wait complete. 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 Processor -- start 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.505+0000 7fb0073d2640 1 -- start start 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb00019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 0x7fb00019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb000113d70 con 0x7fb000104340 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb000113bf0 con 0x7fb000109990 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb000113ef0 con 0x7fb000103980 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38766/0 (socket says 172.21.15.38:38766) 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 -- 172.21.15.38:0/3706233455 learned_addr learned my addr 172.21.15.38:0/3706233455 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00596f640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 0x7fb00019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00516e640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb00019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 -- 172.21.15.38:0/3706233455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 msgr2=0x7fb00019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb00019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 -- 172.21.15.38:0/3706233455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 msgr2=0x7fb00019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 0x7fb00019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 -- 172.21.15.38:0/3706233455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00019f280 con 0x7fb000104340 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00516e640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb00019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:47.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb00496d640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7faff000e9f0 tx=0x7faff000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7faff67fc640 1 -- 172.21.15.38:0/3706233455 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff000cd80 con 0x7fb000104340 2024-04-22T00:09:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7faff67fc640 1 -- 172.21.15.38:0/3706233455 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff0010910 con 0x7fb000104340 2024-04-22T00:09:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00019f570 con 0x7fb000104340 2024-04-22T00:09:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7faff67fc640 1 -- 172.21.15.38:0/3706233455 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0018990 con 0x7fb000104340 2024-04-22T00:09:47.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.506+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0001a6e00 con 0x7fb000104340 2024-04-22T00:09:47.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.508+0000 7faff67fc640 1 -- 172.21.15.38:0/3706233455 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7faff0010ac0 con 0x7fb000104340 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 Processor -- start 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 -- start start 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7faff67fc640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 0x7fafec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b509910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e50b462530 con 0x55e50b4fe940 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e50b40a830 con 0x55e50b4fe020 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb009fb6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e50b45e4f0 con 0x55e50b507520 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7faff67fc640 1 -- 172.21.15.38:0/3706233455 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7faff00cfb00 con 0x7fb000104340 2024-04-22T00:09:47.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb00596f640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 0x7fafec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb00516e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38778/0 (socket says 172.21.15.38:38778) 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 -- 172.21.15.38:0/1689538566 learned_addr learned my addr 172.21.15.38:0/1689538566 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.509+0000 7fb00516e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46670/0 (socket says 172.21.15.38:46670) 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00596f640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b509910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 -- 172.21.15.38:0/1689538566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 msgr2=0x55e50b509910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b509910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 -- 172.21.15.38:0/1689538566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 msgr2=0x55e50b4fe400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 -- 172.21.15.38:0/1689538566 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e50b42e2c0 con 0x55e50b4fe940 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00596f640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b509910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:47.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00516e640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00496d640 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7faff00ccf20 tx=0x7faff0073cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaebdd401a69779b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fb00596f640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 0x7fafec0a5240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faff8005cc0 tx=0x7faff8005f10 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fafeb7fe640 1 -- 172.21.15.38:0/1689538566 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff009bc90 con 0x55e50b4fe940 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fafeb7fe640 1 -- 172.21.15.38:0/1689538566 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff00a0040 con 0x55e50b4fe940 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.510+0000 7fafeb7fe640 1 -- 172.21.15.38:0/1689538566 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff00764c0 con 0x55e50b4fe940 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- 172.21.15.38:0/1689538566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 msgr2=0x55e50b506fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7faff00ccf20 tx=0x7faff0073cd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- 172.21.15.38:0/1689538566 shutdown_connections 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b509910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b506fe0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1689538566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b4fe400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- 172.21.15.38:0/1689538566 >> 172.21.15.38:0/1689538566 conn(0x55e50b4f8750 msgr2=0x55e50b4f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- 172.21.15.38:0/1689538566 shutdown_connections 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- 172.21.15.38:0/1689538566 wait complete. 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 -- start start 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.511+0000 7fb009fb6740 1 Processor -- start 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b506e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b504720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e50b50a310 con 0x55e50b4fe940 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e50b41df10 con 0x55e50b4fe020 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e50b50a490 con 0x55e50b507520 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb009fb6740 10 client.0 _ll_register_callbacks cb 0x55e50b175be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38780/0 (socket says 172.21.15.38:38780) 2024-04-22T00:09:47.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 -- 172.21.15.38:0/3251790189 learned_addr learned my addr 172.21.15.38:0/3251790189 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00596f640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b504720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00516e640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b506e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 -- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 msgr2=0x55e50b504720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b504720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 -- 172.21.15.38:0/3251790189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 msgr2=0x55e50b506e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b506e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 -- 172.21.15.38:0/3251790189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e50b3efc80 con 0x55e50b4fe940 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fb00496d640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7faff009b990 tx=0x7faff0018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8735170f402a7fc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fafe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.512+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff009bc90 con 0x55e50b4fe940 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7faff0086040 con 0x55e50b4fe940 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faff00a0040 con 0x55e50b4fe940 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7faff00a4900 con 0x55e50b4fe940 2024-04-22T00:09:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 10 client.18165 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e50b1763f0 con 0x55e50b4fe940 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff00a0040 con 0x55e50b4fe940 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7faff009b0e0 con 0x55e50b4fe940 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 10 client.18165 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 10 client.18165 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 20 client.18165 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b5197d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 4 client.18165 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e50b46fa70 con 0x55e50b5173e0 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.513+0000 7fb00516e640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b5197d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fafe8ff9640 10 client.18165 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb00516e640 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b5197d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=345a905f7a0f531b server_cookie=802b76614a262a0d in_seq=0 out_seq=0 2024-04-22T00:09:47.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3251790189 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e50b46fa70 con 0x55e50b5173e0 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fafe8ff9640 10 client.18165 handle_command_reply: tid=0 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb009fb6740 1 client.18165 shutdown 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 msgr2=0x55e50b5041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7faff009b990 tx=0x7faff0018420 comp rx=0 tx=0).stop 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 shutdown_connections 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.514+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b5197d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fafe8ff9640 0 client.18165 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b507520 0x55e50b504720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b5041e0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3251790189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b4fe020 0x55e50b506e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 >> 172.21.15.38:0/3251790189 conn(0x55e50b4f8750 msgr2=0x55e50b4fa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 shutdown_connections 2024-04-22T00:09:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 1 -- 172.21.15.38:0/3251790189 wait complete. 2024-04-22T00:09:47.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.515+0000 7fb009fb6740 20 client.18165 trim_cache size 0 max 16384 2024-04-22T00:09:47.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.534+0000 7fb009fb6740 1 Processor -- start 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.535+0000 7fb009fb6740 1 -- start start 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 0x55e50b5131d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 0x55e50b4f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b506910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001b0da0 con 0x55e50b4fe020 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001b29a0 con 0x55e50b507520 2024-04-22T00:09:47.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.536+0000 7fb009fb6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001b14a0 con 0x55e50b4fe940 2024-04-22T00:09:47.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.537+0000 7fb00516e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 0x55e50b5131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.537+0000 7fb00516e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 0x55e50b5131d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38788/0 (socket says 172.21.15.38:38788) 2024-04-22T00:09:47.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.537+0000 7fb00516e640 1 -- 172.21.15.38:0/1601737331 learned_addr learned my addr 172.21.15.38:0/1601737331 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.537+0000 7fb00496d640 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 0x55e50b4f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.537+0000 7fb00596f640 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b506910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 -- 172.21.15.38:0/1601737331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 msgr2=0x55e50b506910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b506910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 -- 172.21.15.38:0/1601737331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 msgr2=0x55e50b5131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 0x55e50b5131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 -- 172.21.15.38:0/1601737331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e50b2407d0 con 0x55e50b4fe940 2024-04-22T00:09:47.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fb00496d640 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 0x55e50b4f6fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faff009bc20 tx=0x7faff0098060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718be5ca6908c8c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.539+0000 7fafeaffd640 1 -- 172.21.15.38:0/1601737331 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff00cd290 con 0x55e50b4fe940 2024-04-22T00:09:47.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.540+0000 7fafeaffd640 1 -- 172.21.15.38:0/1601737331 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff009c390 con 0x55e50b4fe940 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 -- 172.21.15.38:0/1601737331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 msgr2=0x55e50b4f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 0x55e50b4f6fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faff009bc20 tx=0x7faff0098060 comp rx=0 tx=0).stop 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 -- 172.21.15.38:0/1601737331 shutdown_connections 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b506910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe940 0x55e50b4f6fc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 --2- 172.21.15.38:0/1601737331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe020 0x55e50b5131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.541+0000 7fb009fb6740 1 -- 172.21.15.38:0/1601737331 >> 172.21.15.38:0/1601737331 conn(0x55e50b4f8750 msgr2=0x55e50b503210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.548+0000 7fb009fb6740 1 -- 172.21.15.38:0/1601737331 shutdown_connections 2024-04-22T00:09:47.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.549+0000 7fb009fb6740 1 -- 172.21.15.38:0/1601737331 wait complete. 2024-04-22T00:09:47.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.549+0000 7fb009fb6740 1 -- start start 2024-04-22T00:09:47.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.549+0000 7fb009fb6740 1 Processor -- start 2024-04-22T00:09:47.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe020 0x55e50b50f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b50f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001b22d0 con 0x55e50b4fe940 2024-04-22T00:09:47.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001b1ba0 con 0x55e50b507520 2024-04-22T00:09:47.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001aeda0 con 0x55e50b4fe020 2024-04-22T00:09:47.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.552+0000 7fb009fb6740 10 client.0 _ll_register_callbacks cb 0x7faff000c7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.549 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46682/0 (socket says 172.21.15.38:46682) 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 -- 172.21.15.38:0/3710825447 learned_addr learned my addr 172.21.15.38:0/3710825447 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00516e640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe020 0x55e50b50f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 -- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe020 msgr2=0x55e50b50f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00496d640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b50f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe020 0x55e50b50f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 -- 172.21.15.38:0/3710825447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 msgr2=0x55e50b50f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b50f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.550 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00596f640 1 -- 172.21.15.38:0/3710825447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e50b50b220 con 0x55e50b507520 2024-04-22T00:09:47.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.553+0000 7fb00496d640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b50f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:47.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.558+0000 7fb00596f640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faff8009cd0 tx=0x7faff8005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc35ce52b0258f5d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.558+0000 7fafe8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:47.556 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.559+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8055020 con 0x55e50b507520 2024-04-22T00:09:47.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.559+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7faff8004240 con 0x55e50b507520 2024-04-22T00:09:47.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.560+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faff009c390 con 0x55e50b507520 2024-04-22T00:09:47.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.560+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7faff8039040 con 0x55e50b507520 2024-04-22T00:09:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.564+0000 7fb009fb6740 10 client.27476 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.564+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e50b57d170 con 0x55e50b507520 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.565+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff009c390 con 0x55e50b507520 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.565+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7faff803c070 con 0x55e50b507520 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 10 client.27476 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 10 client.27476 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 20 client.27476 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.562 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b57f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 4 client.27476 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:47.563 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.566+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e50b4f7e70 con 0x55e50b5173e0 2024-04-22T00:09:47.566 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.568+0000 7fb00516e640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b57f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.575+0000 7fb00516e640 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b57f9c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=464fc43fa47c8793 server_cookie=4b12e530debfa6c5 in_seq=0 out_seq=0 2024-04-22T00:09:47.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.575+0000 7fafe8ff9640 10 client.27476 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:47.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.575+0000 7fafe8ff9640 1 -- 172.21.15.38:0/3710825447 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e50b4f7e70 con 0x55e50b5173e0 2024-04-22T00:09:47.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.575+0000 7fafe8ff9640 10 client.27476 handle_command_reply: tid=0 2024-04-22T00:09:47.571 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.575+0000 7fb009fb6740 1 client.27476 shutdown 2024-04-22T00:09:47.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.576+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 msgr2=0x55e50b50aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.572 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.576+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faff8009cd0 tx=0x7faff8005c50 comp rx=0 tx=0).stop 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 shutdown_connections 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55e50b5173e0 0x55e50b57f9c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e50b507520 0x55e50b50aab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e50b4fe940 0x55e50b50f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 --2- 172.21.15.38:0/3710825447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e50b4fe020 0x55e50b50f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.573 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.577+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 >> 172.21.15.38:0/3710825447 conn(0x55e50b4f8750 msgr2=0x55e50b4f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.574 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.579+0000 7fafe8ff9640 0 client.27476 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:09:47.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.582+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 shutdown_connections 2024-04-22T00:09:47.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.583+0000 7fb009fb6740 1 -- 172.21.15.38:0/3710825447 wait complete. 2024-04-22T00:09:47.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.585+0000 7fb009fb6740 20 client.27476 trim_cache size 0 max 16384 2024-04-22T00:09:47.583 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.586+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 msgr2=0x7fafec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.586+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 0x7fafec0a5240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faff8005cc0 tx=0x7faff8005f10 comp rx=0 tx=0).stop 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.586+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 msgr2=0x7fb00019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.586+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7faff000e9f0 tx=0x7faff000eeb0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 shutdown_connections 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fafec0a2d80 0x7fafec0a5240 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb000109990 0x7fb00019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb000104340 0x7fb00019a7d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 --2- 172.21.15.38:0/3706233455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb000103980 0x7fb00019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.587+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 >> 172.21.15.38:0/3706233455 conn(0x7fb0000ff4d0 msgr2=0x7fb0000ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.589+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 shutdown_connections 2024-04-22T00:09:47.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.589+0000 7fb0073d2640 1 -- 172.21.15.38:0/3706233455 wait complete. 2024-04-22T00:09:47.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:09:47.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.794+0000 7fc0f92d4640 1 -- 172.21.15.38:0/2908907996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 msgr2=0x7fc0f410e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.794+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/2908907996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f410e440 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009930 tx=0x7fc0dc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.795+0000 7fc0f92d4640 1 -- 172.21.15.38:0/2908907996 shutdown_connections 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.795+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/2908907996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f411d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.795+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/2908907996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f410e440 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.795+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/2908907996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 0x7fc0f410da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.795+0000 7fc0f92d4640 1 -- 172.21.15.38:0/2908907996 >> 172.21.15.38:0/2908907996 conn(0x7fc0f406c850 msgr2=0x7fc0f406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.796+0000 7fc0f92d4640 1 -- 172.21.15.38:0/2908907996 shutdown_connections 2024-04-22T00:09:47.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.796+0000 7fc0f92d4640 1 -- 172.21.15.38:0/2908907996 wait complete. 2024-04-22T00:09:47.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.797+0000 7fc0f92d4640 1 Processor -- start 2024-04-22T00:09:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 -- start start 2024-04-22T00:09:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f41a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 0x7fc0f41a7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f41ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41208b0 con 0x7fc0f410dfc0 2024-04-22T00:09:47.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f4120730 con 0x7fc0f411af20 2024-04-22T00:09:47.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.798+0000 7fc0f92d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f4120a30 con 0x7fc0f410ff90 2024-04-22T00:09:47.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.800+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f41ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.800+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f41ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46700/0 (socket says 172.21.15.38:46700) 2024-04-22T00:09:47.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.800+0000 7fc0f37fe640 1 -- 172.21.15.38:0/518896234 learned_addr learned my addr 172.21.15.38:0/518896234 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.801+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 0x7fc0f41a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.801+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f41a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 -- 172.21.15.38:0/518896234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 msgr2=0x7fc0f41a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 0x7fc0f41a7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 -- 172.21.15.38:0/518896234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 msgr2=0x7fc0f41ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f41ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 -- 172.21.15.38:0/518896234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f41abdf0 con 0x7fc0f410dfc0 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.802+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f41a6e30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005fd0 tx=0x7fc0ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.803+0000 7fc0ebfff640 1 -- 172.21.15.38:0/518896234 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec00cd80 con 0x7fc0f410dfc0 2024-04-22T00:09:47.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.803+0000 7fc0ebfff640 1 -- 172.21.15.38:0/518896234 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0ec005020 con 0x7fc0f410dfc0 2024-04-22T00:09:47.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.803+0000 7fc0ebfff640 1 -- 172.21.15.38:0/518896234 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0116d0 con 0x7fc0f410dfc0 2024-04-22T00:09:47.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.804+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f41ac0e0 con 0x7fc0f410dfc0 2024-04-22T00:09:47.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.804+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f4110700 con 0x7fc0f410dfc0 2024-04-22T00:09:47.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 Processor -- start 2024-04-22T00:09:47.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 -- start start 2024-04-22T00:09:47.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fde940 0x558995fe6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fe7520 0x558995fe9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558995f42530 con 0x558995fde020 2024-04-22T00:09:47.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558995eea830 con 0x558995fde940 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558995f3e4f0 con 0x558995fe7520 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.807+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38816/0 (socket says 172.21.15.38:38816) 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 -- 172.21.15.38:0/13073759 learned_addr learned my addr 172.21.15.38:0/13073759 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fe7520 0x558995fe9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 -- 172.21.15.38:0/13073759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fe7520 msgr2=0x558995fe9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fe7520 0x558995fe9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 -- 172.21.15.38:0/13073759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fde940 msgr2=0x558995fe6fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fde940 0x558995fe6fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 -- 172.21.15.38:0/13073759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558995f0e2c0 con 0x558995fde020 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0ebfff640 1 -- 172.21.15.38:0/518896234 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc0ec004b20 con 0x7fc0f410dfc0 2024-04-22T00:09:47.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400bd80 tx=0x7fc0e40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80925c9c8fb52a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.808+0000 7fc0e97fa640 1 -- 172.21.15.38:0/13073759 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4011640 con 0x558995fde020 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.809+0000 7fc0e97fa640 1 -- 172.21.15.38:0/13073759 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e400be70 con 0x558995fde020 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.809+0000 7fc0e97fa640 1 -- 172.21.15.38:0/13073759 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4019b80 con 0x558995fde020 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.809+0000 7fc0ebfff640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 0x7fc0e00a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.809+0000 7fc0ebfff640 1 -- 172.21.15.38:0/518896234 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fc0ec0cf890 con 0x7fc0f410dfc0 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/13073759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 msgr2=0x558995fde400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400bd80 tx=0x7fc0e40107a0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/13073759 shutdown_connections 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fe7520 0x558995fe9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fde940 0x558995fe6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/13073759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fde400 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.810+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/13073759 >> 172.21.15.38:0/13073759 conn(0x558995fd8750 msgr2=0x558995fd8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.811+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 0x7fc0e00a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.811+0000 7fc0f2ffd640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 0x7fc0e00a5390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x558995fddaf0 tx=0x7fc0ec097d10 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.819+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/13073759 shutdown_connections 2024-04-22T00:09:47.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.823+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/13073759 wait complete. 2024-04-22T00:09:47.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.823+0000 7fc0fbf9e740 1 -- start start 2024-04-22T00:09:47.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.823+0000 7fc0fbf9e740 1 Processor -- start 2024-04-22T00:09:47.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995fe4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558995fea310 con 0x558995fde020 2024-04-22T00:09:47.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558995efdf10 con 0x558995fe7520 2024-04-22T00:09:47.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558995fea490 con 0x558995fde940 2024-04-22T00:09:47.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.828+0000 7fc0fbf9e740 10 client.0 _ll_register_callbacks cb 0x558995c55be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.832+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.832+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.832+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38820/0 (socket says 172.21.15.38:38820) 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.832+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2894514363 learned_addr learned my addr 172.21.15.38:0/2894514363 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.833+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995fe4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.834+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2894514363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 msgr2=0x558995fe4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.834+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995fe4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.834+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 msgr2=0x558995fe4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.834+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.834+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2894514363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558995ecfc80 con 0x558995fde020 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.835+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995fe4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.835+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4019620 tx=0x7fc0e4010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c886396aa6317676 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.835+0000 7fc0da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:47.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.836+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e402e060 con 0x558995fde020 2024-04-22T00:09:47.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.837+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fc0e4004240 con 0x558995fde020 2024-04-22T00:09:47.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.838+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e400be70 con 0x558995fde020 2024-04-22T00:09:47.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.840+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc0e401f040 con 0x558995fde020 2024-04-22T00:09:47.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.841+0000 7fc0fbf9e740 10 client.18180 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.841+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x558995c563f0 con 0x558995fde020 2024-04-22T00:09:47.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e400be70 con 0x558995fde020 2024-04-22T00:09:47.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc0e40115e0 con 0x558995fde020 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 10 client.18180 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 10 client.18180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 20 client.18180 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558995ff7850 0x558995ff9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 4 client.18180 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:47.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.844+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558995f4fa70 con 0x558995ff7850 2024-04-22T00:09:47.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.846+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558995ff7850 0x558995ff9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.858+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558995ff7850 0x558995ff9c80 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7181f341cf0cd3c1 server_cookie=6766f8a114fff3d9 in_seq=0 out_seq=0 2024-04-22T00:09:47.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.859+0000 7fc0da7fc640 10 client.18180 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:47.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.859+0000 7fc0da7fc640 1 -- 172.21.15.38:0/2894514363 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558995f4fa70 con 0x558995ff7850 2024-04-22T00:09:47.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.859+0000 7fc0da7fc640 10 client.18180 handle_command_reply: tid=0 2024-04-22T00:09:47.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.859+0000 7fc0fbf9e740 1 client.18180 shutdown 2024-04-22T00:09:47.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.860+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 msgr2=0x558995fe6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.860+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4019620 tx=0x7fc0e4010fd0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 shutdown_connections 2024-04-22T00:09:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995fe4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558995ff7850 0x558995ff9c80 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2894514363 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fe6e90 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.858 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.861+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 >> 172.21.15.38:0/2894514363 conn(0x558995fd8750 msgr2=0x558995fda4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.864+0000 7fc0da7fc640 0 client.18180 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:47.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.867+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 shutdown_connections 2024-04-22T00:09:47.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.869+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2894514363 wait complete. 2024-04-22T00:09:47.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.870+0000 7fc0fbf9e740 20 client.18180 trim_cache size 0 max 16384 2024-04-22T00:09:47.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.897+0000 7fc0fbf9e740 1 Processor -- start 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 -- start start 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde940 0x558995fe6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:58600/0 (socket says 172.21.15.38:58600) 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2832819627 learned_addr learned my addr 172.21.15.38:0/2832819627 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde940 0x558995fe6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41c6340 con 0x558995fde940 2024-04-22T00:09:47.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.898+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41c7f40 con 0x558995fe7520 2024-04-22T00:09:47.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41c6a40 con 0x558995fde020 2024-04-22T00:09:47.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2832819627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 msgr2=0x558995fe6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2832819627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde940 msgr2=0x558995fe6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde940 0x558995fe6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 -- 172.21.15.38:0/2832819627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558995d207d0 con 0x558995fde020 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.899+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40199b0 tx=0x7fc0e40197e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4aebb22445eec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.902+0000 7fc0e8ff9640 1 -- 172.21.15.38:0/2832819627 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e402e040 con 0x558995fde020 2024-04-22T00:09:47.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.902+0000 7fc0e8ff9640 1 -- 172.21.15.38:0/2832819627 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4019b70 con 0x558995fde020 2024-04-22T00:09:47.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.902+0000 7fc0e8ff9640 1 -- 172.21.15.38:0/2832819627 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4038080 con 0x558995fde020 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 msgr2=0x558995ff31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40199b0 tx=0x7fc0e40197e0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 shutdown_connections 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995fe6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde940 0x558995fe6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/2832819627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde020 0x558995ff31d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.903+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 >> 172.21.15.38:0/2832819627 conn(0x558995fd8750 msgr2=0x558995fe3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.905+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 shutdown_connections 2024-04-22T00:09:47.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.905+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/2832819627 wait complete. 2024-04-22T00:09:47.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 -- start start 2024-04-22T00:09:47.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 Processor -- start 2024-04-22T00:09:47.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995feabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995feb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41c7870 con 0x558995fde020 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f41c7140 con 0x558995fe7520 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f4068560 con 0x558995fde940 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.906+0000 7fc0fbf9e740 10 client.0 _ll_register_callbacks cb 0x7fc0dc005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.907+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.907+0000 7fc0f37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fea6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38836/0 (socket says 172.21.15.38:38836) 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.907+0000 7fc0f37fe640 1 -- 172.21.15.38:0/909879434 learned_addr learned my addr 172.21.15.38:0/909879434 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.907+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995feb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.907+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995feabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.910+0000 7fc0f27fc640 1 -- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 msgr2=0x558995feb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.910+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995feb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.910+0000 7fc0f27fc640 1 -- 172.21.15.38:0/909879434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 msgr2=0x558995fea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.910+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fea6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.910+0000 7fc0f27fc640 1 -- 172.21.15.38:0/909879434 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55899605be40 con 0x558995fde940 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.911+0000 7fc0f27fc640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995feabe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc02c170 tx=0x7fc0dc005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d8014e591ca94 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:47.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.911+0000 7fc0da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:09:47.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.915+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc004040 con 0x558995fde940 2024-04-22T00:09:47.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.915+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558995f49dc0 con 0x558995fde940 2024-04-22T00:09:47.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.915+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7fc0dc0041e0 con 0x558995fde940 2024-04-22T00:09:47.911 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.915+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc0dc059050 con 0x558995fde940 2024-04-22T00:09:47.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.919+0000 7fc0fbf9e740 10 client.27340 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:47.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.919+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55899605d140 con 0x558995fde940 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.920+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558995f49dc0 con 0x558995fde940 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.920+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc0dc048070 con 0x558995fde940 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 10 client.27340 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 10 client.27340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 20 client.27340 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55899605d870 0x55899605fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 4 client.27340 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:09:47.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.923+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558995ec76e0 con 0x55899605d870 2024-04-22T00:09:47.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.925+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55899605d870 0x55899605fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:47.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.926+0000 7fc0f37fe640 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55899605d870 0x55899605fc90 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81c7684e8e06a2db server_cookie=3058b7ea5fc2de2d in_seq=0 out_seq=0 2024-04-22T00:09:47.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.927+0000 7fc0da7fc640 10 client.27340 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:47.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.928+0000 7fc0da7fc640 1 -- 172.21.15.38:0/909879434 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558995ec76e0 con 0x55899605d870 2024-04-22T00:09:47.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.928+0000 7fc0da7fc640 10 client.27340 handle_command_reply: tid=0 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.929+0000 7fc0fbf9e740 1 client.27340 shutdown 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.929+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 msgr2=0x558995feabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.929+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995feabe0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc02c170 tx=0x7fc0dc005c00 comp rx=0 tx=0).stop 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 shutdown_connections 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558995fe7520 0x558995feb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558995fde940 0x558995feabe0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55899605d870 0x55899605fc90 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 --2- 172.21.15.38:0/909879434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558995fde020 0x558995fea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 >> 172.21.15.38:0/909879434 conn(0x558995fd8750 msgr2=0x558995fd6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.930+0000 7fc0da7fc640 0 client.27340 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:47.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.938+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 shutdown_connections 2024-04-22T00:09:47.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.938+0000 7fc0fbf9e740 1 -- 172.21.15.38:0/909879434 wait complete. 2024-04-22T00:09:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.942+0000 7fc0fbf9e740 20 client.27340 trim_cache size 0 max 16384 2024-04-22T00:09:47.941 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:09:47.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.948+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 msgr2=0x7fc0e00a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.948+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 0x7fc0e00a5390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x558995fddaf0 tx=0x7fc0ec097d10 comp rx=0 tx=0).stop 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.948+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 msgr2=0x7fc0f41a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.948+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f41a6e30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005fd0 tx=0x7fc0ec00ee80 comp rx=0 tx=0).stop 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 shutdown_connections 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc0e00a2ed0 0x7fc0e00a5390 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0f411af20 0x7fc0f41ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0f410ff90 0x7fc0f41a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 --2- 172.21.15.38:0/518896234 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0f410dfc0 0x7fc0f41a6e30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:47.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.949+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 >> 172.21.15.38:0/518896234 conn(0x7fc0f406c850 msgr2=0x7fc0f4119a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:47.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.950+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 shutdown_connections 2024-04-22T00:09:47.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:47.950+0000 7fc0f92d4640 1 -- 172.21.15.38:0/518896234 wait complete. 2024-04-22T00:09:48.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:48 smithi203 ceph-mon[31431]: pgmap v236: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 279 KiB/s wr, 157 op/s 2024-04-22T00:09:48.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:48 smithi111 ceph-mon[31543]: pgmap v236: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 279 KiB/s wr, 157 op/s 2024-04-22T00:09:48.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:48 smithi038 ceph-mon[27972]: pgmap v236: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 279 KiB/s wr, 157 op/s 2024-04-22T00:09:48.965 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:09:48.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:09:48.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:09:49.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 -- 172.21.15.38:0/842613293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30106670 msgr2=0x7f5f30106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 --2- 172.21.15.38:0/842613293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30106670 0x7f5f30106a50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5f18009900 tx=0x7f5f1802bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 -- 172.21.15.38:0/842613293 shutdown_connections 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 --2- 172.21.15.38:0/842613293 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3006a4d0 0x7f5f30105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 --2- 172.21.15.38:0/842613293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f30069a50 0x7f5f30069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 --2- 172.21.15.38:0/842613293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30106670 0x7f5f30106a50 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5f18009900 tx=0x7f5f1802bc30 comp rx=0 tx=0).stop 2024-04-22T00:09:49.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.140+0000 7f5f35878640 1 -- 172.21.15.38:0/842613293 >> 172.21.15.38:0/842613293 conn(0x7f5f300761c0 msgr2=0x7f5f300765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.142+0000 7f5f35878640 1 -- 172.21.15.38:0/842613293 shutdown_connections 2024-04-22T00:09:49.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.149+0000 7f5f35878640 1 -- 172.21.15.38:0/842613293 wait complete. 2024-04-22T00:09:49.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.149+0000 7f5f35878640 1 Processor -- start 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 -- start start 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30069a50 0x7f5f30115140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f3006a4d0 0x7f5f3010e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30118110 con 0x7f5f3010e700 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30117f90 con 0x7f5f30069a50 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.150+0000 7f5f35878640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f30118290 con 0x7f5f3006a4d0 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.151+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.151+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38864/0 (socket says 172.21.15.38:38864) 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.151+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/401643254 learned_addr learned my addr 172.21.15.38:0/401643254 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.151+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f3006a4d0 0x7f5f3010e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.152+0000 7f5f2effd640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30069a50 0x7f5f30115140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.153+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/401643254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f3006a4d0 msgr2=0x7f5f3010e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.153+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f3006a4d0 0x7f5f3010e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.153+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/401643254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30069a50 msgr2=0x7f5f30115140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.153+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30069a50 0x7f5f30115140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.153+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/401643254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f30112d30 con 0x7f5f3010e700 2024-04-22T00:09:49.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.156+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800eb10 tx=0x7f5f2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.156+0000 7f5f34876640 1 -- 172.21.15.38:0/401643254 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28004b20 con 0x7f5f3010e700 2024-04-22T00:09:49.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.157+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f30113020 con 0x7f5f3010e700 2024-04-22T00:09:49.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.157+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f30113570 con 0x7f5f3010e700 2024-04-22T00:09:49.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.157+0000 7f5f34876640 1 -- 172.21.15.38:0/401643254 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f28004cc0 con 0x7f5f3010e700 2024-04-22T00:09:49.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.157+0000 7f5f34876640 1 -- 172.21.15.38:0/401643254 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28010760 con 0x7f5f3010e700 2024-04-22T00:09:49.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.162+0000 7f5f34876640 1 -- 172.21.15.38:0/401643254 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5f28010900 con 0x7f5f3010e700 2024-04-22T00:09:49.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.162+0000 7f5f34876640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 0x7f5f1c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 Processor -- start 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 -- start start 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b80470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b80ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559af4ae4170 con 0x559af4b89db0 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559af4a8c7c0 con 0x559af4b80a40 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.163+0000 7f5f3845a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559af4ae0130 con 0x559af4b81720 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f34876640 1 -- 172.21.15.38:0/401643254 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f5f280d0cb0 con 0x7f5f3010e700 2024-04-22T00:09:49.163 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38868/0 (socket says 172.21.15.38:38868) 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/425361347 learned_addr learned my addr 172.21.15.38:0/425361347 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/425361347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 msgr2=0x559af4b80470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b80470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/425361347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 msgr2=0x559af4b80ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b80ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.164+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/425361347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af4a9de60 con 0x559af4b89db0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.165+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5f2809cae0 tx=0x7f5f28014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1fc2406b95ea24 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.166+0000 7f5f25ffb640 1 -- 172.21.15.38:0/425361347 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2809d4e0 con 0x559af4b89db0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.166+0000 7f5f25ffb640 1 -- 172.21.15.38:0/425361347 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2809cde0 con 0x559af4b89db0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.166+0000 7f5f25ffb640 1 -- 172.21.15.38:0/425361347 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2809d040 con 0x559af4b89db0 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 -- 172.21.15.38:0/425361347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 msgr2=0x559af4b8c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5f2809cae0 tx=0x7f5f28014000 comp rx=0 tx=0).stop 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f2effd640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 0x7f5f1c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 -- 172.21.15.38:0/425361347 shutdown_connections 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4b8c1a0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b80ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 --2- 172.21.15.38:0/425361347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b80470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.167+0000 7f5f3845a740 1 -- 172.21.15.38:0/425361347 >> 172.21.15.38:0/425361347 conn(0x559af4b474f0 msgr2=0x559af4b7b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.168+0000 7f5f3845a740 1 -- 172.21.15.38:0/425361347 shutdown_connections 2024-04-22T00:09:49.165 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.168+0000 7f5f2effd640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 0x7f5f1c0a5390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f18036040 tx=0x7f5f1802df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.169+0000 7f5f3845a740 1 -- 172.21.15.38:0/425361347 wait complete. 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.169+0000 7f5f3845a740 1 -- start start 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.169+0000 7f5f3845a740 1 Processor -- start 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b80a40 0x559af4b86c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4b871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559af4baec20 con 0x559af4b81720 2024-04-22T00:09:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559af4a9fd40 con 0x559af4b89db0 2024-04-22T00:09:49.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559af4baeda0 con 0x559af4b80a40 2024-04-22T00:09:49.167 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f3845a740 10 client.0 _ll_register_callbacks cb 0x559af4adb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:49.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.170+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b80a40 0x559af4b86c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46780/0 (socket says 172.21.15.38:46780) 2024-04-22T00:09:49.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/2872452198 learned_addr learned my addr 172.21.15.38:0/2872452198 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/2872452198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b80a40 msgr2=0x559af4b86c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b80a40 0x559af4b86c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/2872452198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 msgr2=0x559af4b871a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4b871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4b871a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/2872452198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af48c07e0 con 0x559af4b89db0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.171+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f280a5520 tx=0x7f5f280a5b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326d4096fba2a7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.172+0000 7f5f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.172+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280b9030 con 0x559af4b89db0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.172+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f5f2809b0e0 con 0x559af4b89db0 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4b871a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:09:49.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2809cde0 con 0x559af4b89db0 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5f280a55c0 con 0x559af4b89db0 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f3845a740 10 client.27497 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559af4920f70 con 0x559af4b89db0 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.173+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2809cde0 con 0x559af4b89db0 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f5f2809bdc0 con 0x559af4b89db0 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 10 client.27497 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 10 client.27497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 20 client.27497 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4bccd20 0x559af4bcf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 4 client.27497 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.174+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559af4ab8450 con 0x559af4bccd20 2024-04-22T00:09:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.175+0000 7f5f2effd640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4bccd20 0x559af4bcf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.179+0000 7f5f2effd640 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4bccd20 0x559af4bcf110 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2937de9f813d798 server_cookie=84a68aac111c6d54 in_seq=0 out_seq=0 2024-04-22T00:09:49.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.179+0000 7f5f0effd640 10 client.27497 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f0effd640 1 -- 172.21.15.38:0/2872452198 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559af4ab8450 con 0x559af4bccd20 2024-04-22T00:09:49.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f0effd640 10 client.27497 handle_command_reply: tid=0 2024-04-22T00:09:49.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 client.27497 shutdown 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 msgr2=0x559af4b876e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f280a5520 tx=0x7f5f280a5b90 comp rx=0 tx=0).stop 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 shutdown_connections 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b89db0 0x559af4b876e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4bccd20 0x559af4bcf110 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4b871a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 --2- 172.21.15.38:0/2872452198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b80a40 0x559af4b86c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.180+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 >> 172.21.15.38:0/2872452198 conn(0x559af4b474f0 msgr2=0x559af4b7c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.181+0000 7f5f0effd640 0 client.27497 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.181+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 shutdown_connections 2024-04-22T00:09:49.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.181+0000 7f5f3845a740 1 -- 172.21.15.38:0/2872452198 wait complete. 2024-04-22T00:09:49.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.181+0000 7f5f3845a740 20 client.27497 trim_cache size 0 max 16384 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.205+0000 7f5f3845a740 1 Processor -- start 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 -- start start 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b89390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b7a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1000ff50 con 0x559af4b89db0 2024-04-22T00:09:49.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f10011b50 con 0x559af4b80a40 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f3845a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f10010650 con 0x559af4b81720 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38878/0 (socket says 172.21.15.38:38878) 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/55811921 learned_addr learned my addr 172.21.15.38:0/55811921 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2effd640 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b89390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b7a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/55811921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 msgr2=0x559af4b7a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b7a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/55811921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 msgr2=0x559af4b89390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b89390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 -- 172.21.15.38:0/55811921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af47b8170 con 0x559af4b89db0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.206+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5f2809c1c0 tx=0x7f5f280a5b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1860fd712dbab1f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f257fa640 1 -- 172.21.15.38:0/55811921 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2809d4e0 con 0x559af4b89db0 2024-04-22T00:09:49.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f257fa640 1 -- 172.21.15.38:0/55811921 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2809cde0 con 0x559af4b89db0 2024-04-22T00:09:49.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f257fa640 1 -- 172.21.15.38:0/55811921 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2809c230 con 0x559af4b89db0 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f3845a740 1 -- 172.21.15.38:0/55811921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 msgr2=0x559af4bd1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f3845a740 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5f2809c1c0 tx=0x7f5f280a5b00 comp rx=0 tx=0).stop 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f3845a740 1 -- 172.21.15.38:0/55811921 shutdown_connections 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.207+0000 7f5f3845a740 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b89db0 0x559af4bd1d50 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b81720 0x559af4b7a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 --2- 172.21.15.38:0/55811921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4b89390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 -- 172.21.15.38:0/55811921 >> 172.21.15.38:0/55811921 conn(0x559af4b474f0 msgr2=0x559af4b79c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 -- 172.21.15.38:0/55811921 shutdown_connections 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 -- 172.21.15.38:0/55811921 wait complete. 2024-04-22T00:09:49.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 -- start start 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.208+0000 7f5f3845a740 1 Processor -- start 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4bcfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b89db0 0x559af4c31f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f10011480 con 0x559af4b81720 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f10010d50 con 0x559af4b80a40 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f1000df50 con 0x559af4b89db0 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f3845a740 10 client.0 _ll_register_callbacks cb 0x7f5f280a5a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2e7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4bcfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2e7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38892/0 (socket says 172.21.15.38:38892) 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2e7fc640 1 -- 172.21.15.38:0/3707973160 learned_addr learned my addr 172.21.15.38:0/3707973160 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2f7fe640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b89db0 0x559af4c31f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 -- 172.21.15.38:0/3707973160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b89db0 msgr2=0x559af4c31f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b89db0 0x559af4c31f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 -- 172.21.15.38:0/3707973160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 msgr2=0x559af4bd00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 -- 172.21.15.38:0/3707973160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af4bd09b0 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.209+0000 7f5f2effd640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4bcfb70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5f18008000 tx=0x7f5f18009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba4063faca062c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f0effd640 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18004480 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f2809cde0 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mon.2 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f5f1802c880 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f5f1802ca80 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f3845a740 10 client.27503 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559af4c33600 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.210+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mon.2 v2:172.21.15.111:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2809cde0 con 0x559af4b80a40 2024-04-22T00:09:49.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mon.2 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f5f180051e0 con 0x559af4b80a40 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 10 client.27503 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 10 client.27503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 20 client.27503 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4c33d60 0x559af4c36180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 4 client.27503 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559af4af16b0 con 0x559af4c33d60 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4c33d60 0x559af4c36180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f2e7fc640 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4c33d60 0x559af4c36180 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a46926b957ceb3 server_cookie=6e0939ff875a3b78 in_seq=0 out_seq=0 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.211+0000 7f5f0effd640 10 client.27503 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f0effd640 1 -- 172.21.15.38:0/3707973160 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559af4af16b0 con 0x559af4c33d60 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f0effd640 10 client.27503 handle_command_reply: tid=0 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 client.27503 shutdown 2024-04-22T00:09:49.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 msgr2=0x559af4bcfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4bcfb70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5f18008000 tx=0x7f5f18009260 comp rx=0 tx=0).stop 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 shutdown_connections 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af4b89db0 0x559af4c31f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559af4c33d60 0x559af4c36180 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559af4b81720 0x559af4bd00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f0effd640 0 client.27503 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 --2- 172.21.15.38:0/3707973160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559af4b80a40 0x559af4bcfb70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 >> 172.21.15.38:0/3707973160 conn(0x559af4b474f0 msgr2=0x559af4b7a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 shutdown_connections 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.212+0000 7f5f3845a740 1 -- 172.21.15.38:0/3707973160 wait complete. 2024-04-22T00:09:49.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f3845a740 20 client.27503 trim_cache size 0 max 16384 2024-04-22T00:09:49.215 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:49.215 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:09:49.215 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "386e4157-3624-439c-bfac-e7ef2c75cdd5", 2024-04-22T00:09:49.215 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:09:49.215 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 msgr2=0x7f5f1c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 0x7f5f1c0a5390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f18036040 tx=0x7f5f1802df90 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 msgr2=0x7f5f3010ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800eb10 tx=0x7f5f2800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 shutdown_connections 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f1c0a2ed0 0x7f5f1c0a5390 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3010e700 0x7f5f3010ebb0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f3006a4d0 0x7f5f3010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 --2- 172.21.15.38:0/401643254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f30069a50 0x7f5f30115140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 >> 172.21.15.38:0/401643254 conn(0x7f5f300761c0 msgr2=0x7f5f30103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 shutdown_connections 2024-04-22T00:09:49.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.213+0000 7f5f35878640 1 -- 172.21.15.38:0/401643254 wait complete. 2024-04-22T00:09:49.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:09:49.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:09:49.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.345+0000 7f91feaa3640 1 -- 172.21.15.38:0/560000275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f810dfc0 msgr2=0x7f91f810e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.345+0000 7f91feaa3640 1 --2- 172.21.15.38:0/560000275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f810dfc0 0x7f91f810e440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009890 tx=0x7f91e002bb70 comp rx=0 tx=0).stop 2024-04-22T00:09:49.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.346+0000 7f91feaa3640 1 -- 172.21.15.38:0/560000275 shutdown_connections 2024-04-22T00:09:49.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.346+0000 7f91feaa3640 1 --2- 172.21.15.38:0/560000275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f811af20 0x7f91f811d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.346+0000 7f91feaa3640 1 --2- 172.21.15.38:0/560000275 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f810dfc0 0x7f91f810e440 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f91e0009890 tx=0x7f91e002bb70 comp rx=0 tx=0).stop 2024-04-22T00:09:49.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.346+0000 7f91feaa3640 1 --2- 172.21.15.38:0/560000275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91f810ff90 0x7f91f810da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.346+0000 7f91feaa3640 1 -- 172.21.15.38:0/560000275 >> 172.21.15.38:0/560000275 conn(0x7f91f806c850 msgr2=0x7f91f806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.352+0000 7f91feaa3640 1 -- 172.21.15.38:0/560000275 shutdown_connections 2024-04-22T00:09:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.355+0000 7f91feaa3640 1 -- 172.21.15.38:0/560000275 wait complete. 2024-04-22T00:09:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.356+0000 7f91feaa3640 1 Processor -- start 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 -- start start 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f810ff90 0x7f91f811a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91f8116e10 0x7f91f81172c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91f81130e0 con 0x7f91f810ff90 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91f8112f60 con 0x7f91f811af20 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91feaa3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91f8113260 con 0x7f91f8116e10 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.359+0000 7f91f7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46812/0 (socket says 172.21.15.38:46812) 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 -- 172.21.15.38:0/3671062238 learned_addr learned my addr 172.21.15.38:0/3671062238 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 -- 172.21.15.38:0/3671062238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91f8116e10 msgr2=0x7f91f81172c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91f8116e10 0x7f91f81172c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 -- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f810ff90 msgr2=0x7f91f811a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f810ff90 0x7f91f811a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.360+0000 7f91f7fff640 1 -- 172.21.15.38:0/3671062238 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f81aaea0 con 0x7f91f811af20 2024-04-22T00:09:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.361+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f91e0037040 tx=0x7f91e0034930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.362+0000 7f91f5ffb640 1 -- 172.21.15.38:0/3671062238 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e0041020 con 0x7f91f811af20 2024-04-22T00:09:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.362+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f81ab130 con 0x7f91f811af20 2024-04-22T00:09:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.362+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f81ab5d0 con 0x7f91f811af20 2024-04-22T00:09:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.362+0000 7f91f5ffb640 1 -- 172.21.15.38:0/3671062238 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e00035f0 con 0x7f91f811af20 2024-04-22T00:09:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.363+0000 7f91f5ffb640 1 -- 172.21.15.38:0/3671062238 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e0039040 con 0x7f91f811af20 2024-04-22T00:09:49.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.371+0000 7f91f5ffb640 1 -- 172.21.15.38:0/3671062238 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f91e0044020 con 0x7f91f811af20 2024-04-22T00:09:49.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 Processor -- start 2024-04-22T00:09:49.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 -- start start 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea70672ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70675820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ea705ce560 con 0x55ea70669f30 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea70576730 con 0x55ea70673430 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.372+0000 7f9201685740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea705ca520 con 0x55ea7066a850 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91f5ffb640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 0x7f91e40a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91fd040640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91fd040640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38918/0 (socket says 172.21.15.38:38918) 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91fd040640 1 -- 172.21.15.38:0/1838474782 learned_addr learned my addr 172.21.15.38:0/1838474782 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91f5ffb640 1 -- 172.21.15.38:0/3671062238 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f91e00f0d40 con 0x7f91f811af20 2024-04-22T00:09:49.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.373+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea70672ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 -- 172.21.15.38:0/1838474782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 msgr2=0x55ea70672ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea70672ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 -- 172.21.15.38:0/1838474782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 msgr2=0x55ea70675820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70675820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 -- 172.21.15.38:0/1838474782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea703ac7d0 con 0x55ea70669f30 2024-04-22T00:09:49.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.375+0000 7f91fd040640 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f91f001ab90 tx=0x7f91f0021750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfd32e2523dffbb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.379+0000 7f91fd040640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 0x7f91e40a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.381+0000 7f91fd040640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 0x7f91e40a53d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f91f002c040 tx=0x7f91f002abc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.381+0000 7f91eeffd640 1 -- 172.21.15.38:0/1838474782 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f0005040 con 0x55ea70669f30 2024-04-22T00:09:49.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.381+0000 7f91eeffd640 1 -- 172.21.15.38:0/1838474782 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91f00047a0 con 0x55ea70669f30 2024-04-22T00:09:49.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.382+0000 7f91eeffd640 1 -- 172.21.15.38:0/1838474782 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f002a940 con 0x55ea70669f30 2024-04-22T00:09:49.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 -- 172.21.15.38:0/1838474782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 msgr2=0x55ea7066a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f91f001ab90 tx=0x7f91f0021750 comp rx=0 tx=0).stop 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 -- 172.21.15.38:0/1838474782 shutdown_connections 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70675820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea70672ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 --2- 172.21.15.38:0/1838474782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea7066a310 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.384+0000 7f9201685740 1 -- 172.21.15.38:0/1838474782 >> 172.21.15.38:0/1838474782 conn(0x55ea70664660 msgr2=0x55ea70664a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.386+0000 7f9201685740 1 -- 172.21.15.38:0/1838474782 shutdown_connections 2024-04-22T00:09:49.384 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.388+0000 7f9201685740 1 -- 172.21.15.38:0/1838474782 wait complete. 2024-04-22T00:09:49.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.388+0000 7f9201685740 1 -- start start 2024-04-22T00:09:49.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.388+0000 7f9201685740 1 Processor -- start 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea706709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70670ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ea70676220 con 0x55ea70669f30 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ea70589e00 con 0x55ea70673430 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea706763a0 con 0x55ea7066a850 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f9201685740 10 client.0 _ll_register_callbacks cb 0x55ea705c4eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38920/0 (socket says 172.21.15.38:38920) 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.389+0000 7f91fc83f640 1 -- 172.21.15.38:0/3723172952 learned_addr learned my addr 172.21.15.38:0/3723172952 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea706709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fd040640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70670ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 -- 172.21.15.38:0/3723172952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 msgr2=0x55ea706709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea706709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 -- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 msgr2=0x55ea70670ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70670ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 -- 172.21.15.38:0/3723172952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea702e23f0 con 0x55ea70669f30 2024-04-22T00:09:49.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.390+0000 7f91fc83f640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011a50 tx=0x7f91e8011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94c56e6808cd392 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.391+0000 7f91d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:49.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.395+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e800fcd0 con 0x55ea70669f30 2024-04-22T00:09:49.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.395+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f91e8004b20 con 0x55ea70669f30 2024-04-22T00:09:49.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.395+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91f00047a0 con 0x55ea70669f30 2024-04-22T00:09:49.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.403+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f91e800fe70 con 0x55ea70669f30 2024-04-22T00:09:49.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.404+0000 7f9201685740 10 client.18210 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:49.400 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.404+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ea702a20f0 con 0x55ea70669f30 2024-04-22T00:09:49.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91f00047a0 con 0x55ea70669f30 2024-04-22T00:09:49.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f91e8004610 con 0x55ea70669f30 2024-04-22T00:09:49.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 10 client.18210 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:49.403 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 10 client.18210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:49.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 20 client.18210 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:49.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea706838c0 0x55ea70685cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 4 client.18210 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:09:49.404 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.407+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea705dbaa0 con 0x55ea706838c0 2024-04-22T00:09:49.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.408+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea706838c0 0x55ea70685cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.409+0000 7f91f7fff640 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea706838c0 0x55ea70685cf0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197601cfbf0a51e5 server_cookie=835e54f7bec18856 in_seq=0 out_seq=0 2024-04-22T00:09:49.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.411+0000 7f91d7fff640 10 client.18210 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.417+0000 7f91d7fff640 1 -- 172.21.15.38:0/3723172952 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea705dbaa0 con 0x55ea706838c0 2024-04-22T00:09:49.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.417+0000 7f91d7fff640 10 client.18210 handle_command_reply: tid=0 2024-04-22T00:09:49.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.417+0000 7f9201685740 1 client.18210 shutdown 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.421+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 msgr2=0x55ea70670470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.421+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011a50 tx=0x7f91e8011f10 comp rx=0 tx=0).stop 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 shutdown_connections 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70673430 0x55ea70670ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea7066a850 0x55ea706709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea706838c0 0x55ea70685cf0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 --2- 172.21.15.38:0/3723172952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70669f30 0x55ea70670470 secure :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011a50 tx=0x7f91e8011f10 comp rx=0 tx=0).stop 2024-04-22T00:09:49.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 >> 172.21.15.38:0/3723172952 conn(0x55ea70664660 msgr2=0x55ea706663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f91d7fff640 0 client.18210 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.422+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 shutdown_connections 2024-04-22T00:09:49.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.424+0000 7f9201685740 1 -- 172.21.15.38:0/3723172952 wait complete. 2024-04-22T00:09:49.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.424+0000 7f9201685740 20 client.18210 trim_cache size 0 max 16384 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 Processor -- start 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 -- start start 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea7067f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70663560 0x55ea7068b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91d800ffe0 con 0x55ea7066a850 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8011be0 con 0x55ea70663560 2024-04-22T00:09:49.459 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.462+0000 7f9201685740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91d80106e0 con 0x55ea70673430 2024-04-22T00:09:49.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91f7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea7067f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38924/0 (socket says 172.21.15.38:38924) 2024-04-22T00:09:49.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 -- 172.21.15.38:0/1491547068 learned_addr learned my addr 172.21.15.38:0/1491547068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 -- 172.21.15.38:0/1491547068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 msgr2=0x55ea7067f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea7067f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 -- 172.21.15.38:0/1491547068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70663560 msgr2=0x55ea7068b600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70663560 0x55ea7068b600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.463+0000 7f91fc83f640 1 -- 172.21.15.38:0/1491547068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea70664490 con 0x55ea7066a850 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.464+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006010 tx=0x7f91e80021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4635f916ea121444 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.464+0000 7f91ee7fc640 1 -- 172.21.15.38:0/1491547068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8032070 con 0x55ea7066a850 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.467+0000 7f91ee7fc640 1 -- 172.21.15.38:0/1491547068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e80051e0 con 0x55ea7066a850 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.467+0000 7f91ee7fc640 1 -- 172.21.15.38:0/1491547068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e80196b0 con 0x55ea7066a850 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 -- 172.21.15.38:0/1491547068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 msgr2=0x55ea70672ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f91e8006010 tx=0x7f91e80021d0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 -- 172.21.15.38:0/1491547068 shutdown_connections 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea70663560 0x55ea7068b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea7067f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 --2- 172.21.15.38:0/1491547068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea7066a850 0x55ea70672ba0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 -- 172.21.15.38:0/1491547068 >> 172.21.15.38:0/1491547068 conn(0x55ea70664660 msgr2=0x55ea70662e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 -- 172.21.15.38:0/1491547068 shutdown_connections 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.468+0000 7f9201685740 1 -- 172.21.15.38:0/1491547068 wait complete. 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.469+0000 7f9201685740 1 -- start start 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.469+0000 7f9201685740 1 Processor -- start 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.469+0000 7f9201685740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea7066a850 0x55ea70677ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea706e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8011510 con 0x55ea70663560 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8010de0 con 0x55ea7066a850 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91d800dfe0 con 0x55ea70673430 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38928/0 (socket says 172.21.15.38:38928) 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 -- 172.21.15.38:0/1279814109 learned_addr learned my addr 172.21.15.38:0/1279814109 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:09:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91f7fff640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea7066a850 0x55ea70677ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f9201685740 10 client.0 _ll_register_callbacks cb 0x7f91e8011680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fd040640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea706e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 -- 172.21.15.38:0/1279814109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 msgr2=0x55ea706e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea706e2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 -- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea7066a850 msgr2=0x55ea70677ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea7066a850 0x55ea70677ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.470+0000 7f91fc83f640 1 -- 172.21.15.38:0/1279814109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea706e34a0 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.472+0000 7f91fc83f640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009fd0 tx=0x7f91e8005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b855ed3041392bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.472+0000 7f91d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.473+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8032070 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.473+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9438+0+0 (secure 0 0 0) 0x7f91e8036450 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.474+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91e80051e0 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.474+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f91e8036650 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.474+0000 7f9201685740 10 client.18216 fetch_fsmap learned FSMap version 38 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.474+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ea7070c840 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e80051e0 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f91e80198d0 con 0x55ea70663560 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 10 client.18216 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:09:49.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 10 client.18216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:09:49.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 20 client.18216 populate_metadata read hostname 'smithi038' 2024-04-22T00:09:49.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea7070cef0 0x55ea7070f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:09:49.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 4 client.18216 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:09:49.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.475+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea705d5df0 con 0x55ea7070cef0 2024-04-22T00:09:49.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.477+0000 7f91f7fff640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea7070cef0 0x55ea7070f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:09:49.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f91f7fff640 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea7070cef0 0x55ea7070f310 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=403325543ae40ebb server_cookie=7ebeeebf575ecc1c in_seq=0 out_seq=0 2024-04-22T00:09:49.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f91d7fff640 10 client.18216 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f91d7fff640 1 -- 172.21.15.38:0/1279814109 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ea705d5df0 con 0x55ea7070cef0 2024-04-22T00:09:49.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f91d7fff640 10 client.18216 handle_command_reply: tid=0 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f9201685740 1 client.18216 shutdown 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 msgr2=0x55ea70677980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.484+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009fd0 tx=0x7f91e8005c10 comp rx=0 tx=0).stop 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 shutdown_connections 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea70673430 0x55ea706e2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ea7066a850 0x55ea70677ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ea7070cef0 0x55ea7070f310 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 --2- 172.21.15.38:0/1279814109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea70663560 0x55ea70677980 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 >> 172.21.15.38:0/1279814109 conn(0x55ea70664660 msgr2=0x55ea70665ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f91d7fff640 0 client.18216 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:09:49.483 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:09:49.483 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "386e4157-3624-439c-bfac-e7ef2c75cdd5": { 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "386e4157-3624-439c-bfac-e7ef2c75cdd5", 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:09:49.484 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:09:49.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 shutdown_connections 2024-04-22T00:09:49.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 1 -- 172.21.15.38:0/1279814109 wait complete. 2024-04-22T00:09:49.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.485+0000 7f9201685740 20 client.18216 trim_cache size 0 max 16384 2024-04-22T00:09:49.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.487+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 msgr2=0x7f91e40a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.487+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 0x7f91e40a53d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f91f002c040 tx=0x7f91f002abc0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.487+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 msgr2=0x7f91f81168d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:09:49.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.487+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f91e0037040 tx=0x7f91e0034930 comp rx=0 tx=0).stop 2024-04-22T00:09:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 shutdown_connections 2024-04-22T00:09:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f91e40a2f10 0x7f91e40a53d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91f8116e10 0x7f91f81172c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91f811af20 0x7f91f81168d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 --2- 172.21.15.38:0/3671062238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f91f810ff90 0x7f91f811a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:09:49.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.491+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 >> 172.21.15.38:0/3671062238 conn(0x7f91f806c850 msgr2=0x7f91f811be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:09:49.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.494+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 shutdown_connections 2024-04-22T00:09:49.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:09:49.494+0000 7f91feaa3640 1 -- 172.21.15.38:0/3671062238 wait complete. 2024-04-22T00:09:49.510 INFO:tasks.cephfs.filesystem:scrub status for tag:386e4157-3624-439c-bfac-e7ef2c75cdd5 - {'path': '/', 'tag': '386e4157-3624-439c-bfac-e7ef2c75cdd5', 'options': 'recursive,force'} 2024-04-22T00:09:50.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:50 smithi203 ceph-mon[31431]: pgmap v237: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-04-22T00:09:50.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:50 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:09:50.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:50 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:50.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:50 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:09:50.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:50 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:50.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:50 smithi111 ceph-mon[31543]: pgmap v237: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-04-22T00:09:50.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:50 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:09:50.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:50 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:50.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:50 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:09:50.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:50 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:50.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:50 smithi038 ceph-mon[27972]: pgmap v237: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-04-22T00:09:50.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:50 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:09:50.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:50 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:50.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:50 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:09:50.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:50 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:09:51.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:51 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:09:51.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:51 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:09:51.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:51 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:09:52.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:52 smithi203 ceph-mon[31431]: pgmap v238: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 132 KiB/s wr, 194 op/s 2024-04-22T00:09:52.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:52 smithi111 ceph-mon[31543]: pgmap v238: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 132 KiB/s wr, 194 op/s 2024-04-22T00:09:52.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:52 smithi038 ceph-mon[27972]: pgmap v238: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 132 KiB/s wr, 194 op/s 2024-04-22T00:09:54.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:54 smithi203 ceph-mon[31431]: pgmap v239: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 135 KiB/s wr, 252 op/s 2024-04-22T00:09:54.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:54 smithi111 ceph-mon[31543]: pgmap v239: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 135 KiB/s wr, 252 op/s 2024-04-22T00:09:54.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:54 smithi038 ceph-mon[27972]: pgmap v239: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 135 KiB/s wr, 252 op/s 2024-04-22T00:09:56.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:56 smithi203 ceph-mon[31431]: pgmap v240: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-04-22T00:09:56.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:56 smithi111 ceph-mon[31543]: pgmap v240: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-04-22T00:09:56.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:56 smithi038 ceph-mon[27972]: pgmap v240: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 202 op/s 2024-04-22T00:09:58.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:09:58 smithi111 ceph-mon[31543]: pgmap v241: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 205 op/s 2024-04-22T00:09:58.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:09:58 smithi038 ceph-mon[27972]: pgmap v241: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 205 op/s 2024-04-22T00:09:59.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:09:58 smithi203 ceph-mon[31431]: pgmap v241: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 KiB/s wr, 205 op/s 2024-04-22T00:10:00.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:00 smithi111 ceph-mon[31543]: pgmap v242: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 199 op/s 2024-04-22T00:10:00.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:00 smithi111 ceph-mon[31543]: overall HEALTH_OK 2024-04-22T00:10:00.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:00 smithi038 ceph-mon[27972]: pgmap v242: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 199 op/s 2024-04-22T00:10:00.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:00 smithi038 ceph-mon[27972]: overall HEALTH_OK 2024-04-22T00:10:01.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:00 smithi203 ceph-mon[31431]: pgmap v242: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 199 op/s 2024-04-22T00:10:01.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:00 smithi203 ceph-mon[31431]: overall HEALTH_OK 2024-04-22T00:10:02.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:02 smithi111 ceph-mon[31543]: pgmap v243: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 KiB/s wr, 155 op/s 2024-04-22T00:10:02.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:02 smithi038 ceph-mon[27972]: pgmap v243: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 KiB/s wr, 155 op/s 2024-04-22T00:10:03.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:02 smithi203 ceph-mon[31431]: pgmap v243: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.3 KiB/s wr, 155 op/s 2024-04-22T00:10:04.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:04 smithi111 ceph-mon[31543]: pgmap v244: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 151 op/s 2024-04-22T00:10:04.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:04 smithi038 ceph-mon[27972]: pgmap v244: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 151 op/s 2024-04-22T00:10:05.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:04 smithi203 ceph-mon[31431]: pgmap v244: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 151 op/s 2024-04-22T00:10:06.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:06 smithi111 ceph-mon[31543]: pgmap v245: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 KiB/s wr, 105 op/s 2024-04-22T00:10:06.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:06 smithi038 ceph-mon[27972]: pgmap v245: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 KiB/s wr, 105 op/s 2024-04-22T00:10:07.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:06 smithi203 ceph-mon[31431]: pgmap v245: 129 pgs: 129 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 46 KiB/s wr, 105 op/s 2024-04-22T00:10:07.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:07 smithi111 ceph-mon[31543]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T00:10:07.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:07 smithi038 ceph-mon[27972]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T00:10:08.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:07 smithi203 ceph-mon[31431]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T00:10:08.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:08 smithi111 ceph-mon[31543]: pgmap v247: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 55 KiB/s wr, 62 op/s 2024-04-22T00:10:08.871 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:08 smithi038 ceph-mon[27972]: pgmap v247: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 55 KiB/s wr, 62 op/s 2024-04-22T00:10:08.934 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:08 smithi203 ceph-mon[31431]: pgmap v247: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 55 KiB/s wr, 62 op/s 2024-04-22T00:10:10.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:10 smithi038 ceph-mon[27972]: pgmap v248: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:11.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:10 smithi203 ceph-mon[31431]: pgmap v248: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:11.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:10 smithi111 ceph-mon[31543]: pgmap v248: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:11.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:11 smithi038 ceph-mon[27972]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:12.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:11 smithi203 ceph-mon[31431]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:12.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:11 smithi111 ceph-mon[31543]: pgmap v249: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-22T00:10:14.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:14 smithi111 ceph-mon[31543]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:10:14.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:14 smithi038 ceph-mon[27972]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:10:14.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:14 smithi203 ceph-mon[31431]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:10:16.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:16 smithi111 ceph-mon[31543]: pgmap v251: 129 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 108 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T00:10:16.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:16 smithi038 ceph-mon[27972]: pgmap v251: 129 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 108 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T00:10:16.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:16 smithi203 ceph-mon[31431]: pgmap v251: 129 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 108 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T00:10:18.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:18 smithi111 ceph-mon[31543]: pgmap v252: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 294 B/s wr, 11 op/s 2024-04-22T00:10:18.461 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:18 smithi038 ceph-mon[27972]: pgmap v252: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 294 B/s wr, 11 op/s 2024-04-22T00:10:18.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:18 smithi203 ceph-mon[31431]: pgmap v252: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 15 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 294 B/s wr, 11 op/s 2024-04-22T00:10:19.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:10:19.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:10:19.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.645+0000 7f3ba4466640 1 -- 172.21.15.38:0/3490679630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c069dc0 msgr2=0x7f3b9c06a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.645+0000 7f3ba4466640 1 --2- 172.21.15.38:0/3490679630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c06a1a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009900 tx=0x7f3b9802bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:19.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.645+0000 7f3ba4466640 1 -- 172.21.15.38:0/3490679630 shutdown_connections 2024-04-22T00:10:19.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 --2- 172.21.15.38:0/3490679630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c10e150 0x7f3b9c110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 --2- 172.21.15.38:0/3490679630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 --2- 172.21.15.38:0/3490679630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c06a1a0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 -- 172.21.15.38:0/3490679630 >> 172.21.15.38:0/3490679630 conn(0x7f3b9c0776a0 msgr2=0x7f3b9c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 -- 172.21.15.38:0/3490679630 shutdown_connections 2024-04-22T00:10:19.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 -- 172.21.15.38:0/3490679630 wait complete. 2024-04-22T00:10:19.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.646+0000 7f3ba4466640 1 Processor -- start 2024-04-22T00:10:19.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 -- start start 2024-04-22T00:10:19.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c10e150 0x7f3b9c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113d10 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113b90 con 0x7f3b9c069dc0 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba4466640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113e90 con 0x7f3b9c10e150 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55498/0 (socket says 172.21.15.38:55498) 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 -- 172.21.15.38:0/368743431 learned_addr learned my addr 172.21.15.38:0/368743431 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba2202640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c10e150 0x7f3b9c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 -- 172.21.15.38:0/368743431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c10e150 msgr2=0x7f3b9c19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c10e150 0x7f3b9c19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 -- 172.21.15.38:0/368743431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c069dc0 msgr2=0x7f3b9c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 -- 172.21.15.38:0/368743431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9c19f230 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.647+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00e9f0 tx=0x7f3b8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.648+0000 7f3b937fe640 1 -- 172.21.15.38:0/368743431 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00cd80 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.648+0000 7f3b937fe640 1 -- 172.21.15.38:0/368743431 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b8c005500 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.648+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9c19f520 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.648+0000 7f3b937fe640 1 -- 172.21.15.38:0/368743431 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c010690 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.648+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9c1a6db0 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.650+0000 7f3b937fe640 1 -- 172.21.15.38:0/368743431 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3b8c0046c0 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 Processor -- start 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 -- start start 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249055ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249058820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561248fb1560 con 0x56124904cf30 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561248f59730 con 0x561249056430 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba7049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561248fad520 con 0x56124904d850 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba2a03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba2a03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55512/0 (socket says 172.21.15.38:55512) 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba2a03640 1 -- 172.21.15.38:0/2847148921 learned_addr learned my addr 172.21.15.38:0/2847148921 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3b937fe640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 0x7f3b880a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba2202640 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249055ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249058820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.651+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 0x7f3b880a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 -- 172.21.15.38:0/2847148921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 msgr2=0x561249055ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249055ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 -- 172.21.15.38:0/2847148921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 msgr2=0x561249058820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249058820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 -- 172.21.15.38:0/2847148921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561248d8f7d0 con 0x56124904cf30 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3b937fe640 1 -- 172.21.15.38:0/368743431 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3b8c0cfc20 con 0x7f3b9c06a6e0 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3b94013a60 tx=0x7f3b94013f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e086dbb5e9fb2d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3b90bfe640 1 -- 172.21.15.38:0/2847148921 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94023030 con 0x56124904cf30 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3b90bfe640 1 -- 172.21.15.38:0/2847148921 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b9400fe40 con 0x56124904cf30 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3b90bfe640 1 -- 172.21.15.38:0/2847148921 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b940107b0 con 0x56124904cf30 2024-04-22T00:10:19.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.652+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 0x7f3b880a5240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b94023000 tx=0x7f3b94020000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- 172.21.15.38:0/2847148921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 msgr2=0x56124904d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3b94013a60 tx=0x7f3b94013f20 comp rx=0 tx=0).stop 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- 172.21.15.38:0/2847148921 shutdown_connections 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249058820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249055ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 --2- 172.21.15.38:0/2847148921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124904d310 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- 172.21.15.38:0/2847148921 >> 172.21.15.38:0/2847148921 conn(0x561249047660 msgr2=0x561249047a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- 172.21.15.38:0/2847148921 shutdown_connections 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- 172.21.15.38:0/2847148921 wait complete. 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 -- start start 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.653+0000 7f3ba7049740 1 Processor -- start 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249053a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249053f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561249059220 con 0x56124904cf30 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561248f6ce00 con 0x561249056430 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5612490593a0 con 0x56124904d850 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba7049740 10 client.0 _ll_register_callbacks cb 0x561248fa7eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55528/0 (socket says 172.21.15.38:55528) 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 -- 172.21.15.38:0/1026041196 learned_addr learned my addr 172.21.15.38:0/1026041196 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249053f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249053a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 -- 172.21.15.38:0/1026041196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 msgr2=0x561249053a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249053a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 -- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 msgr2=0x561249053f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249053f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 -- 172.21.15.38:0/1026041196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561248cc53f0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3b98005ec0 tx=0x7f3b9802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6e4cc57f3195c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.654+0000 7f3b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004300 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3b980046c0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561248fbeaa0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3b980044a0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 10 client.18228 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561248c850f0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561248fbeaa0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f3b980035f0 con 0x56124904cf30 2024-04-22T00:10:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 10 client.18228 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 20 client.18228 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561249066860 0x561249068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 4 client.18228 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.655+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561248fb8df0 con 0x561249066860 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.656+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561249066860 0x561249068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.656+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561249066860 0x561249068cb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3bc6dc580a724db server_cookie=84b5cda0e4c8e590 in_seq=0 out_seq=0 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.656+0000 7f3b85ffb640 10 client.18228 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1026041196 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561248fb8df0 con 0x561249066860 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3b85ffb640 10 client.18228 handle_command_reply: tid=0 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 client.18228 shutdown 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 msgr2=0x5612490534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3b98005ec0 tx=0x7f3b9802ce30 comp rx=0 tx=0).stop 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 shutdown_connections 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561249056430 0x561249053f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56124904d850 0x561249053a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x561249066860 0x561249068cb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1026041196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x5612490534c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 >> 172.21.15.38:0/1026041196 conn(0x561249047660 msgr2=0x5612490493b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3b85ffb640 0 client.18228 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 shutdown_connections 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 1 -- 172.21.15.38:0/1026041196 wait complete. 2024-04-22T00:10:19.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.657+0000 7f3ba7049740 20 client.18228 trim_cache size 0 max 16384 2024-04-22T00:10:19.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 Processor -- start 2024-04-22T00:10:19.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 -- start start 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249042440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x5612490620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1b0e90 con 0x56124904cf30 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1b2a90 con 0x56124904d850 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba7049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1b1590 con 0x561249056430 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55544/0 (socket says 172.21.15.38:55544) 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba2202640 1 -- 172.21.15.38:0/1436866097 learned_addr learned my addr 172.21.15.38:0/1436866097 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x5612490620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.675+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249042440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 -- 172.21.15.38:0/1436866097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 msgr2=0x5612490620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x5612490620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 -- 172.21.15.38:0/1436866097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 msgr2=0x561249042440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249042440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 -- 172.21.15.38:0/1436866097 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56124904edc0 con 0x56124904cf30 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249042440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f3b980039c0 tx=0x7f3b9802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae06e761d68fc2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3b87fff640 1 -- 172.21.15.38:0/1436866097 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98003c30 con 0x56124904cf30 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3b87fff640 1 -- 172.21.15.38:0/1436866097 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b98005500 con 0x56124904cf30 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.676+0000 7f3b87fff640 1 -- 172.21.15.38:0/1436866097 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004920 con 0x56124904cf30 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- 172.21.15.38:0/1436866097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 msgr2=0x561249055bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f3b980039c0 tx=0x7f3b9802bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:19.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- 172.21.15.38:0/1436866097 shutdown_connections 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x5612490620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249042440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1436866097 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x561249055bf0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- 172.21.15.38:0/1436866097 >> 172.21.15.38:0/1436866097 conn(0x561249047660 msgr2=0x561249048a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- 172.21.15.38:0/1436866097 shutdown_connections 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- 172.21.15.38:0/1436866097 wait complete. 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- start start 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 Processor -- start 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249059bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x56124905a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1b23c0 con 0x56124904cf30 2024-04-22T00:10:19.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1b1c90 con 0x56124904d850 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1aee90 con 0x561249056430 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba7049740 10 client.0 _ll_register_callbacks cb 0x7f3b98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba2202640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55550/0 (socket says 172.21.15.38:55550) 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.677+0000 7f3ba2202640 1 -- 172.21.15.38:0/1949891315 learned_addr learned my addr 172.21.15.38:0/1949891315 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2a03640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x56124905a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 -- 172.21.15.38:0/1949891315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 msgr2=0x56124905a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x56124905a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 -- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 msgr2=0x561249059bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249059bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 -- 172.21.15.38:0/1949891315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56124905a7b0 con 0x56124904cf30 2024-04-22T00:10:19.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba2202640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b98004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24263e2904dd9b3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b980035f0 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3b98005020 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b98005500 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3b98003790 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba7049740 10 client.18234 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5612490cccb0 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.678+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b98005500 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f3b98034440 con 0x56124904cf30 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 10 client.18234 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 10 client.18234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 20 client.18234 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:19.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5612490cd180 0x5612490cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 4 client.18234 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561248f365a0 con 0x5612490cd180 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5612490cd180 0x5612490cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3b85ffb640 10 client.18234 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.679+0000 7f3ba1a01640 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5612490cd180 0x5612490cf570 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e362366cfc54799 server_cookie=48c8495e8771e212 in_seq=0 out_seq=0 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3b85ffb640 1 -- 172.21.15.38:0/1949891315 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561248f365a0 con 0x5612490cd180 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3b85ffb640 10 client.18234 handle_command_reply: tid=0 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 client.18234 shutdown 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 msgr2=0x56124905e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b98004170 comp rx=0 tx=0).stop 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 shutdown_connections 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561249056430 0x56124905a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56124904d850 0x561249059bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5612490cd180 0x5612490cf570 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 --2- 172.21.15.38:0/1949891315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56124904cf30 0x56124905e860 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3b85ffb640 0 client.18234 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 >> 172.21.15.38:0/1949891315 conn(0x561249047660 msgr2=0x561249053460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 shutdown_connections 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 1 -- 172.21.15.38:0/1949891315 wait complete. 2024-04-22T00:10:19.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba7049740 20 client.18234 trim_cache size 0 max 16384 2024-04-22T00:10:19.685 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:19.685 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:10:19.685 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:10:19.685 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 msgr2=0x7f3b880a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 0x7f3b880a5240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b94023000 tx=0x7f3b94020000 comp rx=0 tx=0).stop 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 msgr2=0x7f3b9c19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.680+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00e9f0 tx=0x7f3b8c00edf0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 shutdown_connections 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b880a2d80 0x7f3b880a5240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b9c10e150 0x7f3b9c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b9c06a6e0 0x7f3b9c19a7d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 --2- 172.21.15.38:0/368743431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c069dc0 0x7f3b9c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 >> 172.21.15.38:0/368743431 conn(0x7f3b9c0776a0 msgr2=0x7f3b9c10f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 shutdown_connections 2024-04-22T00:10:19.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.681+0000 7f3ba4466640 1 -- 172.21.15.38:0/368743431 wait complete. 2024-04-22T00:10:19.688 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:10:19.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:10:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.794+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2349044556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 msgr2=0x7fa7e810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.794+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2349044556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e810e1d0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009ca0 tx=0x7fa7e002c180 comp rx=0 tx=0).stop 2024-04-22T00:10:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2349044556 shutdown_connections 2024-04-22T00:10:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2349044556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e810e1d0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2349044556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2349044556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2349044556 >> 172.21.15.38:0/2349044556 conn(0x7fa7e80777a0 msgr2=0x7fa7e8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2349044556 shutdown_connections 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2349044556 wait complete. 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 Processor -- start 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- start start 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e8111b30 con 0x7fa7e810bd60 2024-04-22T00:10:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e81119b0 con 0x7fa7e8103340 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.795+0000 7fa7f05f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e8111cb0 con 0x7fa7e8102a20 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55572/0 (socket says 172.21.15.38:55572) 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 -- 172.21.15.38:0/2834791817 learned_addr learned my addr 172.21.15.38:0/2834791817 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7ee391640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7edb90640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 -- 172.21.15.38:0/2834791817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 msgr2=0x7fa7e8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 -- 172.21.15.38:0/2834791817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 msgr2=0x7fa7e8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 -- 172.21.15.38:0/2834791817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e819cf70 con 0x7fa7e810bd60 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7edb90640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7ee391640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7eeb92640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa7e002c790 tx=0x7fa7e0035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0039040 con 0x7fa7e810bd60 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7e0044070 con 0x7fa7e810bd60 2024-04-22T00:10:19.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e819d200 con 0x7fa7e810bd60 2024-04-22T00:10:19.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e00329f0 con 0x7fa7e810bd60 2024-04-22T00:10:19.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.796+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e81a4af0 con 0x7fa7e810bd60 2024-04-22T00:10:19.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.797+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e80689b0 con 0x7fa7e810bd60 2024-04-22T00:10:19.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.798+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa7e0032b90 con 0x7fa7e810bd60 2024-04-22T00:10:19.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.799+0000 7fa7df7fe640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 0x7fa7d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:19.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.799+0000 7fa7ee391640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 0x7fa7d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:19.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.799+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7fa7e00f2440 con 0x7fa7e810bd60 2024-04-22T00:10:19.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.800+0000 7fa7ee391640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 0x7fa7d40a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005fd0 tx=0x7fa7e400b5f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:19.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:19.803+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa7e00bec80 con 0x7fa7e810bd60 2024-04-22T00:10:20.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.028+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e8106f10 con 0x7fa7e810bd60 2024-04-22T00:10:20.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.029+0000 7fa7df7fe640 1 -- 172.21.15.38:0/2834791817 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11941 (secure 0 0 0) 0x7fa7e0003c70 con 0x7fa7e810bd60 2024-04-22T00:10:20.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:10:20.026 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:09:42.375340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:10:20.026 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 38 2024-04-22T00:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 msgr2=0x7fa7d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 0x7fa7d40a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4005fd0 tx=0x7fa7e400b5f0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 msgr2=0x7fa7e819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa7e002c790 tx=0x7fa7e0035ed0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 shutdown_connections 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa7d40a2d00 0x7fa7d40a51c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7e810bd60 0x7fa7e819c840 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7e8103340 0x7fa7e8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 --2- 172.21.15.38:0/2834791817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7e8102a20 0x7fa7e8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 >> 172.21.15.38:0/2834791817 conn(0x7fa7e80777a0 msgr2=0x7fa7e80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 shutdown_connections 2024-04-22T00:10:20.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.031+0000 7fa7f05f5640 1 -- 172.21.15.38:0/2834791817 wait complete. 2024-04-22T00:10:20.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:10:20.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 -- 172.21.15.38:0/1896020779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc06a780 msgr2=0x7f1fcc10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 --2- 172.21.15.38:0/1896020779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc06a780 0x7f1fcc10e5f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009cd0 tx=0x7f1fc402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 -- 172.21.15.38:0/1896020779 shutdown_connections 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 --2- 172.21.15.38:0/1896020779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc06a780 0x7f1fcc10e5f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 --2- 172.21.15.38:0/1896020779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 --2- 172.21.15.38:0/1896020779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc107730 0x7f1fcc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 -- 172.21.15.38:0/1896020779 >> 172.21.15.38:0/1896020779 conn(0x7f1fcc0776a0 msgr2=0x7f1fcc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 -- 172.21.15.38:0/1896020779 shutdown_connections 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.147+0000 7f1fd244e640 1 -- 172.21.15.38:0/1896020779 wait complete. 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 Processor -- start 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 -- start start 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 0x7f1fcc100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 0x7f1fcc1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc111b40 con 0x7f1fcc069dc0 2024-04-22T00:10:20.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1119c0 con 0x7f1fcc06a780 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fd244e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc111cc0 con 0x7f1fcc107730 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55590/0 (socket says 172.21.15.38:55590) 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.148+0000 7f1fcbfff640 1 -- 172.21.15.38:0/2252108343 learned_addr learned my addr 172.21.15.38:0/2252108343 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 0x7f1fcc100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 0x7f1fcc1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 -- 172.21.15.38:0/2252108343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 msgr2=0x7f1fcc1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 0x7f1fcc1010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 -- 172.21.15.38:0/2252108343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 msgr2=0x7f1fcc100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 0x7f1fcc100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 -- 172.21.15.38:0/2252108343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc1a64a0 con 0x7f1fcc069dc0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 0x7f1fcc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 0x7f1fcc100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e9f0 tx=0x7f1fbc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fc97fa640 1 -- 172.21.15.38:0/2252108343 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00cd80 con 0x7f1fcc069dc0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fcc1a6790 con 0x7f1fcc069dc0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fc97fa640 1 -- 172.21.15.38:0/2252108343 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fbc010910 con 0x7f1fcc069dc0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fc97fa640 1 -- 172.21.15.38:0/2252108343 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc018990 con 0x7f1fcc069dc0 2024-04-22T00:10:20.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.149+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fcc1a6ce0 con 0x7f1fcc069dc0 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.151+0000 7f1fd5039740 1 Processor -- start 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.151+0000 7f1fc97fa640 1 -- 172.21.15.38:0/2252108343 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1fbc010ac0 con 0x7f1fcc069dc0 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.151+0000 7f1fd5039740 1 -- start start 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb020 0x560273abb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560273a1f530 con 0x560273ac4520 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602739c7830 con 0x560273abb940 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd5039740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560273a1b4f0 con 0x560273abb020 2024-04-22T00:10:20.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd09eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb020 0x560273abb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55606/0 (socket says 172.21.15.38:55606) 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 -- 172.21.15.38:0/398962116 learned_addr learned my addr 172.21.15.38:0/398962116 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fc97fa640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 0x7f1fb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 0x7f1fb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 -- 172.21.15.38:0/398962116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb020 msgr2=0x560273abb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb020 0x560273abb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 -- 172.21.15.38:0/398962116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 msgr2=0x560273ac3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 -- 172.21.15.38:0/398962116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602739eb2c0 con 0x560273ac4520 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fc97fa640 1 -- 172.21.15.38:0/2252108343 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1fbc0cfc30 con 0x7f1fcc069dc0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.152+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400bd30 tx=0x7f1fb400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97503467c82d14d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 0x7f1fb80a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4006010 tx=0x7f1fc402c0f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fc27fc640 1 -- 172.21.15.38:0/398962116 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4015070 con 0x560273ac4520 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fc27fc640 1 -- 172.21.15.38:0/398962116 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fb400be20 con 0x560273ac4520 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fc27fc640 1 -- 172.21.15.38:0/398962116 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb40119b0 con 0x560273ac4520 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 -- 172.21.15.38:0/398962116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 msgr2=0x560273ac6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400bd30 tx=0x7f1fb400f730 comp rx=0 tx=0).stop 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 -- 172.21.15.38:0/398962116 shutdown_connections 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273ac4520 0x560273ac6910 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 --2- 172.21.15.38:0/398962116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb020 0x560273abb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 -- 172.21.15.38:0/398962116 >> 172.21.15.38:0/398962116 conn(0x560273ab5750 msgr2=0x560273ab5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 -- 172.21.15.38:0/398962116 shutdown_connections 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.153+0000 7f1fd5039740 1 -- 172.21.15.38:0/398962116 wait complete. 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 -- start start 2024-04-22T00:10:20.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 Processor -- start 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560273ac7310 con 0x560273abb020 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5602739daf10 con 0x560273abb940 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560273ac7490 con 0x560273ac4520 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fd5039740 10 client.0 _ll_register_callbacks cb 0x560273732be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55618/0 (socket says 172.21.15.38:55618) 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.154+0000 7f1fcbfff640 1 -- 172.21.15.38:0/962474047 learned_addr learned my addr 172.21.15.38:0/962474047 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 -- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 msgr2=0x560273ac17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 -- 172.21.15.38:0/962474047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 msgr2=0x560273ac1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 -- 172.21.15.38:0/962474047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602739acc80 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc098730 tx=0x7f1fbc09b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ea92cfecf14051 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc018d30 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1fbc09b1b0 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fb400be20 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1fbc0a0110 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 10 client.18252 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5602737333f0 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fb400be20 con 0x560273abb020 2024-04-22T00:10:20.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1fbc097bd0 con 0x560273abb020 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 10 client.18252 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 10 client.18252 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 20 client.18252 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.155+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273ad4660 0x560273ad6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.156+0000 7f1fd5039740 4 client.18252 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.156+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560273a2ca70 con 0x560273ad4660 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.156+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273ad4660 0x560273ad6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.156+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273ad4660 0x560273ad6b20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a8c93a54c306d43f server_cookie=e6f3d0a38641b8cc in_seq=0 out_seq=0 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.156+0000 7f1fab7fe640 10 client.18252 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fab7fe640 1 -- 172.21.15.38:0/962474047 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560273a2ca70 con 0x560273ad4660 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fab7fe640 10 client.18252 handle_command_reply: tid=0 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 client.18252 shutdown 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 msgr2=0x560273ac3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc098730 tx=0x7f1fbc09b960 comp rx=0 tx=0).stop 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 shutdown_connections 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273ad4660 0x560273ad6b20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fab7fe640 0 client.18252 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 --2- 172.21.15.38:0/962474047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac3ec0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 >> 172.21.15.38:0/962474047 conn(0x560273ab5750 msgr2=0x560273ab74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 shutdown_connections 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 1 -- 172.21.15.38:0/962474047 wait complete. 2024-04-22T00:10:20.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.157+0000 7f1fd5039740 20 client.18252 trim_cache size 0 max 16384 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 Processor -- start 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 -- start start 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1b0ec0 con 0x560273abb020 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1b2ac0 con 0x560273abb940 2024-04-22T00:10:20.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd5039740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1b15c0 con 0x560273ac4520 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55630/0 (socket says 172.21.15.38:55630) 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 -- 172.21.15.38:0/163496550 learned_addr learned my addr 172.21.15.38:0/163496550 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 -- 172.21.15.38:0/163496550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 msgr2=0x560273ac3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 -- 172.21.15.38:0/163496550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 msgr2=0x560273ac3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 -- 172.21.15.38:0/163496550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602737fd7d0 con 0x560273abb020 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.171+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0b2010 tx=0x7f1fbc018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafae3064eb46e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fc1ffb640 1 -- 172.21.15.38:0/163496550 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc097cb0 con 0x560273abb020 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fc1ffb640 1 -- 172.21.15.38:0/163496550 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fbc0d2040 con 0x560273abb020 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fc1ffb640 1 -- 172.21.15.38:0/163496550 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc09b1b0 con 0x560273abb020 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 -- 172.21.15.38:0/163496550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 msgr2=0x560273ad01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0b2010 tx=0x7f1fbc018420 comp rx=0 tx=0).stop 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 -- 172.21.15.38:0/163496550 shutdown_connections 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273ac4520 0x560273ac3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273abb940 0x560273ac3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 --2- 172.21.15.38:0/163496550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ad01d0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 -- 172.21.15.38:0/163496550 >> 172.21.15.38:0/163496550 conn(0x560273ab5750 msgr2=0x560273ab4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 -- 172.21.15.38:0/163496550 shutdown_connections 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.172+0000 7f1fd5039740 1 -- 172.21.15.38:0/163496550 wait complete. 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 -- start start 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 Processor -- start 2024-04-22T00:10:20.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb940 0x560273ac8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273ac4520 0x560273b39c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1b23f0 con 0x560273abb020 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1b1cc0 con 0x560273ac4520 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc1aeec0 con 0x560273abb940 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd5039740 10 client.0 _ll_register_callbacks cb 0x7f1fbc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55634/0 (socket says 172.21.15.38:55634) 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 -- 172.21.15.38:0/1684643242 learned_addr learned my addr 172.21.15.38:0/1684643242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb940 0x560273ac8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fd09eb640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273ac4520 0x560273b39c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 -- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb940 msgr2=0x560273ac8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb940 0x560273ac8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 -- 172.21.15.38:0/1684643242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273ac4520 msgr2=0x560273b39c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273ac4520 0x560273b39c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.173+0000 7f1fcbfff640 1 -- 172.21.15.38:0/1684643242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560273b3a1a0 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fcbfff640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0097b0 tx=0x7f1fbc09be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8903ed771da8e17c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc0a49d0 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1fbc018d40 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fbc0d2040 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1fbc010430 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 10 client.18258 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560273b3b250 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fbc0d2040 con 0x560273abb020 2024-04-22T00:10:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1fbc086420 con 0x560273abb020 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 10 client.18258 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 10 client.18258 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 20 client.18258 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273b3b670 0x560273b3da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 4 client.18258 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560273a26dc0 con 0x560273b3b670 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.174+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273b3b670 0x560273b3da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fab7fe640 10 client.18258 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fcb7fe640 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273b3b670 0x560273b3da90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69ed00d165ef25f5 server_cookie=c2eb033e9a9d92e in_seq=0 out_seq=0 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fab7fe640 1 -- 172.21.15.38:0/1684643242 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560273a26dc0 con 0x560273b3b670 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fab7fe640 10 client.18258 handle_command_reply: tid=0 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 client.18258 shutdown 2024-04-22T00:10:20.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 msgr2=0x560273ac8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0097b0 tx=0x7f1fbc09be90 comp rx=0 tx=0).stop 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 shutdown_connections 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560273b3b670 0x560273b3da90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560273ac4520 0x560273b39c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560273abb940 0x560273ac8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fab7fe640 0 client.18258 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 --2- 172.21.15.38:0/1684643242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560273abb020 0x560273ac8540 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 >> 172.21.15.38:0/1684643242 conn(0x560273ab5750 msgr2=0x560273ab6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 shutdown_connections 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 1 -- 172.21.15.38:0/1684643242 wait complete. 2024-04-22T00:10:20.179 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.175+0000 7f1fd5039740 20 client.18258 trim_cache size 0 max 16384 2024-04-22T00:10:20.180 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:20.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 msgr2=0x7f1fb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 0x7f1fb80a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4006010 tx=0x7f1fc402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 msgr2=0x7f1fcc100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e9f0 tx=0x7f1fbc00eeb0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 shutdown_connections 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1fb80a2d00 0x7f1fb80a51c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fcc107730 0x7f1fcc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1fcc06a780 0x7f1fcc100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 --2- 172.21.15.38:0/2252108343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fcc069dc0 0x7f1fcc100670 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 >> 172.21.15.38:0/2252108343 conn(0x7f1fcc0776a0 msgr2=0x7f1fcc0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 shutdown_connections 2024-04-22T00:10:20.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.176+0000 7f1fd244e640 1 -- 172.21.15.38:0/2252108343 wait complete. 2024-04-22T00:10:20.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.291+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1945489493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e18100610 msgr2=0x7f3e18100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.291+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1945489493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e18100610 0x7f3e18100a90 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009930 tx=0x7f3e0c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1945489493 shutdown_connections 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1945489493 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e1810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1945489493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e18100610 0x7f3e18100a90 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1945489493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e181066b0 0x7f3e18106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1945489493 >> 172.21.15.38:0/1945489493 conn(0x7f3e18077740 msgr2=0x7f3e18077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1945489493 shutdown_connections 2024-04-22T00:10:20.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1945489493 wait complete. 2024-04-22T00:10:20.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 Processor -- start 2024-04-22T00:10:20.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.292+0000 7f3e1e21b640 1 -- start start 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 0x7f3e181022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e181027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181109f0 con 0x7f3e181066b0 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18110870 con 0x7f3e18100610 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18110b70 con 0x7f3e18100fd0 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55648/0 (socket says 172.21.15.38:55648) 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 -- 172.21.15.38:0/1972465621 learned_addr learned my addr 172.21.15.38:0/1972465621 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e177fe640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 0x7f3e181022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e181027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 -- 172.21.15.38:0/1972465621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 msgr2=0x7f3e181027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e181027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 -- 172.21.15.38:0/1972465621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 msgr2=0x7f3e181022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 0x7f3e181022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 -- 172.21.15.38:0/1972465621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e181a64a0 con 0x7f3e181066b0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e177fe640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 0x7f3e181022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e181027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e17fff640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009fd0 tx=0x7f3e1000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e14ff9640 1 -- 172.21.15.38:0/1972465621 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10017070 con 0x7f3e181066b0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e14ff9640 1 -- 172.21.15.38:0/1972465621 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e10003e50 con 0x7f3e181066b0 2024-04-22T00:10:20.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.293+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e181a6790 con 0x7f3e181066b0 2024-04-22T00:10:20.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.294+0000 7f3e14ff9640 1 -- 172.21.15.38:0/1972465621 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e100106c0 con 0x7f3e181066b0 2024-04-22T00:10:20.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.294+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181a6c90 con 0x7f3e181066b0 2024-04-22T00:10:20.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e14ff9640 1 -- 172.21.15.38:0/1972465621 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3e10003970 con 0x7f3e181066b0 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 Processor -- start 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 -- start start 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb7334e020 0x55eb7334e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 0x55eb73359910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55eb732b2530 con 0x55eb7334e940 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55eb7325a830 con 0x55eb7334e020 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.296+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb732ae4f0 con 0x55eb73357520 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55662/0 (socket says 172.21.15.38:55662) 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 -- 172.21.15.38:0/2231810637 learned_addr learned my addr 172.21.15.38:0/2231810637 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e14ff9640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 0x7f3e000a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 0x55eb73359910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb7334e020 0x55eb7334e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e17fff640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 0x7f3e000a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 -- 172.21.15.38:0/2231810637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 msgr2=0x55eb73359910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 0x55eb73359910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 -- 172.21.15.38:0/2231810637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb7334e020 msgr2=0x55eb7334e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb7334e020 0x55eb7334e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 -- 172.21.15.38:0/2231810637 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7327e2c0 con 0x55eb7334e940 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 0x55eb73359910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e14ff9640 1 -- 172.21.15.38:0/1972465621 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3e100cf650 con 0x7f3e181066b0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e177fe640 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800bcb0 tx=0x7f3e0800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6ddfd4314d3236 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3dfdffb640 1 -- 172.21.15.38:0/2231810637 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08015070 con 0x55eb7334e940 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3e17fff640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 0x7f3e000a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e100ccfb0 tx=0x7f3e10080040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3dfdffb640 1 -- 172.21.15.38:0/2231810637 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08011920 con 0x55eb7334e940 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.297+0000 7f3dfdffb640 1 -- 172.21.15.38:0/2231810637 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08010930 con 0x55eb7334e940 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- 172.21.15.38:0/2231810637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 msgr2=0x55eb73356fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800bcb0 tx=0x7f3e0800f730 comp rx=0 tx=0).stop 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- 172.21.15.38:0/2231810637 shutdown_connections 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb73357520 0x55eb73359910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73356fe0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/2231810637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb7334e020 0x55eb7334e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- 172.21.15.38:0/2231810637 >> 172.21.15.38:0/2231810637 conn(0x55eb73348750 msgr2=0x55eb73348b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- 172.21.15.38:0/2231810637 shutdown_connections 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- 172.21.15.38:0/2231810637 wait complete. 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.298+0000 7f3e20dfd740 1 -- start start 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 Processor -- start 2024-04-22T00:10:20.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb73356f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55eb7335a310 con 0x55eb7334e940 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55eb7326df10 con 0x55eb73357520 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb7335a490 con 0x55eb7334e020 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e20dfd740 10 client.0 _ll_register_callbacks cb 0x55eb72fc5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55666/0 (socket says 172.21.15.38:55666) 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 -- 172.21.15.38:0/390565816 learned_addr learned my addr 172.21.15.38:0/390565816 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb73356f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 -- 172.21.15.38:0/390565816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 msgr2=0x55eb73356f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb73356f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 -- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 msgr2=0x55eb733548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 -- 172.21.15.38:0/390565816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb7323fc80 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.299+0000 7f3e177fe640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f3e08000c00 tx=0x7f3e0800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecaef32b710551fd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08015070 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3e08011ad0 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e08011920 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3e0802e050 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 10 client.18270 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55eb72fc63f0 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08011920 con 0x55eb7334e940 2024-04-22T00:10:20.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f3e08004920 con 0x55eb7334e940 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 10 client.18270 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 10 client.18270 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 20 client.18270 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733674a0 0x55eb73369890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 4 client.18270 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.300+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb732bfa70 con 0x55eb733674a0 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.301+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733674a0 0x55eb73369890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.301+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733674a0 0x55eb73369890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ae58db89de1b441 server_cookie=5db9adf91bb45d5c in_seq=0 out_seq=0 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.301+0000 7f3df6ffd640 10 client.18270 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3df6ffd640 1 -- 172.21.15.38:0/390565816 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb732bfa70 con 0x55eb733674a0 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3df6ffd640 10 client.18270 handle_command_reply: tid=0 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 client.18270 shutdown 2024-04-22T00:10:20.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 msgr2=0x55eb733542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f3e08000c00 tx=0x7f3e0800bfd0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 shutdown_connections 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733674a0 0x55eb73369890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb733542a0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3df6ffd640 0 client.18270 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/390565816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb73356f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 >> 172.21.15.38:0/390565816 conn(0x55eb73348750 msgr2=0x55eb7334a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 shutdown_connections 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 1 -- 172.21.15.38:0/390565816 wait complete. 2024-04-22T00:10:20.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.302+0000 7f3e20dfd740 20 client.18270 trim_cache size 0 max 16384 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 Processor -- start 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 -- start start 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb733631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733569d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b0fe0 con 0x55eb7334e940 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b2be0 con 0x55eb73357520 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b16e0 con 0x55eb7334e020 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55668/0 (socket says 172.21.15.38:55668) 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 -- 172.21.15.38:0/985736781 learned_addr learned my addr 172.21.15.38:0/985736781 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 -- 172.21.15.38:0/985736781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 msgr2=0x55eb733631d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb733631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 -- 172.21.15.38:0/985736781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 msgr2=0x55eb733569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 -- 172.21.15.38:0/985736781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7334e770 con 0x55eb7334e940 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3e177fe640 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004cf0 tx=0x7f3e08005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66beaa3744390a88 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3dfd7fa640 1 -- 172.21.15.38:0/985736781 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08015070 con 0x55eb7334e940 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3dfd7fa640 1 -- 172.21.15.38:0/985736781 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08005a70 con 0x55eb7334e940 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.316+0000 7f3dfd7fa640 1 -- 172.21.15.38:0/985736781 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080048c0 con 0x55eb7334e940 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- 172.21.15.38:0/985736781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 msgr2=0x55eb73347850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004cf0 tx=0x7f3e08005970 comp rx=0 tx=0).stop 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- 172.21.15.38:0/985736781 shutdown_connections 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb733569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e940 0x55eb73347850 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/985736781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e020 0x55eb733631d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- 172.21.15.38:0/985736781 >> 172.21.15.38:0/985736781 conn(0x55eb73348750 msgr2=0x55eb73348b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- 172.21.15.38:0/985736781 shutdown_connections 2024-04-22T00:10:20.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- 172.21.15.38:0/985736781 wait complete. 2024-04-22T00:10:20.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 -- start start 2024-04-22T00:10:20.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.317+0000 7f3e20dfd740 1 Processor -- start 2024-04-22T00:10:20.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 0x55eb7335ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb7335b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b2510 con 0x55eb7334e020 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b1de0 con 0x55eb73357520 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181aefe0 con 0x55eb7334e940 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e20dfd740 10 client.0 _ll_register_callbacks cb 0x7f3e0800bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55674/0 (socket says 172.21.15.38:55674) 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 -- 172.21.15.38:0/395417714 learned_addr learned my addr 172.21.15.38:0/395417714 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e177fe640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 0x55eb7335ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 -- 172.21.15.38:0/395417714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 msgr2=0x55eb7335ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 0x55eb7335ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 -- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 msgr2=0x55eb7335b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb7335b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 -- 172.21.15.38:0/395417714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb730907d0 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e177fe640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 0x55eb7335ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3e16ffd640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c036040 tx=0x7f3e0c02c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36aed46b45702d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3df6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c004230 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.318+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f3e0c0043d0 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e08005a70 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3e0c0035f0 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 10 client.18276 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55eb733cd620 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08005a70 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f3e0c043860 con 0x55eb7334e020 2024-04-22T00:10:20.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 10 client.18276 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 10 client.18276 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 20 client.18276 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733cddd0 0x55eb733d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 4 client.18276 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb732b9dc0 con 0x55eb733cddd0 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.319+0000 7f3e177fe640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733cddd0 0x55eb733d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e177fe640 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733cddd0 0x55eb733d01f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=beb9dbb93d2c40f3 server_cookie=28e3b4b2d74a4056 in_seq=0 out_seq=0 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3df6ffd640 10 client.18276 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3df6ffd640 1 -- 172.21.15.38:0/395417714 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb732b9dc0 con 0x55eb733cddd0 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3df6ffd640 10 client.18276 handle_command_reply: tid=0 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 client.18276 shutdown 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 msgr2=0x55eb7335fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c036040 tx=0x7f3e0c02c340 comp rx=0 tx=0).stop 2024-04-22T00:10:20.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 shutdown_connections 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb73357520 0x55eb7335b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb7334e940 0x55eb7335ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55eb733cddd0 0x55eb733d01f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 --2- 172.21.15.38:0/395417714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb7334e020 0x55eb7335fac0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3df6ffd640 0 client.18276 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 >> 172.21.15.38:0/395417714 conn(0x55eb73348750 msgr2=0x55eb73347850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 shutdown_connections 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 1 -- 172.21.15.38:0/395417714 wait complete. 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.320+0000 7f3e20dfd740 20 client.18276 trim_cache size 0 max 16384 2024-04-22T00:10:20.323 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:20.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 msgr2=0x7f3e000a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 0x7f3e000a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e100ccfb0 tx=0x7f3e10080040 comp rx=0 tx=0).stop 2024-04-22T00:10:20.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 msgr2=0x7f3e18102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009fd0 tx=0x7f3e1000b260 comp rx=0 tx=0).stop 2024-04-22T00:10:20.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 shutdown_connections 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3e000a2d30 0x7f3e000a51f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e181066b0 0x7f3e18102d30 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e18100fd0 0x7f3e181027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 --2- 172.21.15.38:0/1972465621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e18100610 0x7f3e181022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 >> 172.21.15.38:0/1972465621 conn(0x7f3e18077740 msgr2=0x7f3e180ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 shutdown_connections 2024-04-22T00:10:20.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.321+0000 7f3e1e21b640 1 -- 172.21.15.38:0/1972465621 wait complete. 2024-04-22T00:10:20.327 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:20 smithi038 ceph-mon[27972]: pgmap v253: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T00:10:20.327 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2834791817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:20.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:10:20.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:20 smithi111 ceph-mon[31543]: pgmap v253: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T00:10:20.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:20 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2834791817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:20.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- 172.21.15.38:0/1360095367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 msgr2=0x7f1ef4110540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.434 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 --2- 172.21.15.38:0/1360095367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef4110540 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009ca0 tx=0x7f1eec02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- 172.21.15.38:0/1360095367 shutdown_connections 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 --2- 172.21.15.38:0/1360095367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef4110540 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 --2- 172.21.15.38:0/1360095367 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef406a6e0 0x7f1ef410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 --2- 172.21.15.38:0/1360095367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef4069dc0 0x7f1ef406a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- 172.21.15.38:0/1360095367 >> 172.21.15.38:0/1360095367 conn(0x7f1ef40776a0 msgr2=0x7f1ef4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- 172.21.15.38:0/1360095367 shutdown_connections 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- 172.21.15.38:0/1360095367 wait complete. 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 Processor -- start 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 -- start start 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.436+0000 7f1efc079640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef4069dc0 0x7f1ef419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 0x7f1ef419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef4113d10 con 0x7f1ef410e150 2024-04-22T00:10:20.435 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef4113b90 con 0x7f1ef4069dc0 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef4113e90 con 0x7f1ef406a6e0 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55698/0 (socket says 172.21.15.38:55698) 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 -- 172.21.15.38:0/1522071366 learned_addr learned my addr 172.21.15.38:0/1522071366 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1ef9614640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 0x7f1ef419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef4069dc0 0x7f1ef419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 -- 172.21.15.38:0/1522071366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 msgr2=0x7f1ef419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 0x7f1ef419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 -- 172.21.15.38:0/1522071366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef4069dc0 msgr2=0x7f1ef419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef4069dc0 0x7f1ef419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 -- 172.21.15.38:0/1522071366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef419f180 con 0x7f1ef410e150 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1ef9614640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 0x7f1ef419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.436 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efa616640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1eec004af0 tx=0x7f1eec032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1eeaffd640 1 -- 172.21.15.38:0/1522071366 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec041020 con 0x7f1ef410e150 2024-04-22T00:10:20.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1eeaffd640 1 -- 172.21.15.38:0/1522071366 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eec034cd0 con 0x7f1ef410e150 2024-04-22T00:10:20.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef419f410 con 0x7f1ef410e150 2024-04-22T00:10:20.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1eeaffd640 1 -- 172.21.15.38:0/1522071366 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec039040 con 0x7f1ef410e150 2024-04-22T00:10:20.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.437+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef41a6d00 con 0x7f1ef410e150 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.439+0000 7f1eeaffd640 1 -- 172.21.15.38:0/1522071366 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1eec044020 con 0x7f1ef410e150 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 Processor -- start 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 -- start start 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1eeaffd640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 0x7f1ee00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca940 0x55f9f89d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f9f892e530 con 0x55f9f89ca020 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f9f88d6830 con 0x55f9f89d3520 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1efec5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f9f892a4f0 con 0x55f9f89ca940 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1ef9614640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 0x7f1ee00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1eeaffd640 1 -- 172.21.15.38:0/1522071366 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1eec0f1500 con 0x7f1ef410e150 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55714/0 (socket says 172.21.15.38:55714) 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.440+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3064787144 learned_addr learned my addr 172.21.15.38:0/3064787144 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9614640 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca940 0x55f9f89d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1efa616640 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3064787144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca940 msgr2=0x55f9f89d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca940 0x55f9f89d2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3064787144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 msgr2=0x55f9f89d5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3064787144 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9f88fa2c0 con 0x55f9f89ca020 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9614640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 0x7f1ee00a5240 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005d80 tx=0x7f1ee4013890 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009c90 tx=0x7f1ef00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f89bbe49918fc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1edffff640 1 -- 172.21.15.38:0/3064787144 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0012450 con 0x55f9f89ca020 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1edffff640 1 -- 172.21.15.38:0/3064787144 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0012ad0 con 0x55f9f89ca020 2024-04-22T00:10:20.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.441+0000 7f1edffff640 1 -- 172.21.15.38:0/3064787144 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef000fbf0 con 0x55f9f89ca020 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- 172.21.15.38:0/3064787144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 msgr2=0x55f9f89ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009c90 tx=0x7f1ef00117b0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- 172.21.15.38:0/3064787144 shutdown_connections 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca940 0x55f9f89d2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3064787144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89ca400 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- 172.21.15.38:0/3064787144 >> 172.21.15.38:0/3064787144 conn(0x55f9f89c4750 msgr2=0x55f9f89c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- 172.21.15.38:0/3064787144 shutdown_connections 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- 172.21.15.38:0/3064787144 wait complete. 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 -- start start 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.442+0000 7f1efec5d740 1 Processor -- start 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f9f89d6310 con 0x55f9f89ca020 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55f9f88e9f10 con 0x55f9f89ca940 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f9f89d6490 con 0x55f9f89d3520 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efec5d740 10 client.0 _ll_register_callbacks cb 0x55f9f8641be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55716/0 (socket says 172.21.15.38:55716) 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 -- 172.21.15.38:0/2291871572 learned_addr learned my addr 172.21.15.38:0/2291871572 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9614640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efa616640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 -- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 msgr2=0x55f9f89d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 -- 172.21.15.38:0/2291871572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 msgr2=0x55f9f89d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 -- 172.21.15.38:0/2291871572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9f88bbc80 con 0x55f9f89ca020 2024-04-22T00:10:20.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1efa616640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0005890 tx=0x7f1ef00058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdace0524afe082 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.443+0000 7f1edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0028060 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1ef000f950 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef0012ad0 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1ef002f050 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 10 client.18288 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f9f86423f0 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0012ad0 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1ef0022070 con 0x55f9f89ca020 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 10 client.18288 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 10 client.18288 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 20 client.18288 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f89e3510 0x55f9f89e5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 4 client.18288 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.444+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9f893ba70 con 0x55f9f89e3510 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.445+0000 7f1ef9614640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f89e3510 0x55f9f89e5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.445+0000 7f1ef9614640 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f89e3510 0x55f9f89e5900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca2007785117f081 server_cookie=8659e554d658dd47 in_seq=0 out_seq=0 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.445+0000 7f1edd7fa640 10 client.18288 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1edd7fa640 1 -- 172.21.15.38:0/2291871572 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f9f893ba70 con 0x55f9f89e3510 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1edd7fa640 10 client.18288 handle_command_reply: tid=0 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 client.18288 shutdown 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 msgr2=0x55f9f89d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0005890 tx=0x7f1ef00058c0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 shutdown_connections 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f89e3510 0x55f9f89e5900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.449 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1edd7fa640 0 client.18288 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 --2- 172.21.15.38:0/2291871572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89d2e80 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 >> 172.21.15.38:0/2291871572 conn(0x55f9f89c4750 msgr2=0x55f9f89c64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 shutdown_connections 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 1 -- 172.21.15.38:0/2291871572 wait complete. 2024-04-22T00:10:20.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.446+0000 7f1efec5d740 20 client.18288 trim_cache size 0 max 16384 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.459+0000 7f1efec5d740 1 Processor -- start 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 -- start start 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b0e20 con 0x55f9f89ca020 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b2a20 con 0x55f9f89ca940 2024-04-22T00:10:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efec5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b1520 con 0x55f9f89d3520 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55732/0 (socket says 172.21.15.38:55732) 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3415042558 learned_addr learned my addr 172.21.15.38:0/3415042558 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efa616640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9614640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3415042558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 msgr2=0x55f9f89d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3415042558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 msgr2=0x55f9f89cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 -- 172.21.15.38:0/3415042558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9f870c7d0 con 0x55f9f89ca020 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1efa616640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000f7b0 tx=0x7f1ef000f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844967a54d43b7ee server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1edf7fe640 1 -- 172.21.15.38:0/3415042558 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0028030 con 0x55f9f89ca020 2024-04-22T00:10:20.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.460+0000 7f1edf7fe640 1 -- 172.21.15.38:0/3415042558 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0009530 con 0x55f9f89ca020 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1edf7fe640 1 -- 172.21.15.38:0/3415042558 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef001f040 con 0x55f9f89ca020 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- 172.21.15.38:0/3415042558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 msgr2=0x55f9f89df1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000f7b0 tx=0x7f1ef000f7e0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- 172.21.15.38:0/3415042558 shutdown_connections 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89d3520 0x55f9f89d2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89ca940 0x55f9f89cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 --2- 172.21.15.38:0/3415042558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca020 0x55f9f89df1d0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- 172.21.15.38:0/3415042558 >> 172.21.15.38:0/3415042558 conn(0x55f9f89c4750 msgr2=0x55f9f89cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- 172.21.15.38:0/3415042558 shutdown_connections 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- 172.21.15.38:0/3415042558 wait complete. 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.461+0000 7f1efec5d740 1 -- start start 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 Processor -- start 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca020 0x55f9f89dbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.464 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b2350 con 0x55f9f89ca940 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b1c20 con 0x55f9f89d3520 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41aee20 con 0x55f9f89ca020 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efec5d740 10 client.0 _ll_register_callbacks cb 0x7f1ef0009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55742/0 (socket says 172.21.15.38:55742) 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 -- 172.21.15.38:0/76626316 learned_addr learned my addr 172.21.15.38:0/76626316 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca020 0x55f9f89dbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efa616640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 -- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca020 msgr2=0x55f9f89dbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca020 0x55f9f89dbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 -- 172.21.15.38:0/76626316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 msgr2=0x55f9f89d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1ef9614640 1 -- 172.21.15.38:0/76626316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9f89d7d30 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.462+0000 7f1efa616640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1ef9614640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400bad0 tx=0x7f1ee4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e613051cbd20cddf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4011c90 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef0009530 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f1ee4027070 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1ee4037050 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 10 client.18294 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f9f8a49930 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0009530 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1ee40035f0 con 0x55f9f89ca940 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 10 client.18294 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 10 client.18294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:10:20.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 20 client.18294 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f8a4a110 0x55f9f8a4c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 4 client.18294 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.463+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9f8935dc0 con 0x55f9f8a4a110 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f8a4a110 0x55f9f8a4c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1edd7fa640 10 client.18294 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1ef9e15640 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f8a4a110 0x55f9f8a4c530 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1467521ac51b965 server_cookie=9fb6bd882dd21254 in_seq=0 out_seq=0 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1edd7fa640 1 -- 172.21.15.38:0/76626316 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9f8935dc0 con 0x55f9f8a4a110 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1edd7fa640 10 client.18294 handle_command_reply: tid=0 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.464+0000 7f1efec5d740 1 client.18294 shutdown 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 msgr2=0x55f9f89d6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400bad0 tx=0x7f1ee4004730 comp rx=0 tx=0).stop 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 shutdown_connections 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55f9f8a4a110 0x55f9f8a4c530 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9f89d3520 0x55f9f89d7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9f89ca940 0x55f9f89d6f20 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 --2- 172.21.15.38:0/76626316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f9f89ca020 0x55f9f89dbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1edd7fa640 0 client.18294 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 >> 172.21.15.38:0/76626316 conn(0x55f9f89c4750 msgr2=0x55f9f89c62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 shutdown_connections 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 1 -- 172.21.15.38:0/76626316 wait complete. 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efec5d740 20 client.18294 trim_cache size 0 max 16384 2024-04-22T00:10:20.468 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 msgr2=0x7f1ee00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 0x7f1ee00a5240 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005d80 tx=0x7f1ee4013890 comp rx=0 tx=0).stop 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 msgr2=0x7f1ef419ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1eec004af0 tx=0x7f1eec032ec0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 shutdown_connections 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ee00a2d80 0x7f1ee00a5240 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ef410e150 0x7f1ef419ea50 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ef406a6e0 0x7f1ef419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 --2- 172.21.15.38:0/1522071366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ef4069dc0 0x7f1ef419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 >> 172.21.15.38:0/1522071366 conn(0x7f1ef40776a0 msgr2=0x7f1ef410f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 shutdown_connections 2024-04-22T00:10:20.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.465+0000 7f1efc079640 1 -- 172.21.15.38:0/1522071366 wait complete. 2024-04-22T00:10:20.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:10:20.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:20 smithi203 ceph-mon[31431]: pgmap v253: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T00:10:20.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:20 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2834791817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:20.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 -- 172.21.15.38:0/663326914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0107730 msgr2=0x7f04e0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/663326914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0107730 0x7f04e0107b10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009930 tx=0x7f04d002bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 -- 172.21.15.38:0/663326914 shutdown_connections 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/663326914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 0x7f04e010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/663326914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0069dc0 0x7f04e006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/663326914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0107730 0x7f04e0107b10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.581+0000 7f04e4fc9640 1 -- 172.21.15.38:0/663326914 >> 172.21.15.38:0/663326914 conn(0x7f04e00776a0 msgr2=0x7f04e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- 172.21.15.38:0/663326914 shutdown_connections 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- 172.21.15.38:0/663326914 wait complete. 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 Processor -- start 2024-04-22T00:10:20.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- start start 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 0x7f04e019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0107730 0x7f04e019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0111b00 con 0x7f04e0069dc0 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0111980 con 0x7f04e0107730 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.582+0000 7f04e4fc9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0111c80 con 0x7f04e006a780 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55756/0 (socket says 172.21.15.38:55756) 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 -- 172.21.15.38:0/85846015 learned_addr learned my addr 172.21.15.38:0/85846015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 0x7f04e019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04ded9d640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0107730 0x7f04e019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 -- 172.21.15.38:0/85846015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 msgr2=0x7f04e019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 0x7f04e019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 -- 172.21.15.38:0/85846015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0107730 msgr2=0x7f04e019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0107730 0x7f04e019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 -- 172.21.15.38:0/85846015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e019f1f0 con 0x7f04e0069dc0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04de59c640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f04d0004f50 tx=0x7f04d002ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04cf7fe640 1 -- 172.21.15.38:0/85846015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0041070 con 0x7f04e0069dc0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e019f480 con 0x7f04e0069dc0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04cf7fe640 1 -- 172.21.15.38:0/85846015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d00044d0 con 0x7f04e0069dc0 2024-04-22T00:10:20.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04cf7fe640 1 -- 172.21.15.38:0/85846015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034440 con 0x7f04e0069dc0 2024-04-22T00:10:20.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.583+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e01a6d70 con 0x7f04e0069dc0 2024-04-22T00:10:20.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.585+0000 7f04cf7fe640 1 -- 172.21.15.38:0/85846015 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f04d0044070 con 0x7f04e0069dc0 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 Processor -- start 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 -- start start 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04cf7fe640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf94126400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf9412f520 0x55bf94131910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bf9408a530 con 0x55bf94126940 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf94032830 con 0x55bf9412f520 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bf940864f0 con 0x55bf94126020 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04ddd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04ded9d640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04ddd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55766/0 (socket says 172.21.15.38:55766) 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04ddd9b640 1 -- 172.21.15.38:0/2172693267 learned_addr learned my addr 172.21.15.38:0/2172693267 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.586+0000 7f04cf7fe640 1 -- 172.21.15.38:0/85846015 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f04d00f13c0 con 0x7f04e0069dc0 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04de59c640 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf94126400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 -- 172.21.15.38:0/2172693267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 msgr2=0x55bf94126400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf94126400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 -- 172.21.15.38:0/2172693267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf9412f520 msgr2=0x55bf94131910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf9412f520 0x55bf94131910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 -- 172.21.15.38:0/2172693267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf940562c0 con 0x55bf94126940 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ded9d640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 0x7f04c80a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005cc0 tx=0x7f04d800a710 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f04d400abf0 tx=0x7f04d400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee2b79cf181df7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ccff9640 1 -- 172.21.15.38:0/2172693267 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d401d070 con 0x55bf94126940 2024-04-22T00:10:20.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ccff9640 1 -- 172.21.15.38:0/2172693267 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4018b20 con 0x55bf94126940 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.587+0000 7f04ccff9640 1 -- 172.21.15.38:0/2172693267 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4010a50 con 0x55bf94126940 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- 172.21.15.38:0/2172693267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 msgr2=0x55bf9412efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f04d400abf0 tx=0x7f04d400f790 comp rx=0 tx=0).stop 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- 172.21.15.38:0/2172693267 shutdown_connections 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf9412f520 0x55bf94131910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126940 0x55bf9412efe0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/2172693267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf94126400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- 172.21.15.38:0/2172693267 >> 172.21.15.38:0/2172693267 conn(0x55bf94120750 msgr2=0x55bf94120b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- 172.21.15.38:0/2172693267 shutdown_connections 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- 172.21.15.38:0/2172693267 wait complete. 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 -- start start 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.588+0000 7f04e7bb1740 1 Processor -- start 2024-04-22T00:10:20.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf9412eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf9412c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bf94132310 con 0x55bf9412f520 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bf94045f10 con 0x55bf94126940 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bf94132490 con 0x55bf94126020 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04e7bb1740 10 client.0 _ll_register_callbacks cb 0x55bf93d9dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55780/0 (socket says 172.21.15.38:55780) 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 -- 172.21.15.38:0/3813283259 learned_addr learned my addr 172.21.15.38:0/3813283259 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04de59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf9412eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf9412c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 -- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 msgr2=0x55bf9412eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf9412eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 -- 172.21.15.38:0/3813283259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 msgr2=0x55bf9412c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf9412c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 -- 172.21.15.38:0/3813283259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf94017c80 con 0x55bf9412f520 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04de59c640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf9412eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf9412c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04ded9d640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f04d8012df0 tx=0x7f04d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbf77adc79963c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8023030 con 0x55bf9412f520 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d4018b20 con 0x55bf9412f520 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.589+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f04d800ae10 con 0x55bf9412f520 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f04d802d050 con 0x55bf9412f520 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 10 client.18306 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55bf93d9e3f0 con 0x55bf9412f520 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4018b20 con 0x55bf9412f520 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f04d801f450 con 0x55bf9412f520 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 10 client.18306 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 10 client.18306 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 20 client.18306 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf9413f690 0x55bf94141b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 4 client.18306 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf94097a70 con 0x55bf9413f690 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.590+0000 7f04de59c640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf9413f690 0x55bf94141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.591+0000 7f04c5ffb640 10 client.18306 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.591+0000 7f04de59c640 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf9413f690 0x55bf94141b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d57c1e10397f3b server_cookie=5317285cde5b211 in_seq=0 out_seq=0 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04c5ffb640 1 -- 172.21.15.38:0/3813283259 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf94097a70 con 0x55bf9413f690 2024-04-22T00:10:20.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04c5ffb640 10 client.18306 handle_command_reply: tid=0 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 client.18306 shutdown 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 msgr2=0x55bf9412c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f04d8012df0 tx=0x7f04d8004a40 comp rx=0 tx=0).stop 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 shutdown_connections 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf9413f690 0x55bf94141b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412c7f0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf9412c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04c5ffb640 0 client.18306 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/3813283259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126020 0x55bf9412eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 >> 172.21.15.38:0/3813283259 conn(0x55bf94120750 msgr2=0x55bf941224a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 shutdown_connections 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 1 -- 172.21.15.38:0/3813283259 wait complete. 2024-04-22T00:10:20.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.592+0000 7f04e7bb1740 20 client.18306 trim_cache size 0 max 16384 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 Processor -- start 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 -- start start 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126020 0x55bf9413b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126940 0x55bf9412e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01b0f30 con 0x55bf9412f520 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01b2b30 con 0x55bf94126020 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01b1630 con 0x55bf94126940 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04ded9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55784/0 (socket says 172.21.15.38:55784) 2024-04-22T00:10:20.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.605+0000 7f04ded9d640 1 -- 172.21.15.38:0/485251411 learned_addr learned my addr 172.21.15.38:0/485251411 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126940 0x55bf9412e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 -- 172.21.15.38:0/485251411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126940 msgr2=0x55bf9412e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126940 0x55bf9412e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 -- 172.21.15.38:0/485251411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126020 msgr2=0x55bf9413b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04de59c640 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126020 0x55bf9413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126020 0x55bf9413b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 -- 172.21.15.38:0/485251411 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf93e687d0 con 0x55bf9412f520 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04ded9d640 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f04d801f8f0 tx=0x7f04d8012ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62a6296f8b72068 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04c7fff640 1 -- 172.21.15.38:0/485251411 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8023040 con 0x55bf9412f520 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04c7fff640 1 -- 172.21.15.38:0/485251411 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d8011040 con 0x55bf9412f520 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.606+0000 7f04c7fff640 1 -- 172.21.15.38:0/485251411 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8005030 con 0x55bf9412f520 2024-04-22T00:10:20.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- 172.21.15.38:0/485251411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 msgr2=0x55bf9412eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f04d801f8f0 tx=0x7f04d8012ad0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- 172.21.15.38:0/485251411 shutdown_connections 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf9412f520 0x55bf9412eec0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf94126940 0x55bf9412e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/485251411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126020 0x55bf9413b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- 172.21.15.38:0/485251411 >> 172.21.15.38:0/485251411 conn(0x55bf94120750 msgr2=0x55bf9411f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- 172.21.15.38:0/485251411 shutdown_connections 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- 172.21.15.38:0/485251411 wait complete. 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- start start 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 Processor -- start 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf94128d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 0x55bf94129440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01b2460 con 0x55bf94126020 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01b1d30 con 0x55bf94126940 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01aef30 con 0x55bf9412f520 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04e7bb1740 10 client.0 _ll_register_callbacks cb 0x7f04d8016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04de59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04de59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55796/0 (socket says 172.21.15.38:55796) 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04de59c640 1 -- 172.21.15.38:0/473772607 learned_addr learned my addr 172.21.15.38:0/473772607 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.607+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf94128d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04ded9d640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 0x55bf94129440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 -- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 msgr2=0x55bf94129440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 0x55bf94129440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 -- 172.21.15.38:0/473772607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 msgr2=0x55bf94128d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf94128d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 -- 172.21.15.38:0/473772607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf941a4c50 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04ded9d640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 0x55bf94129440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf94128d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04de59c640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009430 tx=0x7f04d00bcfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d0cace5e2560d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d004b040 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f04d00bb640 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d8011040 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f04d00bd5a0 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04e7bb1740 10 client.18312 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55bf941a5d80 con 0x55bf94126020 2024-04-22T00:10:20.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.608+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d8011040 con 0x55bf94126020 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f04d00ee940 con 0x55bf94126020 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 10 client.18312 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 10 client.18312 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 20 client.18312 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf941a64b0 0x55bf941a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 4 client.18312 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf94091dc0 con 0x55bf941a64b0 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf941a64b0 0x55bf941a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04c5ffb640 10 client.18312 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04ddd9b640 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf941a64b0 0x55bf941a88d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eec65f228809ee88 server_cookie=2643b427b4c6cd4 in_seq=0 out_seq=0 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04c5ffb640 1 -- 172.21.15.38:0/473772607 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf94091dc0 con 0x55bf941a64b0 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04c5ffb640 10 client.18312 handle_command_reply: tid=0 2024-04-22T00:10:20.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.609+0000 7f04e7bb1740 1 client.18312 shutdown 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 msgr2=0x55bf94137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009430 tx=0x7f04d00bcfc0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 shutdown_connections 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55bf941a64b0 0x55bf941a88d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf9412f520 0x55bf94129440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bf94126940 0x55bf94128d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 --2- 172.21.15.38:0/473772607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bf94126020 0x55bf94137c10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04c5ffb640 0 client.18312 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 >> 172.21.15.38:0/473772607 conn(0x55bf94120750 msgr2=0x55bf94126d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 shutdown_connections 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 1 -- 172.21.15.38:0/473772607 wait complete. 2024-04-22T00:10:20.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e7bb1740 20 client.18312 trim_cache size 0 max 16384 2024-04-22T00:10:20.615 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 msgr2=0x7f04c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 0x7f04c80a51c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005cc0 tx=0x7f04d800a710 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 msgr2=0x7f04e019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f04d0004f50 tx=0x7f04d002ce20 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 shutdown_connections 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04e0107730 0x7f04e019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e006a780 0x7f04e019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 --2- 172.21.15.38:0/85846015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f04e0069dc0 0x7f04e019a200 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 >> 172.21.15.38:0/85846015 conn(0x7f04e00776a0 msgr2=0x7f04e010cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 shutdown_connections 2024-04-22T00:10:20.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.610+0000 7f04e4fc9640 1 -- 172.21.15.38:0/85846015 wait complete. 2024-04-22T00:10:20.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:10:20.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- 172.21.15.38:0/3807407763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b01066b0 msgr2=0x7f34b0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 --2- 172.21.15.38:0/3807407763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b01066b0 0x7f34b0106a90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009930 tx=0x7f34ac02bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- 172.21.15.38:0/3807407763 shutdown_connections 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 --2- 172.21.15.38:0/3807407763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 0x7f34b010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 --2- 172.21.15.38:0/3807407763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b0100610 0x7f34b0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 --2- 172.21.15.38:0/3807407763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b01066b0 0x7f34b0106a90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- 172.21.15.38:0/3807407763 >> 172.21.15.38:0/3807407763 conn(0x7f34b0077740 msgr2=0x7f34b0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- 172.21.15.38:0/3807407763 shutdown_connections 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- 172.21.15.38:0/3807407763 wait complete. 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 Processor -- start 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.728+0000 7f34b7be5640 1 -- start start 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 0x7f34b0199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 0x7f34b01a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0110a30 con 0x7f34b0100610 2024-04-22T00:10:20.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01108b0 con 0x7f34b01066b0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b7be5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34b0110bb0 con 0x7f34b0100fd0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55802/0 (socket says 172.21.15.38:55802) 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 -- 172.21.15.38:0/333731584 learned_addr learned my addr 172.21.15.38:0/333731584 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b6182640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 0x7f34b01a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5180640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 0x7f34b0199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 -- 172.21.15.38:0/333731584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 msgr2=0x7f34b0199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 0x7f34b0199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 -- 172.21.15.38:0/333731584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 msgr2=0x7f34b01a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 0x7f34b01a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 -- 172.21.15.38:0/333731584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b01a43f0 con 0x7f34b0100610 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b6182640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 0x7f34b01a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34b5981640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009230 tx=0x7f34ac02cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.729+0000 7f34a6ffd640 1 -- 172.21.15.38:0/333731584 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac041070 con 0x7f34b0100610 2024-04-22T00:10:20.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.730+0000 7f34a6ffd640 1 -- 172.21.15.38:0/333731584 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac004450 con 0x7f34b0100610 2024-04-22T00:10:20.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.730+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b01a45c0 con 0x7f34b0100610 2024-04-22T00:10:20.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.730+0000 7f34a6ffd640 1 -- 172.21.15.38:0/333731584 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac034440 con 0x7f34b0100610 2024-04-22T00:10:20.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.730+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b01a49c0 con 0x7f34b0100610 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.732+0000 7f34a6ffd640 1 -- 172.21.15.38:0/333731584 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f34ac044070 con 0x7f34b0100610 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.732+0000 7f34ba7d2740 1 Processor -- start 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.732+0000 7f34ba7d2740 1 -- start start 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd4861400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55afd47c5530 con 0x55afd4861940 2024-04-22T00:10:20.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55afd476d830 con 0x55afd486a520 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55afd47c14f0 con 0x55afd4861020 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55816/0 (socket says 172.21.15.38:55816) 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 -- 172.21.15.38:0/1997182016 learned_addr learned my addr 172.21.15.38:0/1997182016 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b6182640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd4861400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34a6ffd640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 0x7f349c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5180640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34a6ffd640 1 -- 172.21.15.38:0/333731584 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f34ac0f13d0 con 0x7f34b0100610 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 -- 172.21.15.38:0/1997182016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 msgr2=0x55afd4861400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd4861400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 -- 172.21.15.38:0/1997182016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 msgr2=0x55afd486c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b5981640 1 -- 172.21.15.38:0/1997182016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd47912c0 con 0x55afd4861940 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b6182640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd4861400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.733+0000 7f34b6182640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 0x7f349c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.734+0000 7f34b5981640 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0bd310 tx=0x7f34ac009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d6aa7af14ae6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.734+0000 7f349bfff640 1 -- 172.21.15.38:0/1997182016 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac04b030 con 0x55afd4861940 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.734+0000 7f349bfff640 1 -- 172.21.15.38:0/1997182016 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac004c80 con 0x55afd4861940 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.734+0000 7f349bfff640 1 -- 172.21.15.38:0/1997182016 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0bd610 con 0x55afd4861940 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.734+0000 7f34b6182640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 0x7f349c0a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f34a8005fd0 tx=0x7f34a8004a40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- 172.21.15.38:0/1997182016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 msgr2=0x55afd4869fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0bd310 tx=0x7f34ac009260 comp rx=0 tx=0).stop 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- 172.21.15.38:0/1997182016 shutdown_connections 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd4869fe0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/1997182016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd4861400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- 172.21.15.38:0/1997182016 >> 172.21.15.38:0/1997182016 conn(0x55afd485b750 msgr2=0x55afd485bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- 172.21.15.38:0/1997182016 shutdown_connections 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- 172.21.15.38:0/1997182016 wait complete. 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 -- start start 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.735+0000 7f34ba7d2740 1 Processor -- start 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd48678d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55afd47c5530 con 0x55afd4861020 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55afd4780f10 con 0x55afd4861940 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55afd486d0d0 con 0x55afd486a520 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 10 client.0 _ll_register_callbacks cb 0x55afd44d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55820/0 (socket says 172.21.15.38:55820) 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 -- 172.21.15.38:0/794885623 learned_addr learned my addr 172.21.15.38:0/794885623 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5180640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b6182640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd48678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 -- 172.21.15.38:0/794885623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 msgr2=0x55afd48678d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd48678d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 -- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 msgr2=0x55afd48672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48672c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 -- 172.21.15.38:0/794885623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd4752c80 con 0x55afd4861020 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34b5981640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0eed30 tx=0x7f34ac0ee860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f3fbb968591474 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34997fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac04b070 con 0x55afd4861020 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f34ac0b9c80 con 0x55afd4861020 2024-04-22T00:10:20.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.736+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34ac004c80 con 0x55afd4861020 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f34ac0bd150 con 0x55afd4861020 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 10 client.18324 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55afd44d93f0 con 0x55afd4861020 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac004c80 con 0x55afd4861020 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f34ac0c5df0 con 0x55afd4861020 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 10 client.18324 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 10 client.18324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 20 client.18324 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd487a100 0x55afd487c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 4 client.18324 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd47d2a70 con 0x55afd487a100 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.737+0000 7f34b5180640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd487a100 0x55afd487c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.738+0000 7f34b5180640 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd487a100 0x55afd487c4f0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2afd2f6dfcac900 server_cookie=5e0d3073ef9cc49a in_seq=0 out_seq=0 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.738+0000 7f34997fa640 10 client.18324 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:20.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.738+0000 7f34997fa640 1 -- 172.21.15.38:0/794885623 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55afd47d2a70 con 0x55afd487a100 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.738+0000 7f34997fa640 10 client.18324 handle_command_reply: tid=0 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 client.18324 shutdown 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 msgr2=0x55afd4869f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0eed30 tx=0x7f34ac0ee860 comp rx=0 tx=0).stop 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 shutdown_connections 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd48678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48672c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd487a100 0x55afd487c4f0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/794885623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4869f30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34997fa640 0 client.18324 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 >> 172.21.15.38:0/794885623 conn(0x55afd485b750 msgr2=0x55afd485d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 shutdown_connections 2024-04-22T00:10:20.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 1 -- 172.21.15.38:0/794885623 wait complete. 2024-04-22T00:10:20.743 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.739+0000 7f34ba7d2740 20 client.18324 trim_cache size 0 max 16384 2024-04-22T00:10:20.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 Processor -- start 2024-04-22T00:10:20.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 -- start start 2024-04-22T00:10:20.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.756 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd4869f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01aebe0 con 0x55afd4861020 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01b07e0 con 0x55afd4861940 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01af2e0 con 0x55afd486a520 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55832/0 (socket says 172.21.15.38:55832) 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 -- 172.21.15.38:0/3787955148 learned_addr learned my addr 172.21.15.38:0/3787955148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5180640 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b6182640 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd4869f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 -- 172.21.15.38:0/3787955148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 msgr2=0x55afd4869f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd4869f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 -- 172.21.15.38:0/3787955148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 msgr2=0x55afd48699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.754+0000 7f34b5981640 1 -- 172.21.15.38:0/3787955148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd45a37d0 con 0x55afd4861020 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34b5981640 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0ee8f0 tx=0x7f34ac009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11cdfc578dd8274 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f349b7fe640 1 -- 172.21.15.38:0/3787955148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac04b070 con 0x55afd4861020 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f349b7fe640 1 -- 172.21.15.38:0/3787955148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac0b9040 con 0x55afd4861020 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f349b7fe640 1 -- 172.21.15.38:0/3787955148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0bcf70 con 0x55afd4861020 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3787955148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 msgr2=0x55afd4875e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0ee8f0 tx=0x7f34ac009260 comp rx=0 tx=0).stop 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3787955148 shutdown_connections 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd486a520 0x55afd4869f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd4861940 0x55afd48699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3787955148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861020 0x55afd4875e10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3787955148 >> 172.21.15.38:0/3787955148 conn(0x55afd485b750 msgr2=0x55afd485a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.755+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3787955148 shutdown_connections 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3787955148 wait complete. 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 -- start start 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 Processor -- start 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd48728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01b0110 con 0x55afd4861940 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01af9e0 con 0x55afd486a520 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34b01acbe0 con 0x55afd4861020 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34ba7d2740 10 client.0 _ll_register_callbacks cb 0x7f34a000b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:20.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.756+0000 7f34b5180640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd48728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5180640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55836/0 (socket says 172.21.15.38:55836) 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5180640 1 -- 172.21.15.38:0/3861301891 learned_addr learned my addr 172.21.15.38:0/3861301891 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b6182640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 -- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 msgr2=0x55afd486e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 -- 172.21.15.38:0/3861301891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 msgr2=0x55afd486dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 -- 172.21.15.38:0/3861301891 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd48df5e0 con 0x55afd4861020 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5180640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34b5981640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd48728b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55afd4868840 tx=0x7f34ac0ee920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a1397197015ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34997fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:10:20.760 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac04b070 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34ac0b9040 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.757+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9526+0+0 (secure 0 0 0) 0x7f34ac0b9c80 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f34ac0b98b0 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 10 client.27469 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55afd48df5e0 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac0b9040 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f34ac0d1b30 con 0x55afd4861020 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 10 client.27469 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 10 client.27469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 20 client.27469 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd48e0a50 0x55afd48e2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 4 client.27469 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afd47ccdc0 con 0x55afd48e0a50 2024-04-22T00:10:20.761 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.758+0000 7f34b5180640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd48e0a50 0x55afd48e2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34997fa640 10 client.27469 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34b5180640 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd48e0a50 0x55afd48e2e70 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3112b5bac29b387 server_cookie=66bd0b4b6f6e97f4 in_seq=0 out_seq=0 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34997fa640 1 -- 172.21.15.38:0/3861301891 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afd47ccdc0 con 0x55afd48e0a50 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34997fa640 10 client.27469 handle_command_reply: tid=0 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 client.27469 shutdown 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 msgr2=0x55afd48728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd48728b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55afd4868840 tx=0x7f34ac0ee920 comp rx=0 tx=0).stop 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 shutdown_connections 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55afd486a520 0x55afd486e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55afd48e0a50 0x55afd48e2e70 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55afd4861940 0x55afd486dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 --2- 172.21.15.38:0/3861301891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afd4861020 0x55afd48728b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.762 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34997fa640 0 client.27469 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 >> 172.21.15.38:0/3861301891 conn(0x55afd485b750 msgr2=0x55afd4861e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 shutdown_connections 2024-04-22T00:10:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 1 -- 172.21.15.38:0/3861301891 wait complete. 2024-04-22T00:10:20.763 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.759+0000 7f34ba7d2740 20 client.27469 trim_cache size 0 max 16384 2024-04-22T00:10:20.763 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 msgr2=0x7f349c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 0x7f349c0a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f34a8005fd0 tx=0x7f34a8004a40 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 msgr2=0x7f34b0198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009230 tx=0x7f34ac02cbf0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 shutdown_connections 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f349c0a2d00 0x7f349c0a51c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34b01066b0 0x7f34b01a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34b0100fd0 0x7f34b0199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 --2- 172.21.15.38:0/333731584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f34b0100610 0x7f34b0198e30 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 >> 172.21.15.38:0/333731584 conn(0x7f34b0077740 msgr2=0x7f34b00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 shutdown_connections 2024-04-22T00:10:20.764 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:20.760+0000 7f34b7be5640 1 -- 172.21.15.38:0/333731584 wait complete. 2024-04-22T00:10:21.767 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:10:21.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:10:21.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:10:21.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 -- 172.21.15.38:0/1157332049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e98102150 msgr2=0x7f2e9810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/1157332049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e98102150 0x7f2e9810e660 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009cd0 tx=0x7f2e9002c0f0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 -- 172.21.15.38:0/1157332049 shutdown_connections 2024-04-22T00:10:21.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/1157332049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e98102150 0x7f2e9810e660 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/1157332049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 0x7f2e98101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/1157332049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e981077a0 0x7f2e98107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 -- 172.21.15.38:0/1157332049 >> 172.21.15.38:0/1157332049 conn(0x7f2e980fd2a0 msgr2=0x7f2e980ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 -- 172.21.15.38:0/1157332049 shutdown_connections 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.894+0000 7f2e9e45f640 1 -- 172.21.15.38:0/1157332049 wait complete. 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 Processor -- start 2024-04-22T00:10:21.892 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 -- start start 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 0x7f2e98198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 0x7f2e981985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98111b70 con 0x7f2e981077a0 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981119f0 con 0x7f2e98101790 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9e45f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98111cf0 con 0x7f2e98102150 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.895+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 0x7f2e981985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e97fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 0x7f2e98198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55848/0 (socket says 172.21.15.38:55848) 2024-04-22T00:10:21.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/400769605 learned_addr learned my addr 172.21.15.38:0/400769605 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 0x7f2e981985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:44036/0 (socket says 172.21.15.38:44036) 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/400769605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 msgr2=0x7f2e981985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 0x7f2e981985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/400769605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 msgr2=0x7f2e98198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 0x7f2e98198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/400769605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9819d080 con 0x7f2e981077a0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2e90004af0 tx=0x7f2e90032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e957fa640 1 -- 172.21.15.38:0/400769605 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90041020 con 0x7f2e981077a0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9819d310 con 0x7f2e981077a0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e957fa640 1 -- 172.21.15.38:0/400769605 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e90034cd0 con 0x7f2e981077a0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981a4c00 con 0x7f2e981077a0 2024-04-22T00:10:21.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.896+0000 7f2e957fa640 1 -- 172.21.15.38:0/400769605 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90039040 con 0x7f2e981077a0 2024-04-22T00:10:21.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e957fa640 1 -- 172.21.15.38:0/400769605 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2e90044020 con 0x7f2e981077a0 2024-04-22T00:10:21.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 Processor -- start 2024-04-22T00:10:21.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 -- start start 2024-04-22T00:10:21.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e957fa640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 0x7f2e800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611d8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611e61c0 0x559d611e85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d6113c170 con 0x559d611d9720 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559d610e47c0 con 0x559d611d8a40 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2ea1042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559d61138130 con 0x559d611e61c0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e977fe640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 0x7f2e800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e957fa640 1 -- 172.21.15.38:0/400769605 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f2e900f1450 con 0x7f2e981077a0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e97fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e97fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55858/0 (socket says 172.21.15.38:55858) 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.899+0000 7f2e97fff640 1 -- 172.21.15.38:0/340509186 learned_addr learned my addr 172.21.15.38:0/340509186 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e977fe640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611d8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611e61c0 0x559d611e85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 -- 172.21.15.38:0/340509186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611e61c0 msgr2=0x559d611e85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611e61c0 0x559d611e85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 -- 172.21.15.38:0/340509186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 msgr2=0x559d611d8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611d8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 -- 172.21.15.38:0/340509186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d610f5e60 con 0x559d611d9720 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e977fe640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611d8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e977fe640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 0x7f2e800a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2e980ff8d0 tx=0x7f2e8c013880 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e97fff640 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800bc50 tx=0x7f2e880107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8537b09601a2719 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e7e7fc640 1 -- 172.21.15.38:0/340509186 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8801c480 con 0x559d611d9720 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e7e7fc640 1 -- 172.21.15.38:0/340509186 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8801cb00 con 0x559d611d9720 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.900+0000 7f2e7e7fc640 1 -- 172.21.15.38:0/340509186 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88011b00 con 0x559d611d9720 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- 172.21.15.38:0/340509186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 msgr2=0x559d611d8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800bc50 tx=0x7f2e880107a0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- 172.21.15.38:0/340509186 shutdown_connections 2024-04-22T00:10:21.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611e61c0 0x559d611e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611d8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 --2- 172.21.15.38:0/340509186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611d8470 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- 172.21.15.38:0/340509186 >> 172.21.15.38:0/340509186 conn(0x559d6119f4f0 msgr2=0x559d611d3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- 172.21.15.38:0/340509186 shutdown_connections 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- 172.21.15.38:0/340509186 wait complete. 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.901+0000 7f2ea1042740 1 -- start start 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 Processor -- start 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d9720 0x559d611e3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d611e9030 con 0x559d611e61c0 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559d610f7d40 con 0x559d611d9720 2024-04-22T00:10:21.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559d611e91b0 con 0x559d611d8a40 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2ea1042740 10 client.0 _ll_register_callbacks cb 0x559d61133820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55874/0 (socket says 172.21.15.38:55874) 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1330202889 learned_addr learned my addr 172.21.15.38:0/1330202889 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e977fe640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d9720 0x559d611e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1330202889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 msgr2=0x559d611e3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d9720 msgr2=0x559d611e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d9720 0x559d611e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.902+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1330202889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d60f187e0 con 0x559d611e61c0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009ca0 tx=0x7f2e900b9930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60934ad749e15274 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900bcff0 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f2e900eea30 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8801cb00 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2e900411c0 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 10 client.18342 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559d60f78f70 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8801cb00 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f2e9003c030 con 0x559d611e61c0 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 10 client.18342 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 10 client.18342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 20 client.18342 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d611f6260 0x559d611f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 4 client.18342 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.903+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d61110450 con 0x559d611f6260 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.904+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d611f6260 0x559d611f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.904+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d611f6260 0x559d611f86c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=992de8114508723a server_cookie=fc3a3293b0b82fa4 in_seq=0 out_seq=0 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.904+0000 7f2e777fe640 10 client.18342 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2e777fe640 1 -- 172.21.15.38:0/1330202889 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d61110450 con 0x559d611f6260 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2e777fe640 10 client.18342 handle_command_reply: tid=0 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 client.18342 shutdown 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 msgr2=0x559d611e3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009ca0 tx=0x7f2e900b9930 comp rx=0 tx=0).stop 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 shutdown_connections 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d611f6260 0x559d611f86c0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611e3b80 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d9720 0x559d611e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1330202889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2e777fe640 0 client.18342 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:21.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 >> 172.21.15.38:0/1330202889 conn(0x559d6119f4f0 msgr2=0x559d611d51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 shutdown_connections 2024-04-22T00:10:21.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 1 -- 172.21.15.38:0/1330202889 wait complete. 2024-04-22T00:10:21.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.905+0000 7f2ea1042740 20 client.18342 trim_cache size 0 max 16384 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.923+0000 7f2ea1042740 1 Processor -- start 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.923+0000 7f2ea1042740 1 -- start start 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.923+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.923+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611e61c0 0x559d611fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2ea1042740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981ac890 con 0x559d611d9720 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2ea1042740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981ae490 con 0x559d611e61c0 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2ea1042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981acf90 con 0x559d611d8a40 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55878/0 (socket says 172.21.15.38:55878) 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 -- 172.21.15.38:0/511899478 learned_addr learned my addr 172.21.15.38:0/511899478 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:21.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611e61c0 0x559d611fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e97fff640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 -- 172.21.15.38:0/511899478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 msgr2=0x559d611e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 -- 172.21.15.38:0/511899478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611e61c0 msgr2=0x559d611fdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611e61c0 0x559d611fdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 -- 172.21.15.38:0/511899478 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d60e10170 con 0x559d611d9720 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e97fff640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e977fe640 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c012390 tx=0x7f2e8c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9839ad4e845dabe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e7dffb640 1 -- 172.21.15.38:0/511899478 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c003d70 con 0x559d611d9720 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e7dffb640 1 -- 172.21.15.38:0/511899478 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c005500 con 0x559d611d9720 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.924+0000 7f2e7dffb640 1 -- 172.21.15.38:0/511899478 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00f7a0 con 0x559d611d9720 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- 172.21.15.38:0/511899478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 msgr2=0x559d611f1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c012390 tx=0x7f2e8c01f000 comp rx=0 tx=0).stop 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- 172.21.15.38:0/511899478 shutdown_connections 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611e61c0 0x559d611fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611d9720 0x559d611f1f90 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 --2- 172.21.15.38:0/511899478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d8a40 0x559d611e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- 172.21.15.38:0/511899478 >> 172.21.15.38:0/511899478 conn(0x559d6119f4f0 msgr2=0x559d611d24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- 172.21.15.38:0/511899478 shutdown_connections 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- 172.21.15.38:0/511899478 wait complete. 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.925+0000 7f2ea1042740 1 -- start start 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 Processor -- start 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 0x559d611ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981addc0 con 0x559d611e61c0 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981ad690 con 0x559d611d8a40 2024-04-22T00:10:21.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981aa890 con 0x559d611d9720 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2ea1042740 10 client.0 _ll_register_callbacks cb 0x7f2e900b9270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55890/0 (socket says 172.21.15.38:55890) 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1490355338 learned_addr learned my addr 172.21.15.38:0/1490355338 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e977fe640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 0x559d611ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1490355338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 msgr2=0x559d611ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 0x559d611ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 msgr2=0x559d611ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611ee4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 -- 172.21.15.38:0/1490355338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d611dc580 con 0x559d611e61c0 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e977fe640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 0x559d611ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:21.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e9c9fc640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2e900c1bb0 tx=0x7f2e90032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305976fe41c155f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.926+0000 7f2e777fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900bcff0 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f2e900a6dd0 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c005500 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f2e900bd460 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 10 client.18348 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559d61261bd0 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c005500 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f2e900c5e70 con 0x559d611e61c0 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 10 client.18348 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 10 client.18348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 20 client.18348 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d612620a0 0x559d61264490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:21.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 4 client.18348 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:10:21.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.927+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559d611496b0 con 0x559d612620a0 2024-04-22T00:10:21.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.928+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d612620a0 0x559d61264490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:21.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.928+0000 7f2e97fff640 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d612620a0 0x559d61264490 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a991257a4a2c2c9 server_cookie=e1a41bd933b3d150 in_seq=0 out_seq=0 2024-04-22T00:10:21.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.928+0000 7f2e777fe640 10 client.18348 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2e777fe640 1 -- 172.21.15.38:0/1490355338 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559d611496b0 con 0x559d612620a0 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2e777fe640 10 client.18348 handle_command_reply: tid=0 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 client.18348 shutdown 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 msgr2=0x559d611dbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f2e900c1bb0 tx=0x7f2e90032b00 comp rx=0 tx=0).stop 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 shutdown_connections 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559d612620a0 0x559d61264490 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2e777fe640 0 client.18348 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d611e61c0 0x559d611dbd50 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d611d9720 0x559d611ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 --2- 172.21.15.38:0/1490355338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559d611d8a40 0x559d611ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 >> 172.21.15.38:0/1490355338 conn(0x559d6119f4f0 msgr2=0x559d611d4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 shutdown_connections 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.929+0000 7f2ea1042740 1 -- 172.21.15.38:0/1490355338 wait complete. 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2ea1042740 20 client.18348 trim_cache size 0 max 16384 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:10:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "78afef8b-5d99-4500-b61f-4aec90ffb416", 2024-04-22T00:10:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:10:21.933 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:21.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 msgr2=0x7f2e800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 0x7f2e800a51c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2e980ff8d0 tx=0x7f2e8c013880 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 msgr2=0x7f2e9819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2e90004af0 tx=0x7f2e90032ec0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 shutdown_connections 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2e800a2d00 0x7f2e800a51c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e981077a0 0x7f2e9819c950 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98102150 0x7f2e981985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 --2- 172.21.15.38:0/400769605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e98101790 0x7f2e98198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 >> 172.21.15.38:0/400769605 conn(0x7f2e980fd2a0 msgr2=0x7f2e980fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 shutdown_connections 2024-04-22T00:10:21.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:21.930+0000 7f2e9e45f640 1 -- 172.21.15.38:0/400769605 wait complete. 2024-04-22T00:10:21.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:10:21.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 -- 172.21.15.38:0/2777172914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 msgr2=0x7fa61410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 --2- 172.21.15.38:0/2777172914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61410e1a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009ca0 tx=0x7fa60c02c230 comp rx=0 tx=0).stop 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 -- 172.21.15.38:0/2777172914 shutdown_connections 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 --2- 172.21.15.38:0/2777172914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61410e1a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 --2- 172.21.15.38:0/2777172914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa61410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 --2- 172.21.15.38:0/2777172914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 0x7fa614102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 -- 172.21.15.38:0/2777172914 >> 172.21.15.38:0/2777172914 conn(0x7fa614077740 msgr2=0x7fa614077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 -- 172.21.15.38:0/2777172914 shutdown_connections 2024-04-22T00:10:22.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.046+0000 7fa619db1640 1 -- 172.21.15.38:0/2777172914 wait complete. 2024-04-22T00:10:22.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 Processor -- start 2024-04-22T00:10:22.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 -- start start 2024-04-22T00:10:22.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 0x7fa614198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa614198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa614111b00 con 0x7fa61410bd00 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa614111980 con 0x7fa6141032e0 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa619db1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa614111c80 con 0x7fa6141029c0 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55896/0 (socket says 172.21.15.38:55896) 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 -- 172.21.15.38:0/970379186 learned_addr learned my addr 172.21.15.38:0/970379186 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa612ffd640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa614198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa6137fe640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 0x7fa614198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 -- 172.21.15.38:0/970379186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 msgr2=0x7fa614198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 0x7fa614198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 -- 172.21.15.38:0/970379186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 msgr2=0x7fa614198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa614198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.047+0000 7fa613fff640 1 -- 172.21.15.38:0/970379186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61419cfa0 con 0x7fa61410bd00 2024-04-22T00:10:22.047 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa612ffd640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa614198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa613fff640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa60c02c840 tx=0x7fa60c035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa610ff9640 1 -- 172.21.15.38:0/970379186 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c039040 con 0x7fa61410bd00 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa61419d230 con 0x7fa61410bd00 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa610ff9640 1 -- 172.21.15.38:0/970379186 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa60c044070 con 0x7fa61410bd00 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa610ff9640 1 -- 172.21.15.38:0/970379186 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c004330 con 0x7fa61410bd00 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.048+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6141a4b20 con 0x7fa61410bd00 2024-04-22T00:10:22.048 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.050+0000 7fa610ff9640 1 -- 172.21.15.38:0/970379186 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa60c0044d0 con 0x7fa61410bd00 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.050+0000 7fa61c99a740 1 Processor -- start 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.050+0000 7fa61c99a740 1 -- start start 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2e14560 con 0x55d1f2eb0850 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2dbc730 con 0x55d1f2eb9430 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa61c99a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2e10520 con 0x55d1f2eaff30 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa610ff9640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 0x7fa5fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55906/0 (socket says 172.21.15.38:55906) 2024-04-22T00:10:22.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 -- 172.21.15.38:0/1486779671 learned_addr learned my addr 172.21.15.38:0/1486779671 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa610ff9640 1 -- 172.21.15.38:0/970379186 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa60c0f1c90 con 0x7fa61410bd00 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa613fff640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 0x7fa5fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 -- 172.21.15.38:0/1486779671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 msgr2=0x55d1f2eb0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 -- 172.21.15.38:0/1486779671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 msgr2=0x55d1f2ebb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 -- 172.21.15.38:0/1486779671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f2bf27d0 con 0x55d1f2eb0850 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa60800bcf0 tx=0x7fa60800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b697b051e715b630 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.051+0000 7fa613fff640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 0x7fa5fc0a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0ef210 tx=0x7fa60c02c230 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa5f9ffb640 1 -- 172.21.15.38:0/1486779671 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608015070 con 0x55d1f2eb0850 2024-04-22T00:10:22.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa5f9ffb640 1 -- 172.21.15.38:0/1486779671 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa60801a940 con 0x55d1f2eb0850 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa5f9ffb640 1 -- 172.21.15.38:0/1486779671 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6080119d0 con 0x55d1f2eb0850 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 -- 172.21.15.38:0/1486779671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 msgr2=0x55d1f2eb8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa60800bcf0 tx=0x7fa60800f730 comp rx=0 tx=0).stop 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 -- 172.21.15.38:0/1486779671 shutdown_connections 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb8ef0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1486779671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 -- 172.21.15.38:0/1486779671 >> 172.21.15.38:0/1486779671 conn(0x55d1f2eaa660 msgr2=0x55d1f2eaaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 -- 172.21.15.38:0/1486779671 shutdown_connections 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.052+0000 7fa61c99a740 1 -- 172.21.15.38:0/1486779671 wait complete. 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 -- start start 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 Processor -- start 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2eb6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2ebc220 con 0x55d1f2eb0850 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2dcfe00 con 0x55d1f2eb9430 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1f2ebc3a0 con 0x55d1f2eaff30 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa61c99a740 10 client.0 _ll_register_callbacks cb 0x55d1f2e0aeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa6137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa612ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa6137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55920/0 (socket says 172.21.15.38:55920) 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa6137fe640 1 -- 172.21.15.38:0/1252788667 learned_addr learned my addr 172.21.15.38:0/1252788667 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.053+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2eb6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 -- 172.21.15.38:0/1252788667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 msgr2=0x55d1f2eb6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 -- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 msgr2=0x55d1f2eb6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2eb6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 -- 172.21.15.38:0/1252788667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f2b283f0 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa6137fe640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa608006010 tx=0x7fa60800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a9f4f2d17b3938 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa5f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608028060 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa60801a6f0 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa60801a940 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa60802f050 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa61c99a740 10 client.18360 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d1f2ae80f0 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.054+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa60801a940 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa608011940 con 0x55d1f2eb0850 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 10 client.18360 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:22.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 10 client.18360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 20 client.18360 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2ec9810 0x55d1f2ecbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 4 client.18360 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1f2e21aa0 con 0x55d1f2ec9810 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2ec9810 0x55d1f2ecbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa5f2ffd640 10 client.18360 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.055+0000 7fa612ffd640 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2ec9810 0x55d1f2ecbc00 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f7e4bf5bb780bd server_cookie=cd617bcf408c19fe in_seq=0 out_seq=0 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/1252788667 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1f2e21aa0 con 0x55d1f2ec9810 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa5f2ffd640 10 client.18360 handle_command_reply: tid=0 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 client.18360 shutdown 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 msgr2=0x55d1f2eb6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa608006010 tx=0x7fa60800b920 comp rx=0 tx=0).stop 2024-04-22T00:10:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 shutdown_connections 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb9430 0x55d1f2eb6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2ec9810 0x55d1f2ecbc00 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb0850 0x55d1f2eb6a40 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 --2- 172.21.15.38:0/1252788667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 >> 172.21.15.38:0/1252788667 conn(0x55d1f2eaa660 msgr2=0x55d1f2eac3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa5f2ffd640 0 client.18360 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 shutdown_connections 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 1 -- 172.21.15.38:0/1252788667 wait complete. 2024-04-22T00:10:22.061 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.056+0000 7fa61c99a740 20 client.18360 trim_cache size 0 max 16384 2024-04-22T00:10:22.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 Processor -- start 2024-04-22T00:10:22.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 -- start start 2024-04-22T00:10:22.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb0850 0x55d1f2ec5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141aebd0 con 0x55d1f2eb9430 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141b07d0 con 0x55d1f2eb0850 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa61c99a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141af2d0 con 0x55d1f2eaff30 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55924/0 (socket says 172.21.15.38:55924) 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 -- 172.21.15.38:0/3849035201 learned_addr learned my addr 172.21.15.38:0/3849035201 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa6137fe640 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb0850 0x55d1f2ec5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 -- 172.21.15.38:0/3849035201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 msgr2=0x55d1f2eb8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 -- 172.21.15.38:0/3849035201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb0850 msgr2=0x55d1f2ec5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.075+0000 7fa612ffd640 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb0850 0x55d1f2ec5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa612ffd640 1 -- 172.21.15.38:0/3849035201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f2eaf900 con 0x55d1f2eb9430 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa612ffd640 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa604004280 tx=0x7fa604013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2acfadf8a98292 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa5f97fa640 1 -- 172.21.15.38:0/3849035201 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6040233e0 con 0x55d1f2eb9430 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa5f97fa640 1 -- 172.21.15.38:0/3849035201 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6040035f0 con 0x55d1f2eb9430 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa5f97fa640 1 -- 172.21.15.38:0/3849035201 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60400f450 con 0x55d1f2eb9430 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa61c99a740 1 -- 172.21.15.38:0/3849035201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 msgr2=0x55d1f2ea9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.076+0000 7fa61c99a740 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fa604004280 tx=0x7fa604013040 comp rx=0 tx=0).stop 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- 172.21.15.38:0/3849035201 shutdown_connections 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ea9560 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eb0850 0x55d1f2ec5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- 172.21.15.38:0/3849035201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eaff30 0x55d1f2eb8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- 172.21.15.38:0/3849035201 >> 172.21.15.38:0/3849035201 conn(0x55d1f2eaa660 msgr2=0x55d1f2ea8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- 172.21.15.38:0/3849035201 shutdown_connections 2024-04-22T00:10:22.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- 172.21.15.38:0/3849035201 wait complete. 2024-04-22T00:10:22.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- start start 2024-04-22T00:10:22.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 Processor -- start 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eaff30 0x55d1f2ec1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eb0850 0x55d1f2ebcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141b0100 con 0x55d1f2eb9430 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141af9d0 con 0x55d1f2eaff30 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141acbd0 con 0x55d1f2eb0850 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa61c99a740 10 client.0 _ll_register_callbacks cb 0x7fa6040099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.077+0000 7fa612ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55930/0 (socket says 172.21.15.38:55930) 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 -- 172.21.15.38:0/2366407421 learned_addr learned my addr 172.21.15.38:0/2366407421 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eaff30 0x55d1f2ec1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa6137fe640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eb0850 0x55d1f2ebcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 -- 172.21.15.38:0/2366407421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eb0850 msgr2=0x55d1f2ebcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eb0850 0x55d1f2ebcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 -- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eaff30 msgr2=0x55d1f2ec1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eaff30 0x55d1f2ec1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 -- 172.21.15.38:0/2366407421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f2ebdb90 con 0x55d1f2eb9430 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa612ffd640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa6040057a0 tx=0x7fa604009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ca8137c2ab76c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa5f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604003830 con 0x55d1f2eb9430 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa6040039d0 con 0x55d1f2eb9430 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6040035f0 con 0x55d1f2eb9430 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa6040233e0 con 0x55d1f2eb9430 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa61c99a740 10 client.18366 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:22.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d1f2f2fac0 con 0x55d1f2eb9430 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.078+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6040035f0 con 0x55d1f2eb9430 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa604021050 con 0x55d1f2eb9430 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 10 client.18366 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 20 client.18366 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2f30160 0x55d1f2f32580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 4 client.18366 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d1f2e1bdf0 con 0x55d1f2f30160 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa6137fe640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2f30160 0x55d1f2f32580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa5f2ffd640 10 client.18366 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa6137fe640 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2f30160 0x55d1f2f32580 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4bdc25bb951b71f server_cookie=76903a471384ef62 in_seq=0 out_seq=0 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa5f2ffd640 1 -- 172.21.15.38:0/2366407421 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d1f2e1bdf0 con 0x55d1f2f30160 2024-04-22T00:10:22.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa5f2ffd640 10 client.18366 handle_command_reply: tid=0 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.079+0000 7fa61c99a740 1 client.18366 shutdown 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 msgr2=0x55d1f2ebd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa6040057a0 tx=0x7fa604009600 comp rx=0 tx=0).stop 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 shutdown_connections 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d1f2f30160 0x55d1f2f32580 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d1f2eb9430 0x55d1f2ebd360 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa5f2ffd640 0 client.18366 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1f2eb0850 0x55d1f2ebcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 --2- 172.21.15.38:0/2366407421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f2eaff30 0x55d1f2ec1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 >> 172.21.15.38:0/2366407421 conn(0x55d1f2eaa660 msgr2=0x55d1f2ea96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 shutdown_connections 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 1 -- 172.21.15.38:0/2366407421 wait complete. 2024-04-22T00:10:22.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa61c99a740 20 client.18366 trim_cache size 0 max 16384 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "78afef8b-5d99-4500-b61f-4aec90ffb416": { 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "78afef8b-5d99-4500-b61f-4aec90ffb416", 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:10:22.084 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:22.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 msgr2=0x7fa5fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 0x7fa5fc0a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0ef210 tx=0x7fa60c02c230 comp rx=0 tx=0).stop 2024-04-22T00:10:22.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 msgr2=0x7fa61419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:22.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa60c02c840 tx=0x7fa60c035d90 comp rx=0 tx=0).stop 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 shutdown_connections 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa5fc0a2d00 0x7fa5fc0a51c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61410bd00 0x7fa61419c870 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6141032e0 0x7fa614198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 --2- 172.21.15.38:0/970379186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6141029c0 0x7fa614198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.080+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 >> 172.21.15.38:0/970379186 conn(0x7fa614077740 msgr2=0x7fa6140fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.081+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 shutdown_connections 2024-04-22T00:10:22.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:22.081+0000 7fa619db1640 1 -- 172.21.15.38:0/970379186 wait complete. 2024-04-22T00:10:22.088 INFO:tasks.cephfs.filesystem:scrub status for tag:78afef8b-5d99-4500-b61f-4aec90ffb416 - {'path': '/', 'tag': '78afef8b-5d99-4500-b61f-4aec90ffb416', 'options': 'recursive,force'} 2024-04-22T00:10:22.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:22 smithi038 ceph-mon[27972]: pgmap v254: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T00:10:22.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:22 smithi038 ceph-mon[27972]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T00:10:22.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:22 smithi203 ceph-mon[31431]: pgmap v254: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T00:10:22.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:22 smithi203 ceph-mon[31431]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T00:10:22.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:22 smithi111 ceph-mon[31543]: pgmap v254: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T00:10:22.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:22 smithi111 ceph-mon[31543]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T00:10:23.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:23 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:10:23.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:23 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:23.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:23 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:10:23.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:23 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:23.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:23 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:10:23.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:23 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:23.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:23 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:10:23.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:23 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:23.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:23 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:10:23.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:23 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:23.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:23 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:10:23.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:23 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:24.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:24 smithi038 ceph-mon[27972]: pgmap v256: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-22T00:10:24.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:24 smithi203 ceph-mon[31431]: pgmap v256: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-22T00:10:24.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:24 smithi111 ceph-mon[31543]: pgmap v256: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-22T00:10:26.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:26 smithi038 ceph-mon[27972]: pgmap v257: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T00:10:26.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:26 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:10:26.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:26 smithi203 ceph-mon[31431]: pgmap v257: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T00:10:26.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:26 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:10:26.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:26 smithi111 ceph-mon[31543]: pgmap v257: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T00:10:26.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:26 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:10:28.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:28 smithi038 ceph-mon[27972]: pgmap v258: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:28.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:28 smithi203 ceph-mon[31431]: pgmap v258: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:28.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:28 smithi111 ceph-mon[31543]: pgmap v258: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:30.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:30 smithi038 ceph-mon[27972]: pgmap v259: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:30.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:30 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:10:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:30 smithi203 ceph-mon[31431]: pgmap v259: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:30 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:10:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:30 smithi111 ceph-mon[31543]: pgmap v259: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:30 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:10:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:10:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:10:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:10:31.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:10:31.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:10:31.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:10:31.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:10:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:10:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:10:32.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:32 smithi203 ceph-mon[31431]: pgmap v260: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:32.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:32 smithi111 ceph-mon[31543]: pgmap v260: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:32.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:32 smithi038 ceph-mon[27972]: pgmap v260: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-04-22T00:10:34.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:34 smithi203 ceph-mon[31431]: pgmap v261: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:34.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:34 smithi111 ceph-mon[31543]: pgmap v261: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:34.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:34 smithi038 ceph-mon[27972]: pgmap v261: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:36.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:36 smithi203 ceph-mon[31431]: pgmap v262: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-22T00:10:36.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:36 smithi111 ceph-mon[31543]: pgmap v262: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-22T00:10:36.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:36 smithi038 ceph-mon[27972]: pgmap v262: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-22T00:10:38.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:38 smithi111 ceph-mon[31543]: pgmap v263: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:38.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:38 smithi038 ceph-mon[27972]: pgmap v263: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:39.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:38 smithi203 ceph-mon[31431]: pgmap v263: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-04-22T00:10:40.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:40 smithi111 ceph-mon[31543]: pgmap v264: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:40.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:40 smithi038 ceph-mon[27972]: pgmap v264: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:41.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:40 smithi203 ceph-mon[31431]: pgmap v264: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:42.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:42 smithi111 ceph-mon[31543]: pgmap v265: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:42.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:42 smithi038 ceph-mon[27972]: pgmap v265: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:43.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:42 smithi203 ceph-mon[31431]: pgmap v265: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:44.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:44 smithi111 ceph-mon[31543]: pgmap v266: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:44.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:44 smithi038 ceph-mon[27972]: pgmap v266: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:45.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:44 smithi203 ceph-mon[31431]: pgmap v266: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:46.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:46 smithi111 ceph-mon[31543]: pgmap v267: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:10:46.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:46 smithi038 ceph-mon[27972]: pgmap v267: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:10:47.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:46 smithi203 ceph-mon[31431]: pgmap v267: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:10:48.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:48 smithi111 ceph-mon[31543]: pgmap v268: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:10:48.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:48 smithi038 ceph-mon[27972]: pgmap v268: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:10:49.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:48 smithi203 ceph-mon[31431]: pgmap v268: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:10:50.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:50 smithi038 ceph-mon[27972]: pgmap v269: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:51.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:50 smithi203 ceph-mon[31431]: pgmap v269: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:51.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:50 smithi111 ceph-mon[31543]: pgmap v269: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:10:52.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:10:52.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:10:52.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 -- 172.21.15.38:0/836310132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101b40 msgr2=0x7fe88c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 --2- 172.21.15.38:0/836310132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101b40 0x7fe88c10d4f0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe884009cd0 tx=0x7fe88402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 -- 172.21.15.38:0/836310132 shutdown_connections 2024-04-22T00:10:52.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 --2- 172.21.15.38:0/836310132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101b40 0x7fe88c10d4f0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 --2- 172.21.15.38:0/836310132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101180 0x7fe88c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 --2- 172.21.15.38:0/836310132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 -- 172.21.15.38:0/836310132 >> 172.21.15.38:0/836310132 conn(0x7fe88c0758d0 msgr2=0x7fe88c075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 -- 172.21.15.38:0/836310132 shutdown_connections 2024-04-22T00:10:52.211 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.213+0000 7fe89119a640 1 -- 172.21.15.38:0/836310132 wait complete. 2024-04-22T00:10:52.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 Processor -- start 2024-04-22T00:10:52.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 -- start start 2024-04-22T00:10:52.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101b40 0x7fe88c197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c110a40 con 0x7fe88c101180 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1108c0 con 0x7fe88c101b40 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe89119a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c110bc0 con 0x7fe88c105610 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe88ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe88ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50526/0 (socket says 172.21.15.38:50526) 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.214+0000 7fe88ad9d640 1 -- 172.21.15.38:0/2076196123 learned_addr learned my addr 172.21.15.38:0/2076196123 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88b59e640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe883fff640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101b40 0x7fe88c197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 -- 172.21.15.38:0/2076196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 msgr2=0x7fe88c19b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c19b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 -- 172.21.15.38:0/2076196123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101b40 msgr2=0x7fe88c197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101b40 0x7fe88c197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 -- 172.21.15.38:0/2076196123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88c19bea0 con 0x7fe88c101180 2024-04-22T00:10:52.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88b59e640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009850 tx=0x7fe87c009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe888d99640 1 -- 172.21.15.38:0/2076196123 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c01d070 con 0x7fe88c101180 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe888d99640 1 -- 172.21.15.38:0/2076196123 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c007c80 con 0x7fe88c101180 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88c19c190 con 0x7fe88c101180 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe888d99640 1 -- 172.21.15.38:0/2076196123 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c016320 con 0x7fe88c101180 2024-04-22T00:10:52.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.215+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88c100070 con 0x7fe88c101180 2024-04-22T00:10:52.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.217+0000 7fe888d99640 1 -- 172.21.15.38:0/2076196123 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe87c020070 con 0x7fe88c101180 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 Processor -- start 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 -- start start 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 0x55ee92d69820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ee92cc2560 con 0x55ee92d5df30 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ee92c6a730 con 0x55ee92d5e850 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe893d88740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee92cbe520 con 0x55ee92d67430 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe888d99640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 0x7fe8780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe88b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe88b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50528/0 (socket says 172.21.15.38:50528) 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe88b59e640 1 -- 172.21.15.38:0/3014461640 learned_addr learned my addr 172.21.15.38:0/3014461640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe88b59e640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 0x7fe8780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.218+0000 7fe883fff640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 0x55ee92d69820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe888d99640 1 -- 172.21.15.38:0/2076196123 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fe87c0d0d00 con 0x7fe88c101180 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 -- 172.21.15.38:0/3014461640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 msgr2=0x55ee92d69820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 0x55ee92d69820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 -- 172.21.15.38:0/3014461640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 msgr2=0x55ee92d66ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d66ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 -- 172.21.15.38:0/3014461640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee92aa07d0 con 0x55ee92d5df30 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe883fff640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 0x55ee92d69820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe884005fe0 tx=0x7fe8840328c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d113b39384af12f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe88b59e640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 0x7fe8780a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe88402c700 tx=0x7fe884041000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe8817fa640 1 -- 172.21.15.38:0/3014461640 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884050070 con 0x55ee92d5df30 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe8817fa640 1 -- 172.21.15.38:0/3014461640 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884004b40 con 0x55ee92d5df30 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.219+0000 7fe8817fa640 1 -- 172.21.15.38:0/3014461640 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884039040 con 0x55ee92d5df30 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- 172.21.15.38:0/3014461640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 msgr2=0x55ee92d5e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe884005fe0 tx=0x7fe8840328c0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- 172.21.15.38:0/3014461640 shutdown_connections 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d67430 0x55ee92d69820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d66ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 --2- 172.21.15.38:0/3014461640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5df30 0x55ee92d5e310 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- 172.21.15.38:0/3014461640 >> 172.21.15.38:0/3014461640 conn(0x55ee92d58660 msgr2=0x55ee92d58a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- 172.21.15.38:0/3014461640 shutdown_connections 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- 172.21.15.38:0/3014461640 wait complete. 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 -- start start 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.220+0000 7fe893d88740 1 Processor -- start 2024-04-22T00:10:52.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d64460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d64ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ee92d6a1c0 con 0x55ee92d5e850 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ee92c7de00 con 0x55ee92d67430 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee92d6a340 con 0x55ee92d5df30 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe893d88740 10 client.0 _ll_register_callbacks cb 0x55ee92cb8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50544/0 (socket says 172.21.15.38:50544) 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 -- 172.21.15.38:0/1932916997 learned_addr learned my addr 172.21.15.38:0/1932916997 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe88b59e640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d64ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d64460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 -- 172.21.15.38:0/1932916997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 msgr2=0x55ee92d64460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d64460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 -- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 msgr2=0x55ee92d64ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d64ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 -- 172.21.15.38:0/1932916997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee929d63f0 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe88b59e640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d64ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe883fff640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe874013590 tx=0x7fe874013a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897f1a8dc9566dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.221+0000 7fe8727fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874011a70 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fe874011c10 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe884004b40 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe874028050 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 10 client.18378 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ee929960f0 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe884004b40 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fe87401d510 con 0x55ee92d5e850 2024-04-22T00:10:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 10 client.18378 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 10 client.18378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 20 client.18378 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92d777b0 0x55ee92d79c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 4 client.18378 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.222+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee92ccfaa0 con 0x55ee92d777b0 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.223+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92d777b0 0x55ee92d79c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.223+0000 7fe8727fc640 10 client.18378 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.223+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92d777b0 0x55ee92d79c30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4239bc76b02642 server_cookie=fb8b984ba54cbe21 in_seq=0 out_seq=0 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe8727fc640 1 -- 172.21.15.38:0/1932916997 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee92ccfaa0 con 0x55ee92d777b0 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe8727fc640 10 client.18378 handle_command_reply: tid=0 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 client.18378 shutdown 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 msgr2=0x55ee92d649a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe874013590 tx=0x7fe874013a50 comp rx=0 tx=0).stop 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 shutdown_connections 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d64ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92d777b0 0x55ee92d79c30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d649a0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 --2- 172.21.15.38:0/1932916997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d64460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe8727fc640 0 client.18378 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 >> 172.21.15.38:0/1932916997 conn(0x55ee92d58660 msgr2=0x55ee92d5a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 shutdown_connections 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 1 -- 172.21.15.38:0/1932916997 wait complete. 2024-04-22T00:10:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.224+0000 7fe893d88740 20 client.18378 trim_cache size 0 max 16384 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 Processor -- start 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 -- start start 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d66b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d57560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1ac9d0 con 0x55ee92d5e850 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1ae5d0 con 0x55ee92d67430 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe893d88740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1ad0d0 con 0x55ee92d5df30 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50556/0 (socket says 172.21.15.38:50556) 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 -- 172.21.15.38:0/1389867791 learned_addr learned my addr 172.21.15.38:0/1389867791 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d66b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe88b59e640 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d57560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 -- 172.21.15.38:0/1389867791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 msgr2=0x55ee92d66b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d66b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 -- 172.21.15.38:0/1389867791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 msgr2=0x55ee92d57560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.242+0000 7fe883fff640 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d57560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe883fff640 1 -- 172.21.15.38:0/1389867791 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee92d69820 con 0x55ee92d5e850 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe883fff640 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe87400a010 tx=0x7fe87401eff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416d660e64d9f3c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe880ff9640 1 -- 172.21.15.38:0/1389867791 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87401dd70 con 0x55ee92d5e850 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe880ff9640 1 -- 172.21.15.38:0/1389867791 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874005590 con 0x55ee92d5e850 2024-04-22T00:10:52.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe880ff9640 1 -- 172.21.15.38:0/1389867791 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874011e40 con 0x55ee92d5e850 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- 172.21.15.38:0/1389867791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 msgr2=0x55ee92d73080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe87400a010 tx=0x7fe87401eff0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- 172.21.15.38:0/1389867791 shutdown_connections 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d67430 0x55ee92d57560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d5e850 0x55ee92d73080 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- 172.21.15.38:0/1389867791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- 172.21.15.38:0/1389867791 >> 172.21.15.38:0/1389867791 conn(0x55ee92d58660 msgr2=0x55ee92d56e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- 172.21.15.38:0/1389867791 shutdown_connections 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- 172.21.15.38:0/1389867791 wait complete. 2024-04-22T00:10:52.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- start start 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 Processor -- start 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d6fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d60f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1adf00 con 0x55ee92d67430 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1ad7d0 con 0x55ee92d5e850 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe88c1aa9d0 con 0x55ee92d5df30 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe893d88740 10 client.0 _ll_register_callbacks cb 0x7fe874009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe88b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe88b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50572/0 (socket says 172.21.15.38:50572) 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.243+0000 7fe88b59e640 1 -- 172.21.15.38:0/3683783926 learned_addr learned my addr 172.21.15.38:0/3683783926 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d6fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe883fff640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d60f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 -- 172.21.15.38:0/3683783926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 msgr2=0x55ee92d6fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d6fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 -- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 msgr2=0x55ee92d60f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d60f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 -- 172.21.15.38:0/3683783926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee92d61cb0 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe883fff640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d60f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe88b59e640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe88402f020 tx=0x7fe8840044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbacb24bdf2bdfca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe8727fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884059060 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fe8840045d0 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe874005590 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe884050040 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe893d88740 10 client.18384 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ee92ddde20 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.244+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874005590 con 0x55ee92d67430 2024-04-22T00:10:52.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fe88404a070 con 0x55ee92d67430 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 10 client.18384 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 10 client.18384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 20 client.18384 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92dde150 0x55ee92de0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 4 client.18384 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee92cc9df0 con 0x55ee92dde150 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92dde150 0x55ee92de0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe8727fc640 10 client.18384 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe88ad9d640 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92dde150 0x55ee92de0540 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b09346f867f9ab5 server_cookie=da62f8d9c5f6ed84 in_seq=0 out_seq=0 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe8727fc640 1 -- 172.21.15.38:0/3683783926 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ee92cc9df0 con 0x55ee92dde150 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.245+0000 7fe8727fc640 10 client.18384 handle_command_reply: tid=0 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 client.18384 shutdown 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 msgr2=0x55ee92d61480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe88402f020 tx=0x7fe8840044e0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 shutdown_connections 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ee92dde150 0x55ee92de0540 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee92d67430 0x55ee92d61480 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee92d5e850 0x55ee92d60f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 --2- 172.21.15.38:0/3683783926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee92d5df30 0x55ee92d6fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe8727fc640 0 client.18384 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 >> 172.21.15.38:0/3683783926 conn(0x55ee92d58660 msgr2=0x55ee92d64c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 shutdown_connections 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 1 -- 172.21.15.38:0/3683783926 wait complete. 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe893d88740 20 client.18384 trim_cache size 0 max 16384 2024-04-22T00:10:52.250 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:52.251 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:10:52.251 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:10:52.251 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 msgr2=0x7fe8780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 0x7fe8780a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe88402c700 tx=0x7fe884041000 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 msgr2=0x7fe88c196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009850 tx=0x7fe87c009d10 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 shutdown_connections 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe8780a2d00 0x7fe8780a51c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe88c105610 0x7fe88c19b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88c101b40 0x7fe88c197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 --2- 172.21.15.38:0/2076196123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe88c101180 0x7fe88c196f00 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 >> 172.21.15.38:0/2076196123 conn(0x7fe88c0758d0 msgr2=0x7fe88c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 shutdown_connections 2024-04-22T00:10:52.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.246+0000 7fe89119a640 1 -- 172.21.15.38:0/2076196123 wait complete. 2024-04-22T00:10:52.255 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:10:52.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:10:52.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 -- 172.21.15.38:0/2223076504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0102a20 msgr2=0x7f9da0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 --2- 172.21.15.38:0/2223076504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0102a20 0x7f9da0102e00 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009900 tx=0x7f9d8802bcd0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 -- 172.21.15.38:0/2223076504 shutdown_connections 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 --2- 172.21.15.38:0/2223076504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da010bd60 0x7f9da010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 --2- 172.21.15.38:0/2223076504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da0103340 0x7f9da010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 --2- 172.21.15.38:0/2223076504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0102a20 0x7f9da0102e00 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 -- 172.21.15.38:0/2223076504 >> 172.21.15.38:0/2223076504 conn(0x7f9da00777a0 msgr2=0x7f9da0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 -- 172.21.15.38:0/2223076504 shutdown_connections 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.360+0000 7f9da64c9640 1 -- 172.21.15.38:0/2223076504 wait complete. 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 Processor -- start 2024-04-22T00:10:52.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 -- start start 2024-04-22T00:10:52.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da0102a20 0x7f9da0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da010bd60 0x7f9da019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9da0111b30 con 0x7f9da0103340 2024-04-22T00:10:52.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9da01119b0 con 0x7f9da0102a20 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9da64c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9da0111cb0 con 0x7f9da010bd60 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.361+0000 7f9d9f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50596/0 (socket says 172.21.15.38:50596) 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 -- 172.21.15.38:0/1198118563 learned_addr learned my addr 172.21.15.38:0/1198118563 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9da4a66640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da010bd60 0x7f9da019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 -- 172.21.15.38:0/1198118563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da010bd60 msgr2=0x7f9da019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da010bd60 0x7f9da019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 -- 172.21.15.38:0/1198118563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da0102a20 msgr2=0x7f9da0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da0102a20 0x7f9da0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 -- 172.21.15.38:0/1198118563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da019cfc0 con 0x7f9da0103340 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9f7fe640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000d770 tx=0x7f9d9000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90017070 con 0x7f9da0103340 2024-04-22T00:10:52.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d90005500 con 0x7f9da0103340 2024-04-22T00:10:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da019d2b0 con 0x7f9da0103340 2024-04-22T00:10:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d90004e00 con 0x7f9da0103340 2024-04-22T00:10:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.362+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da01a4b40 con 0x7f9da0103340 2024-04-22T00:10:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.363+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da00689b0 con 0x7f9da0103340 2024-04-22T00:10:52.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.364+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9d90004fa0 con 0x7f9da0103340 2024-04-22T00:10:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.365+0000 7f9d9d7fa640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 0x7f9d8c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.365+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9d900cf2a0 con 0x7f9da0103340 2024-04-22T00:10:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.365+0000 7f9d9ffff640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 0x7f9d8c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.365+0000 7f9d9ffff640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 0x7f9d8c0a5240 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d880096c0 tx=0x7f9d8802cf90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.369+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9d9009bb30 con 0x7f9da0103340 2024-04-22T00:10:52.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.594+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9da002d170 con 0x7f9da0103340 2024-04-22T00:10:52.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.602+0000 7f9d9d7fa640 1 -- 172.21.15.38:0/1198118563 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11941 (secure 0 0 0) 0x7f9d9009b430 con 0x7f9da0103340 2024-04-22T00:10:52.599 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:10:52.600 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:09:42.375340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:10:52.600 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 38 2024-04-22T00:10:52.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 msgr2=0x7f9d8c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 0x7f9d8c0a5240 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d880096c0 tx=0x7f9d8802cf90 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 msgr2=0x7f9da0198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000d770 tx=0x7f9d9000dc30 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 shutdown_connections 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9d8c0a2d80 0x7f9d8c0a5240 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9da010bd60 0x7f9da019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9da0103340 0x7f9da0198510 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 --2- 172.21.15.38:0/1198118563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da0102a20 0x7f9da0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 >> 172.21.15.38:0/1198118563 conn(0x7f9da00777a0 msgr2=0x7f9da00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 shutdown_connections 2024-04-22T00:10:52.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.604+0000 7f9da64c9640 1 -- 172.21.15.38:0/1198118563 wait complete. 2024-04-22T00:10:52.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 -- 172.21.15.38:0/1212958054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 msgr2=0x7fc47c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 --2- 172.21.15.38:0/1212958054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c10b7c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc470009900 tx=0x7fc47002bd60 comp rx=0 tx=0).stop 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 -- 172.21.15.38:0/1212958054 shutdown_connections 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 --2- 172.21.15.38:0/1212958054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c10bd00 0x7fc47c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 --2- 172.21.15.38:0/1212958054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c10b7c0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 --2- 172.21.15.38:0/1212958054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c1029c0 0x7fc47c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 -- 172.21.15.38:0/1212958054 >> 172.21.15.38:0/1212958054 conn(0x7fc47c077740 msgr2=0x7fc47c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 -- 172.21.15.38:0/1212958054 shutdown_connections 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.720+0000 7fc480a27640 1 -- 172.21.15.38:0/1212958054 wait complete. 2024-04-22T00:10:52.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 Processor -- start 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 -- start start 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c1029c0 0x7fc47c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 0x7fc47c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c111b50 con 0x7fc47c1032e0 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1119d0 con 0x7fc47c1029c0 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc480a27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c111cd0 con 0x7fc47c10bd00 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50606/0 (socket says 172.21.15.38:50606) 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 -- 172.21.15.38:0/3485915723 learned_addr learned my addr 172.21.15.38:0/3485915723 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc47a59c640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c1029c0 0x7fc47c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 0x7fc47c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 -- 172.21.15.38:0/3485915723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 msgr2=0x7fc47c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 0x7fc47c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 -- 172.21.15.38:0/3485915723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c1029c0 msgr2=0x7fc47c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c1029c0 0x7fc47c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 -- 172.21.15.38:0/3485915723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47c19d040 con 0x7fc47c1032e0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 0x7fc47c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.721+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc470009a20 tx=0x7fc47002dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.722+0000 7fc46b7fe640 1 -- 172.21.15.38:0/3485915723 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470043020 con 0x7fc47c1032e0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.722+0000 7fc46b7fe640 1 -- 172.21.15.38:0/3485915723 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4700054e0 con 0x7fc47c1032e0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.722+0000 7fc46b7fe640 1 -- 172.21.15.38:0/3485915723 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470003bd0 con 0x7fc47c1032e0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.722+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47c19d2d0 con 0x7fc47c1032e0 2024-04-22T00:10:52.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.722+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47c1a4bc0 con 0x7fc47c1032e0 2024-04-22T00:10:52.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.724+0000 7fc46b7fe640 1 -- 172.21.15.38:0/3485915723 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc470046020 con 0x7fc47c1032e0 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.724+0000 7fc483610740 1 Processor -- start 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.724+0000 7fc483610740 1 -- start start 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de43ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de442910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4de39b530 con 0x55b4de437020 2024-04-22T00:10:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4de343830 con 0x55b4de437940 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc483610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b4de3974f0 con 0x55b4de440520 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc46b7fe640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 0x7fc4640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50608/0 (socket says 172.21.15.38:50608) 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 -- 172.21.15.38:0/653959884 learned_addr learned my addr 172.21.15.38:0/653959884 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 0x7fc4640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc46b7fe640 1 -- 172.21.15.38:0/3485915723 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fc4700f18f0 con 0x7fc47c1032e0 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47a59c640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de43ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc479d9b640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de442910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 -- 172.21.15.38:0/653959884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 msgr2=0x55b4de442910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de442910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 -- 172.21.15.38:0/653959884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 msgr2=0x55b4de43ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de43ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 -- 172.21.15.38:0/653959884 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4de3672c0 con 0x55b4de437020 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47a59c640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de43ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc474013de0 tx=0x7fc474010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bdb376e9ad1ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.725+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 0x7fc4640a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc474010c40 tx=0x7fc474015000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc468ff9640 1 -- 172.21.15.38:0/653959884 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474010e50 con 0x55b4de437020 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc468ff9640 1 -- 172.21.15.38:0/653959884 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc474013740 con 0x55b4de437020 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc468ff9640 1 -- 172.21.15.38:0/653959884 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474005420 con 0x55b4de437020 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 -- 172.21.15.38:0/653959884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 msgr2=0x55b4de437400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fc474013de0 tx=0x7fc474010740 comp rx=0 tx=0).stop 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 -- 172.21.15.38:0/653959884 shutdown_connections 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de442910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de43ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 --2- 172.21.15.38:0/653959884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de437400 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 -- 172.21.15.38:0/653959884 >> 172.21.15.38:0/653959884 conn(0x55b4de431750 msgr2=0x55b4de431b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 -- 172.21.15.38:0/653959884 shutdown_connections 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.726+0000 7fc483610740 1 -- 172.21.15.38:0/653959884 wait complete. 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 -- start start 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 Processor -- start 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 0x55b4de43d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 0x55b4de43d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4de4432b0 con 0x55b4de437020 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b4de356f10 con 0x55b4de440520 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b4de443430 con 0x55b4de437940 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc483610740 10 client.0 _ll_register_callbacks cb 0x55b4de0aebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50622/0 (socket says 172.21.15.38:50622) 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 -- 172.21.15.38:0/2672104052 learned_addr learned my addr 172.21.15.38:0/2672104052 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc479d9b640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 0x55b4de43d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 0x55b4de43d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 -- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 msgr2=0x55b4de43d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 0x55b4de43d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 -- 172.21.15.38:0/2672104052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 msgr2=0x55b4de43d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 0x55b4de43d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47a59c640 1 -- 172.21.15.38:0/2672104052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4de328c80 con 0x55b4de437020 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc479d9b640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 0x55b4de43d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.727+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 0x55b4de43d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc47a59c640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc46c013790 tx=0x7fc46c013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c5c4058a7d35dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c020060 con 0x55b4de437020 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fc46c011dd0 con 0x55b4de437020 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc474013740 con 0x55b4de437020 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc46c004020 con 0x55b4de437020 2024-04-22T00:10:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 10 client.18402 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b4de0af3f0 con 0x55b4de437020 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc474013740 con 0x55b4de437020 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc46c01a030 con 0x55b4de437020 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 10 client.18402 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 10 client.18402 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 20 client.18402 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de450600 0x55b4de452a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 4 client.18402 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.728+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4de3a8a70 con 0x55b4de450600 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.729+0000 7fc479d9b640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de450600 0x55b4de452a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.729+0000 7fc461ffb640 10 client.18402 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.729+0000 7fc479d9b640 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de450600 0x55b4de452a60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=33294bea6bc9a910 server_cookie=a0b2a1d65c75fbe2 in_seq=0 out_seq=0 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc461ffb640 1 -- 172.21.15.38:0/2672104052 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4de3a8a70 con 0x55b4de450600 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc461ffb640 10 client.18402 handle_command_reply: tid=0 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 client.18402 shutdown 2024-04-22T00:10:52.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 msgr2=0x55b4de43fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fc46c013790 tx=0x7fc46c013c50 comp rx=0 tx=0).stop 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 shutdown_connections 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de450600 0x55b4de452a60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de440520 0x55b4de43d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc461ffb640 0 client.18402 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de437940 0x55b4de43d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 --2- 172.21.15.38:0/2672104052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de43fee0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 >> 172.21.15.38:0/2672104052 conn(0x55b4de431750 msgr2=0x55b4de4334a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 shutdown_connections 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 1 -- 172.21.15.38:0/2672104052 wait complete. 2024-04-22T00:10:52.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.730+0000 7fc483610740 20 client.18402 trim_cache size 0 max 16384 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 Processor -- start 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 -- start start 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 0x55b4de44c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.747 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1aeb60 con 0x55b4de437940 2024-04-22T00:10:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1b0760 con 0x55b4de437020 2024-04-22T00:10:52.748 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc483610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1af260 con 0x55b4de440520 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50638/0 (socket says 172.21.15.38:50638) 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 -- 172.21.15.38:0/1752419657 learned_addr learned my addr 172.21.15.38:0/1752419657 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc47a59c640 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 0x55b4de44c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 -- 172.21.15.38:0/1752419657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 msgr2=0x55b4de43f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 -- 172.21.15.38:0/1752419657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 msgr2=0x55b4de44c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 0x55b4de44c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 -- 172.21.15.38:0/1752419657 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4de1797d0 con 0x55b4de437940 2024-04-22T00:10:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc47a59c640 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 0x55b4de44c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.744+0000 7fc479d9b640 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fc4700bdbe0 tx=0x7fc470041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42265a73bf608a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc463fff640 1 -- 172.21.15.38:0/1752419657 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4700db070 con 0x55b4de437940 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc463fff640 1 -- 172.21.15.38:0/1752419657 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4700d6e50 con 0x55b4de437940 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc463fff640 1 -- 172.21.15.38:0/1752419657 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4700d3040 con 0x55b4de437940 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- 172.21.15.38:0/1752419657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 msgr2=0x55b4de430850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fc4700bdbe0 tx=0x7fc470041040 comp rx=0 tx=0).stop 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- 172.21.15.38:0/1752419657 shutdown_connections 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437940 0x55b4de430850 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 --2- 172.21.15.38:0/1752419657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437020 0x55b4de44c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- 172.21.15.38:0/1752419657 >> 172.21.15.38:0/1752419657 conn(0x55b4de431750 msgr2=0x55b4de43c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- 172.21.15.38:0/1752419657 shutdown_connections 2024-04-22T00:10:52.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- 172.21.15.38:0/1752419657 wait complete. 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.745+0000 7fc483610740 1 -- start start 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 Processor -- start 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de439ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1b0090 con 0x55b4de437020 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1af960 con 0x55b4de437940 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1acb60 con 0x55b4de440520 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc483610740 10 client.0 _ll_register_callbacks cb 0x7fc46c009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50654/0 (socket says 172.21.15.38:50654) 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 -- 172.21.15.38:0/3705537638 learned_addr learned my addr 172.21.15.38:0/3705537638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de439ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 -- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 msgr2=0x55b4de43a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 -- 172.21.15.38:0/3705537638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 msgr2=0x55b4de439ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de439ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 -- 172.21.15.38:0/3705537638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4de43a560 con 0x55b4de437020 2024-04-22T00:10:52.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47ad9d640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de439ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc47a59c640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00f570 tx=0x7fc46c004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c94573a2b848561 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.746+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c02f020 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fc46c00f9e0 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4700d6e50 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc46c037050 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 10 client.18408 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b4de4b6940 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4700d6e50 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc46c01a070 con 0x55b4de437020 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 10 client.18408 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 10 client.18408 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 20 client.18408 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de4b71c0 0x55b4de4b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 4 client.18408 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4de3a2dc0 con 0x55b4de4b71c0 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.747+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de4b71c0 0x55b4de4b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc479d9b640 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de4b71c0 0x55b4de4b95e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6bdf7fbb0cf953f3 server_cookie=286c3b9111c517ad in_seq=0 out_seq=0 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc461ffb640 10 client.18408 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc461ffb640 1 -- 172.21.15.38:0/3705537638 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4de3a2dc0 con 0x55b4de4b71c0 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc461ffb640 10 client.18408 handle_command_reply: tid=0 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 client.18408 shutdown 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 msgr2=0x55b4de4488a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc46c00f570 tx=0x7fc46c004240 comp rx=0 tx=0).stop 2024-04-22T00:10:52.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 shutdown_connections 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55b4de4b71c0 0x55b4de4b95e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4de440520 0x55b4de43a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4de437940 0x55b4de439ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc461ffb640 0 client.18408 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 --2- 172.21.15.38:0/3705537638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4de437020 0x55b4de4488a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 >> 172.21.15.38:0/3705537638 conn(0x55b4de431750 msgr2=0x55b4de430850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 shutdown_connections 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 1 -- 172.21.15.38:0/3705537638 wait complete. 2024-04-22T00:10:52.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.748+0000 7fc483610740 20 client.18408 trim_cache size 0 max 16384 2024-04-22T00:10:52.756 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 msgr2=0x7fc4640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 0x7fc4640a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc474010c40 tx=0x7fc474015000 comp rx=0 tx=0).stop 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 msgr2=0x7fc47c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc470009a20 tx=0x7fc47002dd30 comp rx=0 tx=0).stop 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 shutdown_connections 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc4640a2d80 0x7fc4640a5240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc47c10bd00 0x7fc47c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc47c1032e0 0x7fc47c198590 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 --2- 172.21.15.38:0/3485915723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc47c1029c0 0x7fc47c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 >> 172.21.15.38:0/3485915723 conn(0x7fc47c077740 msgr2=0x7fc47c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 shutdown_connections 2024-04-22T00:10:52.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.749+0000 7fc480a27640 1 -- 172.21.15.38:0/3485915723 wait complete. 2024-04-22T00:10:52.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:10:52.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 -- 172.21.15.38:0/2671159000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34106690 msgr2=0x7f8f34106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/2671159000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34106690 0x7f8f34106a70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009900 tx=0x7f8f3002bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:52.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 -- 172.21.15.38:0/2671159000 shutdown_connections 2024-04-22T00:10:52.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/2671159000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f3410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/2671159000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34100990 0x7f8f34100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/2671159000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34106690 0x7f8f34106a70 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 -- 172.21.15.38:0/2671159000 >> 172.21.15.38:0/2671159000 conn(0x7f8f34076440 msgr2=0x7f8f34076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 -- 172.21.15.38:0/2671159000 shutdown_connections 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.865+0000 7f8f3c43b640 1 -- 172.21.15.38:0/2671159000 wait complete. 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 Processor -- start 2024-04-22T00:10:52.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 -- start start 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f341983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 0x7f8f3419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34111b20 con 0x7f8f34100990 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341119a0 con 0x7f8f34101330 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3c43b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34111ca0 con 0x7f8f34106690 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50674/0 (socket says 172.21.15.38:50674) 2024-04-22T00:10:52.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/1063182503 learned_addr learned my addr 172.21.15.38:0/1063182503 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 0x7f8f3419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f399d6640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f341983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/1063182503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 msgr2=0x7f8f3419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 0x7f8f3419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/1063182503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 msgr2=0x7f8f341983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f341983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/1063182503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3419ce90 con 0x7f8f34100990 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f399d6640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f341983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.866+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 0x7f8f3419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8f3002c240 tx=0x7f8f300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f2b7fe640 1 -- 172.21.15.38:0/1063182503 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30043020 con 0x7f8f34100990 2024-04-22T00:10:52.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f2b7fe640 1 -- 172.21.15.38:0/1063182503 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f300054e0 con 0x7f8f34100990 2024-04-22T00:10:52.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3419d120 con 0x7f8f34100990 2024-04-22T00:10:52.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f2b7fe640 1 -- 172.21.15.38:0/1063182503 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30003b30 con 0x7f8f34100990 2024-04-22T00:10:52.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.867+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f341a4a10 con 0x7f8f34100990 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.869+0000 7f8f2b7fe640 1 -- 172.21.15.38:0/1063182503 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8f30046020 con 0x7f8f34100990 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 Processor -- start 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 -- start start 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 0x55ee5e490fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e491520 0x55ee5e493910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e3ec530 con 0x55ee5e488020 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e394830 con 0x55ee5e488940 2024-04-22T00:10:52.875 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3f019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e3e84f0 con 0x55ee5e491520 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f2b7fe640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 0x7f8f200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3a9d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3a9d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50682/0 (socket says 172.21.15.38:50682) 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/2604208882 learned_addr learned my addr 172.21.15.38:0/2604208882 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 0x7f8f200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f399d6640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e491520 0x55ee5e493910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f2b7fe640 1 -- 172.21.15.38:0/1063182503 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8f300f17d0 con 0x7f8f34100990 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.870+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 0x55ee5e490fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/2604208882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e491520 msgr2=0x55ee5e493910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e491520 0x55ee5e493910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/2604208882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 msgr2=0x55ee5e490fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.876 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 0x55ee5e490fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/2604208882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5e3b82c0 con 0x55ee5e488020 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 0x55ee5e490fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c012e10 tx=0x7f8f2c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7b6c43a417ecc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 0x7f8f200a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c010ca0 tx=0x7f8f2c01d000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f28bfe640 1 -- 172.21.15.38:0/2604208882 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0128f0 con 0x55ee5e488020 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f28bfe640 1 -- 172.21.15.38:0/2604208882 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c009d80 con 0x55ee5e488020 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.871+0000 7f8f28bfe640 1 -- 172.21.15.38:0/2604208882 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0056c0 con 0x55ee5e488020 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- 172.21.15.38:0/2604208882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 msgr2=0x55ee5e488400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c012e10 tx=0x7f8f2c0107a0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- 172.21.15.38:0/2604208882 shutdown_connections 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e491520 0x55ee5e493910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488940 0x55ee5e490fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 --2- 172.21.15.38:0/2604208882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e488400 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- 172.21.15.38:0/2604208882 >> 172.21.15.38:0/2604208882 conn(0x55ee5e482750 msgr2=0x55ee5e482b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- 172.21.15.38:0/2604208882 shutdown_connections 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- 172.21.15.38:0/2604208882 wait complete. 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 -- start start 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.872+0000 7f8f3f019740 1 Processor -- start 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488020 0x55ee5e490f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e494250 con 0x55ee5e488940 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e3a7f10 con 0x55ee5e491520 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee5e4943d0 con 0x55ee5e488020 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 10 client.0 _ll_register_callbacks cb 0x55ee5e0ffbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50696/0 (socket says 172.21.15.38:50696) 2024-04-22T00:10:52.878 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 -- 172.21.15.38:0/3684792507 learned_addr learned my addr 172.21.15.38:0/3684792507 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488020 0x55ee5e490f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 -- 172.21.15.38:0/3684792507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488020 msgr2=0x55ee5e490f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488020 0x55ee5e490f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 -- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 msgr2=0x55ee5e48e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 -- 172.21.15.38:0/3684792507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5e379c80 con 0x55ee5e488940 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f8f24010820 tx=0x7f8f24010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50911d9f8c5847f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2401d060 con 0x55ee5e488940 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f2c009d80 con 0x55ee5e488940 2024-04-22T00:10:52.879 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.873+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8f2400eda0 con 0x55ee5e488940 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8f24024050 con 0x55ee5e488940 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 10 client.18420 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ee5e1003f0 con 0x55ee5e488940 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c009d80 con 0x55ee5e488940 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f8f240164f0 con 0x55ee5e488940 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 10 client.18420 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 10 client.18420 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 20 client.18420 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e4a1670 0x55ee5e4a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 4 client.18420 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee5e3f9a70 con 0x55ee5e4a1670 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.874+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e4a1670 0x55ee5e4a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.880 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.875+0000 7f8f1dffb640 10 client.18420 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.875+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e4a1670 0x55ee5e4a3ab0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa92e2fc98a0e74c server_cookie=81a2ba6dbe723cab in_seq=0 out_seq=0 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.875+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3684792507 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee5e3f9a70 con 0x55ee5e4a1670 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.875+0000 7f8f1dffb640 10 client.18420 handle_command_reply: tid=0 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 client.18420 shutdown 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 msgr2=0x55ee5e48e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f8f24010820 tx=0x7f8f24010c20 comp rx=0 tx=0).stop 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 shutdown_connections 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e4a1670 0x55ee5e4a3ab0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488940 0x55ee5e48e2c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3684792507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488020 0x55ee5e490f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f1dffb640 0 client.18420 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:52.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 >> 172.21.15.38:0/3684792507 conn(0x55ee5e482750 msgr2=0x55ee5e4844a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 shutdown_connections 2024-04-22T00:10:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 1 -- 172.21.15.38:0/3684792507 wait complete. 2024-04-22T00:10:52.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.876+0000 7f8f3f019740 20 client.18420 trim_cache size 0 max 16384 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.889+0000 7f8f3f019740 1 Processor -- start 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.889+0000 7f8f3f019740 1 -- start start 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.889+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488020 0x55ee5e49d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.889+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e481850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3f019740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341aea00 con 0x55ee5e491520 2024-04-22T00:10:52.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3f019740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341b0600 con 0x55ee5e488020 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3f019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341af100 con 0x55ee5e488940 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50710/0 (socket says 172.21.15.38:50710) 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/347722026 learned_addr learned my addr 172.21.15.38:0/347722026 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f399d6640 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e481850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488020 0x55ee5e49d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/347722026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 msgr2=0x55ee5e481850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e481850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/347722026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488020 msgr2=0x55ee5e49d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488020 0x55ee5e49d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 -- 172.21.15.38:0/347722026 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5e48d270 con 0x55ee5e491520 2024-04-22T00:10:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c028040 tx=0x7f8f2c009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7238c14a8f52bc71 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f1ffff640 1 -- 172.21.15.38:0/347722026 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c033020 con 0x55ee5e491520 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f1ffff640 1 -- 172.21.15.38:0/347722026 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c00dd50 con 0x55ee5e491520 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.890+0000 7f8f1ffff640 1 -- 172.21.15.38:0/347722026 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0122a0 con 0x55ee5e491520 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- 172.21.15.38:0/347722026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 msgr2=0x55ee5e4909f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c028040 tx=0x7f8f2c009a00 comp rx=0 tx=0).stop 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- 172.21.15.38:0/347722026 shutdown_connections 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e491520 0x55ee5e4909f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e481850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 --2- 172.21.15.38:0/347722026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e488020 0x55ee5e49d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- 172.21.15.38:0/347722026 >> 172.21.15.38:0/347722026 conn(0x55ee5e482750 msgr2=0x55ee5e482b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- 172.21.15.38:0/347722026 shutdown_connections 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- 172.21.15.38:0/347722026 wait complete. 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 -- start start 2024-04-22T00:10:52.895 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.891+0000 7f8f3f019740 1 Processor -- start 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e48b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341aff30 con 0x55ee5e488020 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341af800 con 0x55ee5e491520 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341aca00 con 0x55ee5e488940 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 10 client.0 _ll_register_callbacks cb 0x7f8f24010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50718/0 (socket says 172.21.15.38:50718) 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/3676520165 learned_addr learned my addr 172.21.15.38:0/3676520165 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e48b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/3676520165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 msgr2=0x55ee5e48b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e48b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 msgr2=0x55ee5e48bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 -- 172.21.15.38:0/3676520165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5e48ec40 con 0x55ee5e488020 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a9d8640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3a1d7640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8f300ba440 tx=0x7f8f3002ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0fe1d4a92ab6bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3003c350 con 0x55ee5e488020 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8f300eee10 con 0x55ee5e488020 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f2c00dd50 con 0x55ee5e488020 2024-04-22T00:10:52.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8f30087050 con 0x55ee5e488020 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 10 client.18426 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.892+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ee5e1ca7d0 con 0x55ee5e488020 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c00dd50 con 0x55ee5e488020 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f8f300ba0e0 con 0x55ee5e488020 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 10 client.18426 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 10 client.18426 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 20 client.18426 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e508480 0x55ee5e50a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 4 client.18426 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee5e3f3dc0 con 0x55ee5e508480 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e508480 0x55ee5e50a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f1dffb640 10 client.18426 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:52.898 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.893+0000 7f8f399d6640 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e508480 0x55ee5e50a8a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1c9d3bb6bb0246e7 server_cookie=80ef074f15d7d9a2 in_seq=0 out_seq=0 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f1dffb640 1 -- 172.21.15.38:0/3676520165 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee5e3f3dc0 con 0x55ee5e508480 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f1dffb640 10 client.18426 handle_command_reply: tid=0 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 client.18426 shutdown 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 msgr2=0x55ee5e48b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8f300ba440 tx=0x7f8f3002ca90 comp rx=0 tx=0).stop 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 shutdown_connections 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ee5e491520 0x55ee5e48bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee5e488940 0x55ee5e48b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55ee5e508480 0x55ee5e50a8a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 --2- 172.21.15.38:0/3676520165 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ee5e488020 0x55ee5e48b130 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 >> 172.21.15.38:0/3676520165 conn(0x55ee5e482750 msgr2=0x55ee5e481850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f1dffb640 0 client.18426 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 shutdown_connections 2024-04-22T00:10:52.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 1 -- 172.21.15.38:0/3676520165 wait complete. 2024-04-22T00:10:52.900 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3f019740 20 client.18426 trim_cache size 0 max 16384 2024-04-22T00:10:52.900 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 msgr2=0x7f8f200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 0x7f8f200a51c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c010ca0 tx=0x7f8f2c01d000 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 msgr2=0x7f8f34197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8f3002c240 tx=0x7f8f300042f0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 shutdown_connections 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f200a2d00 0x7f8f200a51c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f34106690 0x7f8f3419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f34101330 0x7f8f341983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 --2- 172.21.15.38:0/1063182503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f34100990 0x7f8f34197ea0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 >> 172.21.15.38:0/1063182503 conn(0x7f8f34076440 msgr2=0x7f8f340ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 shutdown_connections 2024-04-22T00:10:52.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:52.894+0000 7f8f3c43b640 1 -- 172.21.15.38:0/1063182503 wait complete. 2024-04-22T00:10:52.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:52 smithi038 ceph-mon[27972]: pgmap v270: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:52.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:10:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 -- 172.21.15.38:0/386541202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c1077a0 msgr2=0x7fbd2c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 --2- 172.21.15.38:0/386541202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c107b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009930 tx=0x7fbd2002bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 -- 172.21.15.38:0/386541202 shutdown_connections 2024-04-22T00:10:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 --2- 172.21.15.38:0/386541202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 0x7fbd2c10e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 --2- 172.21.15.38:0/386541202 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c101790 0x7fbd2c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 --2- 172.21.15.38:0/386541202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c107b80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 -- 172.21.15.38:0/386541202 >> 172.21.15.38:0/386541202 conn(0x7fbd2c0fd2a0 msgr2=0x7fbd2c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 -- 172.21.15.38:0/386541202 shutdown_connections 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.010+0000 7fbd3158a640 1 -- 172.21.15.38:0/386541202 wait complete. 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 Processor -- start 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 -- start start 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c101790 0x7fbd2c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 0x7fbd2c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c111b70 con 0x7fbd2c1077a0 2024-04-22T00:10:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1119f0 con 0x7fbd2c102150 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd3158a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c111cf0 con 0x7fbd2c101790 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50738/0 (socket says 172.21.15.38:50738) 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4129207772 learned_addr learned my addr 172.21.15.38:0/4129207772 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2affd640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c101790 0x7fbd2c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 0x7fbd2c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4129207772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c101790 msgr2=0x7fbd2c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c101790 0x7fbd2c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4129207772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 msgr2=0x7fbd2c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 0x7fbd2c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.011+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4129207772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2c1a6600 con 0x7fbd2c1077a0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400efd0 tx=0x7fbd2400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd13fff640 1 -- 172.21.15.38:0/4129207772 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd24003b70 con 0x7fbd2c1077a0 2024-04-22T00:10:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd2c1a68f0 con 0x7fbd2c1077a0 2024-04-22T00:10:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd13fff640 1 -- 172.21.15.38:0/4129207772 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd24003d10 con 0x7fbd2c1077a0 2024-04-22T00:10:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd13fff640 1 -- 172.21.15.38:0/4129207772 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240109f0 con 0x7fbd2c1077a0 2024-04-22T00:10:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.012+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd2c1a6e40 con 0x7fbd2c1077a0 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.014+0000 7fbd13fff640 1 -- 172.21.15.38:0/4129207772 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fbd24004020 con 0x7fbd2c1077a0 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 Processor -- start 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 -- start start 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a79114d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd13fff640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 0x7fbd140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791158910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a7910b1530 con 0x55a79114d940 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a791059830 con 0x55a79114d020 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd34173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a7910ad4f0 con 0x55a791156520 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 0x7fbd140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd13fff640 1 -- 172.21.15.38:0/4129207772 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fbd240cfba0 con 0x7fbd2c1077a0 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50754/0 (socket says 172.21.15.38:50754) 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2a7fc640 1 -- 172.21.15.38:0/3661065188 learned_addr learned my addr 172.21.15.38:0/3661065188 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a79114d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 -- 172.21.15.38:0/3661065188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 msgr2=0x55a791158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 -- 172.21.15.38:0/3661065188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 msgr2=0x55a791155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2affd640 1 -- 172.21.15.38:0/3661065188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79107d2c0 con 0x55a79114d020 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.015+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.016+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 0x7fbd140a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c012980 tx=0x7fbd1c00bb80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.016+0000 7fbd2affd640 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a79114d400 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009230 tx=0x7fbd20005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0b847d1620901d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.016+0000 7fbd117fa640 1 -- 172.21.15.38:0/3661065188 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20042070 con 0x55a79114d020 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.016+0000 7fbd117fa640 1 -- 172.21.15.38:0/3661065188 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd2002c690 con 0x55a79114d020 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.016+0000 7fbd117fa640 1 -- 172.21.15.38:0/3661065188 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20004c20 con 0x55a79114d020 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- 172.21.15.38:0/3661065188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 msgr2=0x55a79114d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a79114d400 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009230 tx=0x7fbd20005bf0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- 172.21.15.38:0/3661065188 shutdown_connections 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- 172.21.15.38:0/3661065188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a79114d400 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- 172.21.15.38:0/3661065188 >> 172.21.15.38:0/3661065188 conn(0x55a791147750 msgr2=0x55a791147b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- 172.21.15.38:0/3661065188 shutdown_connections 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- 172.21.15.38:0/3661065188 wait complete. 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- start start 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 Processor -- start 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a791155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791153800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a7911592b0 con 0x55a79114d940 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55a79106cf10 con 0x55a79114d020 2024-04-22T00:10:53.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a791159430 con 0x55a791156520 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd34173740 10 client.0 _ll_register_callbacks cb 0x55a790dc4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd2a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd2a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50756/0 (socket says 172.21.15.38:50756) 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.017+0000 7fbd2a7fc640 1 -- 172.21.15.38:0/1859881351 learned_addr learned my addr 172.21.15.38:0/1859881351 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2affd640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a791155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 -- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 msgr2=0x55a791153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 -- 172.21.15.38:0/1859881351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 msgr2=0x55a791155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a791155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 -- 172.21.15.38:0/1859881351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79103ec80 con 0x55a79114d940 2024-04-22T00:10:53.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00bfa0 tx=0x7fbd1c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e456251e1601b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c01ec30 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fbd1c01edd0 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7910bea70 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbd1c032050 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd34173740 10 client.18438 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a790dc53f0 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.018+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a7910bea70 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fbd1c01f070 con 0x55a79114d940 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 10 client.18438 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 10 client.18438 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:10:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 20 client.18438 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911689f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 4 client.18438 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7910b8dc0 con 0x55a791166550 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd2affd640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd0eff5640 10 client.18438 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.019+0000 7fbd2affd640 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911689f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=472e7d705da1e1fc server_cookie=2102f8c41028d9d1 in_seq=0 out_seq=0 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd0eff5640 1 -- 172.21.15.38:0/1859881351 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7910b8dc0 con 0x55a791166550 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd0eff5640 10 client.18438 handle_command_reply: tid=0 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 client.18438 shutdown 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 msgr2=0x55a791153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00bfa0 tx=0x7fbd1c00b410 comp rx=0 tx=0).stop 2024-04-22T00:10:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 shutdown_connections 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911689f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a791156520 0x55a791153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a79114d940 0x55a791153260 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd0eff5640 0 client.18438 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 --2- 172.21.15.38:0/1859881351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d020 0x55a791155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 >> 172.21.15.38:0/1859881351 conn(0x55a791147750 msgr2=0x55a7911494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 shutdown_connections 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.020+0000 7fbd34173740 1 -- 172.21.15.38:0/1859881351 wait complete. 2024-04-22T00:10:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.021+0000 7fbd34173740 20 client.18438 trim_cache size 0 max 16384 2024-04-22T00:10:53.037 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:52 smithi203 ceph-mon[31431]: pgmap v270: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:53.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 Processor -- start 2024-04-22T00:10:53.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 -- start start 2024-04-22T00:10:53.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a791162170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a791155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1b0d20 con 0x55a791156520 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1b2920 con 0x55a79114d940 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd34173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1b1420 con 0x55a79114d020 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50760/0 (socket says 172.21.15.38:50760) 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/2480837837 learned_addr learned my addr 172.21.15.38:0/2480837837 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2affd640 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a791162170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/2480837837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 msgr2=0x55a791162170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a791162170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/2480837837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 msgr2=0x55a791155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a791155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/2480837837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a790e8f7d0 con 0x55a791156520 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2affd640 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a791162170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400cba0 tx=0x7fbd24004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00543c01335e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd10ff9640 1 -- 172.21.15.38:0/2480837837 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2409c430 con 0x55a791156520 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd10ff9640 1 -- 172.21.15.38:0/2480837837 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd2409bd30 con 0x55a791156520 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.034+0000 7fbd10ff9640 1 -- 172.21.15.38:0/2480837837 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240bd160 con 0x55a791156520 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- 172.21.15.38:0/2480837837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 msgr2=0x55a791155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400cba0 tx=0x7fbd24004920 comp rx=0 tx=0).stop 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- 172.21.15.38:0/2480837837 shutdown_connections 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a791155ed0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a791155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 --2- 172.21.15.38:0/2480837837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a791162170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- 172.21.15.38:0/2480837837 >> 172.21.15.38:0/2480837837 conn(0x55a791147750 msgr2=0x55a791149410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- 172.21.15.38:0/2480837837 shutdown_connections 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- 172.21.15.38:0/2480837837 wait complete. 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 -- start start 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.035+0000 7fbd34173740 1 Processor -- start 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a79115e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a79114fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1b2250 con 0x55a791156520 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1b1b20 con 0x55a79114d940 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2c1aed20 con 0x55a79114d020 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 10 client.0 _ll_register_callbacks cb 0x7fbd1c00b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50766/0 (socket says 172.21.15.38:50766) 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4022662845 learned_addr learned my addr 172.21.15.38:0/4022662845 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2affd640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a79115e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2a7fc640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a79114fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 msgr2=0x55a79115e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a79115e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4022662845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 msgr2=0x55a79114fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a79114fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 -- 172.21.15.38:0/4022662845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a791150630 con 0x55a791156520 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd2b7fe640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbd2407e0b0 tx=0x7fbd24009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab134566502c821 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240b8070 con 0x55a791156520 2024-04-22T00:10:53.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fbd240983a0 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.036+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd2409bd30 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fbd240a00e0 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 10 client.18444 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55a7911cc8d0 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd2409bd30 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fbd2401f020 con 0x55a791156520 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 10 client.18444 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 10 client.18444 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 20 client.18444 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 4 client.18444 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7910366e0 con 0x55a791166550 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.037+0000 7fbd2affd640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd0eff5640 10 client.18444 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd2affd640 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911cf060 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b67105649287145 server_cookie=86dd464236edb34 in_seq=0 out_seq=0 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd0eff5640 1 -- 172.21.15.38:0/4022662845 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7910366e0 con 0x55a791166550 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd0eff5640 10 client.18444 handle_command_reply: tid=0 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 client.18444 shutdown 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 msgr2=0x55a79114fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbd2407e0b0 tx=0x7fbd24009be0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 shutdown_connections 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55a791166550 0x55a7911cf060 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a791156520 0x55a79114fff0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a79114d940 0x55a79114fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.043 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd0eff5640 0 client.18444 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 --2- 172.21.15.38:0/4022662845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a79114d020 0x55a79115e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 >> 172.21.15.38:0/4022662845 conn(0x55a791147750 msgr2=0x55a79114df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 shutdown_connections 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 1 -- 172.21.15.38:0/4022662845 wait complete. 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd34173740 20 client.18444 trim_cache size 0 max 16384 2024-04-22T00:10:53.044 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 msgr2=0x7fbd140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 0x7fbd140a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c012980 tx=0x7fbd1c00bb80 comp rx=0 tx=0).stop 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 msgr2=0x7fbd2c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.038+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400efd0 tx=0x7fbd2400c6a0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 shutdown_connections 2024-04-22T00:10:53.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fbd140a2d00 0x7fbd140a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd2c1077a0 0x7fbd2c0744c0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c102150 0x7fbd2c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 --2- 172.21.15.38:0/4129207772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd2c101790 0x7fbd2c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 >> 172.21.15.38:0/4129207772 conn(0x7fbd2c0fd2a0 msgr2=0x7fbd2c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 shutdown_connections 2024-04-22T00:10:53.046 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.039+0000 7fbd3158a640 1 -- 172.21.15.38:0/4129207772 wait complete. 2024-04-22T00:10:53.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:10:53.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:52 smithi111 ceph-mon[31543]: pgmap v270: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- 172.21.15.38:0/4177063227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100610 msgr2=0x7f1e04100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/4177063227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100610 0x7f1e04100a90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009930 tx=0x7f1df802bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- 172.21.15.38:0/4177063227 shutdown_connections 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/4177063227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100fd0 0x7f1e0410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/4177063227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100610 0x7f1e04100a90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/4177063227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e04106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- 172.21.15.38:0/4177063227 >> 172.21.15.38:0/4177063227 conn(0x7f1e04077740 msgr2=0x7f1e04077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- 172.21.15.38:0/4177063227 shutdown_connections 2024-04-22T00:10:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- 172.21.15.38:0/4177063227 wait complete. 2024-04-22T00:10:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 Processor -- start 2024-04-22T00:10:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.159+0000 7f1e0a56c640 1 -- start start 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100610 0x7f1e04196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e0419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041109f0 con 0x7f1e04100fd0 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04110870 con 0x7f1e04100610 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e0a56c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e04110b70 con 0x7f1e041066b0 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50786/0 (socket says 172.21.15.38:50786) 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 -- 172.21.15.38:0/3100835243 learned_addr learned my addr 172.21.15.38:0/3100835243 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e0419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e03fff640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100610 0x7f1e04196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 -- 172.21.15.38:0/3100835243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 msgr2=0x7f1e0419b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e0419b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 -- 172.21.15.38:0/3100835243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100610 msgr2=0x7f1e04196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100610 0x7f1e04196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 -- 172.21.15.38:0/3100835243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0419be90 con 0x7f1e04100fd0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e0419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.160+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009230 tx=0x7f1df802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.161+0000 7f1e017fa640 1 -- 172.21.15.38:0/3100835243 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df8004450 con 0x7f1e04100fd0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.161+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0419c120 con 0x7f1e04100fd0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.161+0000 7f1e017fa640 1 -- 172.21.15.38:0/3100835243 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df80054e0 con 0x7f1e04100fd0 2024-04-22T00:10:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.161+0000 7f1e017fa640 1 -- 172.21.15.38:0/3100835243 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df803b440 con 0x7f1e04100fd0 2024-04-22T00:10:53.162 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.161+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e040688d0 con 0x7f1e04100fd0 2024-04-22T00:10:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.163+0000 7f1e017fa640 1 -- 172.21.15.38:0/3100835243 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1df8003780 con 0x7f1e04100fd0 2024-04-22T00:10:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.163+0000 7f1e0d232740 1 Processor -- start 2024-04-22T00:10:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.163+0000 7f1e0d232740 1 -- start start 2024-04-22T00:10:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8b30530 con 0x55d6c8bcc020 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8ad8830 con 0x55d6c8bd5520 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e0d232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8b2c4f0 con 0x55d6c8bcc940 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e017fa640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 0x7f1dec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50788/0 (socket says 172.21.15.38:50788) 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 -- 172.21.15.38:0/3711282306 learned_addr learned my addr 172.21.15.38:0/3711282306 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e03fff640 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 0x7f1dec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e017fa640 1 -- 172.21.15.38:0/3100835243 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1df80f32b0 con 0x7f1e04100fd0 2024-04-22T00:10:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 -- 172.21.15.38:0/3711282306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bcc940 msgr2=0x55d6c8bd4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 -- 172.21.15.38:0/3711282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bd5520 msgr2=0x55d6c8bd7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 -- 172.21.15.38:0/3711282306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c8afc2c0 con 0x55d6c8bcc020 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.164+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc013b80 tx=0x7f1dfc011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948919f297eafb80 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1dea7fc640 1 -- 172.21.15.38:0/3711282306 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0135b0 con 0x55d6c8bcc020 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1dea7fc640 1 -- 172.21.15.38:0/3711282306 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc013d90 con 0x55d6c8bcc020 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1dea7fc640 1 -- 172.21.15.38:0/3711282306 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc01dc50 con 0x55d6c8bcc020 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 -- 172.21.15.38:0/3711282306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 msgr2=0x55d6c8bcc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 0x7f1dec0a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc020000 tx=0x7f1dfc01f000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc013b80 tx=0x7f1dfc011750 comp rx=0 tx=0).stop 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 -- 172.21.15.38:0/3711282306 shutdown_connections 2024-04-22T00:10:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3711282306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8bcc400 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 -- 172.21.15.38:0/3711282306 >> 172.21.15.38:0/3711282306 conn(0x55d6c8bc6750 msgr2=0x55d6c8bc6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 -- 172.21.15.38:0/3711282306 shutdown_connections 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.165+0000 7f1e0d232740 1 -- 172.21.15.38:0/3711282306 wait complete. 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 -- start start 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 Processor -- start 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8bd8310 con 0x55d6c8bcc940 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8aebf10 con 0x55d6c8bcc020 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8bd8490 con 0x55d6c8bd5520 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e0d232740 10 client.0 _ll_register_callbacks cb 0x55d6c8843be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50800/0 (socket says 172.21.15.38:50800) 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e037fe640 1 -- 172.21.15.38:0/3639800533 learned_addr learned my addr 172.21.15.38:0/3639800533 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.166+0000 7f1e08b09640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e03fff640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 -- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 msgr2=0x55d6c8bd2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 -- 172.21.15.38:0/3639800533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 msgr2=0x55d6c8bd4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 -- 172.21.15.38:0/3639800533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c8abdc80 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e037fe640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1df802c820 tx=0x7f1df8003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf5d348c02e80f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df80bee50 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1df8048c00 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc013d90 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1df80bb790 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 10 client.18456 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d6c88443f0 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc013d90 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1df8042070 con 0x55d6c8bcc940 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 10 client.18456 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 10 client.18456 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:10:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.167+0000 7f1e0d232740 20 client.18456 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8be5570 0x55d6c8be7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1e0d232740 4 client.18456 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6c8b3da70 con 0x55d6c8be5570 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1e03fff640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8be5570 0x55d6c8be7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1e03fff640 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8be5570 0x55d6c8be7a10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c504d89c6e420d52 server_cookie=dd7adabde375e898 in_seq=0 out_seq=0 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.168+0000 7f1de37fe640 10 client.18456 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1de37fe640 1 -- 172.21.15.38:0/3639800533 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6c8b3da70 con 0x55d6c8be5570 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1de37fe640 10 client.18456 handle_command_reply: tid=0 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 client.18456 shutdown 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 msgr2=0x55d6c8bd2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1df802c820 tx=0x7f1df8003b20 comp rx=0 tx=0).stop 2024-04-22T00:10:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 shutdown_connections 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8be5570 0x55d6c8be7a10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd2280 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1de37fe640 0 client.18456 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 --2- 172.21.15.38:0/3639800533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 >> 172.21.15.38:0/3639800533 conn(0x55d6c8bc6750 msgr2=0x55d6c8bc84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 shutdown_connections 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 1 -- 172.21.15.38:0/3639800533 wait complete. 2024-04-22T00:10:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.169+0000 7f1e0d232740 20 client.18456 trim_cache size 0 max 16384 2024-04-22T00:10:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.182+0000 7f1e0d232740 1 Processor -- start 2024-04-22T00:10:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 -- start start 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ac870 con 0x55d6c8bcc020 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ae470 con 0x55d6c8bcc940 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e0d232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041acf70 con 0x55d6c8bd5520 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50802/0 (socket says 172.21.15.38:50802) 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 -- 172.21.15.38:0/277108807 learned_addr learned my addr 172.21.15.38:0/277108807 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 -- 172.21.15.38:0/277108807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 msgr2=0x55d6c8bd4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 -- 172.21.15.38:0/277108807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc940 msgr2=0x55d6c8bd49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e037fe640 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 -- 172.21.15.38:0/277108807 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c890e7d0 con 0x55d6c8bcc020 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1e03fff640 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1df4004700 tx=0x7f1df4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352f61889e9bc4cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1de9ffb640 1 -- 172.21.15.38:0/277108807 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df40142d0 con 0x55d6c8bcc020 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1de9ffb640 1 -- 172.21.15.38:0/277108807 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df4014e30 con 0x55d6c8bcc020 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.183+0000 7f1de9ffb640 1 -- 172.21.15.38:0/277108807 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4003970 con 0x55d6c8bcc020 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- 172.21.15.38:0/277108807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 msgr2=0x55d6c8be11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1df4004700 tx=0x7f1df4009f90 comp rx=0 tx=0).stop 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- 172.21.15.38:0/277108807 shutdown_connections 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8bd4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 --2- 172.21.15.38:0/277108807 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc020 0x55d6c8be11d0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- 172.21.15.38:0/277108807 >> 172.21.15.38:0/277108807 conn(0x55d6c8bc6750 msgr2=0x55d6c8bd0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- 172.21.15.38:0/277108807 shutdown_connections 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- 172.21.15.38:0/277108807 wait complete. 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.184+0000 7f1e0d232740 1 -- start start 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 Processor -- start 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8c4ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041adda0 con 0x55d6c8bcc940 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ad670 con 0x55d6c8bcc020 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041aa870 con 0x55d6c8bd5520 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e0d232740 10 client.0 _ll_register_callbacks cb 0x7f1df8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50816/0 (socket says 172.21.15.38:50816) 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 -- 172.21.15.38:0/1756301834 learned_addr learned my addr 172.21.15.38:0/1756301834 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e03fff640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 -- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 msgr2=0x55d6c8c4ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8c4ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 -- 172.21.15.38:0/1756301834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 msgr2=0x55d6c8bd97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 -- 172.21.15.38:0/1756301834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c8c4b0d0 con 0x55d6c8bcc940 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1e037fe640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1df80bbe70 tx=0x7f1df8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85b5ac0c9dc6d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1de37fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.185+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df80bee50 con 0x55d6c8bcc940 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1df80c7ed0 con 0x55d6c8bcc940 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df4014e30 con 0x55d6c8bcc940 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1df80c81b0 con 0x55d6c8bcc940 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 10 client.18462 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d6c8c4c340 con 0x55d6c8bcc940 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df4014e30 con 0x55d6c8bcc940 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1df8042070 con 0x55d6c8bcc940 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 10 client.18462 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 10 client.18462 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 20 client.18462 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8c4c8c0 0x55d6c8c4ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 4 client.18462 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6c8b37dc0 con 0x55d6c8c4c8c0 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.186+0000 7f1e03fff640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8c4c8c0 0x55d6c8c4ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e03fff640 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8c4c8c0 0x55d6c8c4ece0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48b1c607fbbdbb5 server_cookie=30dd68166c444f6d in_seq=0 out_seq=0 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1de37fe640 10 client.18462 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1de37fe640 1 -- 172.21.15.38:0/1756301834 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6c8b37dc0 con 0x55d6c8c4c8c0 2024-04-22T00:10:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1de37fe640 10 client.18462 handle_command_reply: tid=0 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 client.18462 shutdown 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 msgr2=0x55d6c8bd9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1df80bbe70 tx=0x7f1df8000bc0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 shutdown_connections 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d6c8c4c8c0 0x55d6c8c4ece0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6c8bd5520 0x55d6c8c4ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d6c8bcc940 0x55d6c8bd9cf0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1de37fe640 0 client.18462 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 --2- 172.21.15.38:0/1756301834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d6c8bcc020 0x55d6c8bd97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 >> 172.21.15.38:0/1756301834 conn(0x55d6c8bc6750 msgr2=0x55d6c8bcce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 shutdown_connections 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 1 -- 172.21.15.38:0/1756301834 wait complete. 2024-04-22T00:10:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.187+0000 7f1e0d232740 20 client.18462 trim_cache size 0 max 16384 2024-04-22T00:10:53.193 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 msgr2=0x7f1dec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 0x7f1dec0a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc020000 tx=0x7f1dfc01f000 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 msgr2=0x7f1e04197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009230 tx=0x7f1df802cba0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 shutdown_connections 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1dec0a2d00 0x7f1dec0a51c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e041066b0 0x7f1e0419b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1e04100fd0 0x7f1e04197430 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 --2- 172.21.15.38:0/3100835243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e04100610 0x7f1e04196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 >> 172.21.15.38:0/3100835243 conn(0x7f1e04077740 msgr2=0x7f1e040ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 shutdown_connections 2024-04-22T00:10:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.188+0000 7f1e0a56c640 1 -- 172.21.15.38:0/3100835243 wait complete. 2024-04-22T00:10:53.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:10:53.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 -- 172.21.15.38:0/776206121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb918107730 msgr2=0x7fb918107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 --2- 172.21.15.38:0/776206121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb918107730 0x7fb918107b10 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fb914009930 tx=0x7fb91402bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:53.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 -- 172.21.15.38:0/776206121 shutdown_connections 2024-04-22T00:10:53.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 --2- 172.21.15.38:0/776206121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb91806a780 0x7fb91810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 --2- 172.21.15.38:0/776206121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb91806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 --2- 172.21.15.38:0/776206121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb918107730 0x7fb918107b10 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 -- 172.21.15.38:0/776206121 >> 172.21.15.38:0/776206121 conn(0x7fb9180776a0 msgr2=0x7fb918077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 -- 172.21.15.38:0/776206121 shutdown_connections 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.303+0000 7fb9203f7640 1 -- 172.21.15.38:0/776206121 wait complete. 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 Processor -- start 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 -- start start 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb9181981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb918107730 0x7fb91819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb918111b40 con 0x7fb91806a780 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181119c0 con 0x7fb918069dc0 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb9203f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb918111cc0 con 0x7fb918107730 2024-04-22T00:10:53.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50824/0 (socket says 172.21.15.38:50824) 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb91d992640 1 -- 172.21.15.38:0/1892246586 learned_addr learned my addr 172.21.15.38:0/1892246586 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb91e994640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb918107730 0x7fb91819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.304+0000 7fb91e193640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb9181981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 -- 172.21.15.38:0/1892246586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb918107730 msgr2=0x7fb91819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb918107730 0x7fb91819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 -- 172.21.15.38:0/1892246586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 msgr2=0x7fb9181981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb9181981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 -- 172.21.15.38:0/1892246586 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91819d190 con 0x7fb91806a780 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91e193640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb9181981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb91d992640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9f0 tx=0x7fb90c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb90b7fe640 1 -- 172.21.15.38:0/1892246586 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00cd80 con 0x7fb91806a780 2024-04-22T00:10:53.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91819d480 con 0x7fb91806a780 2024-04-22T00:10:53.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb90b7fe640 1 -- 172.21.15.38:0/1892246586 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb90c010910 con 0x7fb91806a780 2024-04-22T00:10:53.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb90b7fe640 1 -- 172.21.15.38:0/1892246586 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0188f0 con 0x7fb91806a780 2024-04-22T00:10:53.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.305+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9181a4d10 con 0x7fb91806a780 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.307+0000 7fb90b7fe640 1 -- 172.21.15.38:0/1892246586 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fb90c010ac0 con 0x7fb91806a780 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 Processor -- start 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 -- start start 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc14c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc157910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556cbc0b0530 con 0x556cbc155520 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556cbc058830 con 0x556cbc14c020 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb922fe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556cbc0ac4f0 con 0x556cbc14c940 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb90b7fe640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.308+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50832/0 (socket says 172.21.15.38:50832) 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91d992640 1 -- 172.21.15.38:0/3154637640 learned_addr learned my addr 172.21.15.38:0/3154637640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e994640 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc14c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e994640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb90b7fe640 1 -- 172.21.15.38:0/1892246586 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fb90c0cfc40 con 0x7fb91806a780 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 -- 172.21.15.38:0/3154637640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 msgr2=0x556cbc14c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc14c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 -- 172.21.15.38:0/3154637640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 msgr2=0x556cbc157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 -- 172.21.15.38:0/3154637640 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cbc07c2c0 con 0x556cbc14c940 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e994640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 0x7fb9040a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb910005fd0 tx=0x7fb910009500 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb91e193640 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc154fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb914009230 tx=0x7fb914005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8e4a11ef08d4166 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb908bfe640 1 -- 172.21.15.38:0/3154637640 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91402ca80 con 0x556cbc14c940 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb908bfe640 1 -- 172.21.15.38:0/3154637640 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91402cc20 con 0x556cbc14c940 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.309+0000 7fb908bfe640 1 -- 172.21.15.38:0/3154637640 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9140046c0 con 0x556cbc14c940 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 -- 172.21.15.38:0/3154637640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 msgr2=0x556cbc154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc154fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb914009230 tx=0x7fb914005dd0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 -- 172.21.15.38:0/3154637640 shutdown_connections 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc154fe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 --2- 172.21.15.38:0/3154637640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc14c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 -- 172.21.15.38:0/3154637640 >> 172.21.15.38:0/3154637640 conn(0x556cbc146750 msgr2=0x556cbc146b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 -- 172.21.15.38:0/3154637640 shutdown_connections 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.310+0000 7fb922fe7740 1 -- 172.21.15.38:0/3154637640 wait complete. 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 -- start start 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 Processor -- start 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc154df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc155520 0x556cbc1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556cbc1582b0 con 0x556cbc14c940 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556cbc06bf10 con 0x556cbc14c020 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556cbc158430 con 0x556cbc155520 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb922fe7740 10 client.0 _ll_register_callbacks cb 0x556cbbdc3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50844/0 (socket says 172.21.15.38:50844) 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91e994640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc155520 0x556cbc1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 -- 172.21.15.38:0/721418503 learned_addr learned my addr 172.21.15.38:0/721418503 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91e193640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 -- 172.21.15.38:0/721418503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc155520 msgr2=0x556cbc1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc155520 0x556cbc1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 -- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 msgr2=0x556cbc154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 -- 172.21.15.38:0/721418503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cbc03dc80 con 0x556cbc14c940 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb91d992640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00c6a0 tx=0x7fb90c09c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02a890a906fb49c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.311+0000 7fb901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0cd490 con 0x556cbc14c940 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fb90c09b420 con 0x556cbc14c940 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb91402cc20 con 0x556cbc14c940 2024-04-22T00:10:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb90c09bdd0 con 0x556cbc14c940 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 10 client.18474 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556cbbdc43f0 con 0x556cbc14c940 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91402cc20 con 0x556cbc14c940 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fb90c0b1020 con 0x556cbc14c940 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 10 client.18474 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 10 client.18474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 20 client.18474 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc165480 0x556cbc167890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 4 client.18474 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.312+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cbc0bda70 con 0x556cbc165480 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.313+0000 7fb91e193640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc165480 0x556cbc167890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.313+0000 7fb91e193640 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc165480 0x556cbc167890 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e1fa36ec5c926c4 server_cookie=89876412095a7bd1 in_seq=0 out_seq=0 2024-04-22T00:10:53.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.313+0000 7fb901ffb640 10 client.18474 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb901ffb640 1 -- 172.21.15.38:0/721418503 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cbc0bda70 con 0x556cbc165480 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb901ffb640 10 client.18474 handle_command_reply: tid=0 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 client.18474 shutdown 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 msgr2=0x556cbc152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00c6a0 tx=0x7fb90c09c640 comp rx=0 tx=0).stop 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 shutdown_connections 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc155520 0x556cbc1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc165480 0x556cbc167890 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc14c940 0x556cbc152180 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 --2- 172.21.15.38:0/721418503 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb901ffb640 0 client.18474 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 >> 172.21.15.38:0/721418503 conn(0x556cbc146750 msgr2=0x556cbc1484a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 shutdown_connections 2024-04-22T00:10:53.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 1 -- 172.21.15.38:0/721418503 wait complete. 2024-04-22T00:10:53.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.314+0000 7fb922fe7740 20 client.18474 trim_cache size 0 max 16384 2024-04-22T00:10:53.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.327+0000 7fb922fe7740 1 Processor -- start 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.327+0000 7fb922fe7740 1 -- start start 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c020 0x556cbc161170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 0x556cbc1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181aeb60 con 0x556cbc155520 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181b0760 con 0x556cbc14c940 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb922fe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181af260 con 0x556cbc14c020 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50856/0 (socket says 172.21.15.38:50856) 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 -- 172.21.15.38:0/134633723 learned_addr learned my addr 172.21.15.38:0/134633723 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e193640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c020 0x556cbc161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91d992640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 0x556cbc1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 -- 172.21.15.38:0/134633723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c020 msgr2=0x556cbc161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c020 0x556cbc161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 -- 172.21.15.38:0/134633723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 msgr2=0x556cbc1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 0x556cbc1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 -- 172.21.15.38:0/134633723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cbbe8e7d0 con 0x556cbc155520 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91d992640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 0x556cbc1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb91e994640 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb9100095c0 tx=0x7fb910008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74a2e16710aa5cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb903fff640 1 -- 172.21.15.38:0/134633723 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910011d30 con 0x556cbc155520 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb903fff640 1 -- 172.21.15.38:0/134633723 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb910005020 con 0x556cbc155520 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.328+0000 7fb903fff640 1 -- 172.21.15.38:0/134633723 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910010670 con 0x556cbc155520 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- 172.21.15.38:0/134633723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 msgr2=0x556cbc154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb9100095c0 tx=0x7fb910008000 comp rx=0 tx=0).stop 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- 172.21.15.38:0/134633723 shutdown_connections 2024-04-22T00:10:53.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc154df0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c940 0x556cbc1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 --2- 172.21.15.38:0/134633723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c020 0x556cbc161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- 172.21.15.38:0/134633723 >> 172.21.15.38:0/134633723 conn(0x556cbc146750 msgr2=0x556cbc14d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- 172.21.15.38:0/134633723 shutdown_connections 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- 172.21.15.38:0/134633723 wait complete. 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 -- start start 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.329+0000 7fb922fe7740 1 Processor -- start 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181b0090 con 0x556cbc155520 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181af960 con 0x556cbc14c020 2024-04-22T00:10:53.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9181acb60 con 0x556cbc14c940 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 10 client.0 _ll_register_callbacks cb 0x7fb90c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50872/0 (socket says 172.21.15.38:50872) 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 -- 172.21.15.38:0/1287826091 learned_addr learned my addr 172.21.15.38:0/1287826091 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91d992640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e193640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 -- 172.21.15.38:0/1287826091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 msgr2=0x556cbc1597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc1597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 -- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 msgr2=0x556cbc159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 -- 172.21.15.38:0/1287826091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cbc1cab80 con 0x556cbc155520 2024-04-22T00:10:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb91e994640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb910009590 tx=0x7fb910004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8055617ec9bf4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb901ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910037060 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fb91002f040 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb910005020 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb910003970 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 10 client.18480 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.330+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556cbc1cbdb0 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb910005020 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fb91001d070 con 0x556cbc155520 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 10 client.18480 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 10 client.18480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 20 client.18480 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc1cc4d0 0x556cbc1ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 4 client.18480 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cbc0b7dc0 con 0x556cbc1cc4d0 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb91e193640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc1cc4d0 0x556cbc1ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb901ffb640 10 client.18480 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.331+0000 7fb91e193640 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc1cc4d0 0x556cbc1ce8f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c83cf5df137cc8b server_cookie=6af09960080cfde in_seq=0 out_seq=0 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb901ffb640 1 -- 172.21.15.38:0/1287826091 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cbc0b7dc0 con 0x556cbc1cc4d0 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb901ffb640 10 client.18480 handle_command_reply: tid=0 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 client.18480 shutdown 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 msgr2=0x556cbc159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb910009590 tx=0x7fb910004eb0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 shutdown_connections 2024-04-22T00:10:53.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x556cbc1cc4d0 0x556cbc1ce8f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556cbc155520 0x556cbc159ce0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556cbc14c940 0x556cbc1597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb901ffb640 0 client.18480 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 --2- 172.21.15.38:0/1287826091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556cbc14c020 0x556cbc159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 >> 172.21.15.38:0/1287826091 conn(0x556cbc146750 msgr2=0x556cbc144fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 shutdown_connections 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 1 -- 172.21.15.38:0/1287826091 wait complete. 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb922fe7740 20 client.18480 trim_cache size 0 max 16384 2024-04-22T00:10:53.338 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:10:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 msgr2=0x7fb9040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 0x7fb9040a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb910005fd0 tx=0x7fb910009500 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 msgr2=0x7fb918198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9f0 tx=0x7fb90c00eeb0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 shutdown_connections 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fb9040a2d00 0x7fb9040a51c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb918107730 0x7fb91819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb91806a780 0x7fb918198730 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 --2- 172.21.15.38:0/1892246586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb918069dc0 0x7fb9181981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.332+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 >> 172.21.15.38:0/1892246586 conn(0x7fb9180776a0 msgr2=0x7fb9180fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.333+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 shutdown_connections 2024-04-22T00:10:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:53.333+0000 7fb9203f7640 1 -- 172.21.15.38:0/1892246586 wait complete. 2024-04-22T00:10:53.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:53 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1198118563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:53.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:53 smithi038 ceph-mon[27972]: pgmap v271: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:54.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:53 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1198118563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:54.033 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:53 smithi203 ceph-mon[31431]: pgmap v271: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:54.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:53 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1198118563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:10:54.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:53 smithi111 ceph-mon[31543]: pgmap v271: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-22T00:10:54.343 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:10:54.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:10:54.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.465+0000 7f0f966fb640 1 -- 172.21.15.38:0/3171404296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90106630 msgr2=0x7f0f90106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.465+0000 7f0f966fb640 1 --2- 172.21.15.38:0/3171404296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90106630 0x7f0f90106a10 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009930 tx=0x7f0f7802bc30 comp rx=0 tx=0).stop 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 -- 172.21.15.38:0/3171404296 shutdown_connections 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 --2- 172.21.15.38:0/3171404296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100fe0 0x7f0f9010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 --2- 172.21.15.38:0/3171404296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90100620 0x7f0f90100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 --2- 172.21.15.38:0/3171404296 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90106630 0x7f0f90106a10 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 -- 172.21.15.38:0/3171404296 >> 172.21.15.38:0/3171404296 conn(0x7f0f900776a0 msgr2=0x7f0f90077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 -- 172.21.15.38:0/3171404296 shutdown_connections 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 -- 172.21.15.38:0/3171404296 wait complete. 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 Processor -- start 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.466+0000 7f0f966fb640 1 -- start start 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100620 0x7f0f901980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90106630 0x7f0f9019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90110a40 con 0x7f0f90100fe0 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901108c0 con 0x7f0f90100620 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f966fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90110bc0 con 0x7f0f90106630 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36874/0 (socket says 172.21.15.38:36874) 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/1766366858 learned_addr learned my addr 172.21.15.38:0/1766366858 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100620 0x7f0f901980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f94c98640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90106630 0x7f0f9019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/1766366858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90106630 msgr2=0x7f0f9019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90106630 0x7f0f9019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/1766366858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100620 msgr2=0x7f0f901980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100620 0x7f0f901980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/1766366858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9019d0e0 con 0x7f0f90100fe0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000e9f0 tx=0x7f0f8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.467+0000 7f0f8d7fa640 1 -- 172.21.15.38:0/1766366858 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8000cd80 con 0x7f0f90100fe0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.468+0000 7f0f8d7fa640 1 -- 172.21.15.38:0/1766366858 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f80005500 con 0x7f0f90100fe0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.468+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f9019d3d0 con 0x7f0f90100fe0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.468+0000 7f0f8d7fa640 1 -- 172.21.15.38:0/1766366858 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80010690 con 0x7f0f90100fe0 2024-04-22T00:10:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.468+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f901a4c60 con 0x7f0f90100fe0 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.470+0000 7f0f8d7fa640 1 -- 172.21.15.38:0/1766366858 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f0f800046c0 con 0x7f0f90100fe0 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.470+0000 7f0f993c2740 1 Processor -- start 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.470+0000 7f0f993c2740 1 -- start start 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.470+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b55048470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b55048ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b550541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f993c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559b54fac170 con 0x559b55049720 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f993c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559b54f547c0 con 0x559b55048a40 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f993c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b54fa8130 con 0x559b55051db0 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8d7fa640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f94c98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b55048470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f94c98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b55048470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36876/0 (socket says 172.21.15.38:36876) 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f94c98640 1 -- 172.21.15.38:0/3025077394 learned_addr learned my addr 172.21.15.38:0/3025077394 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b55048ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b550541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f94c98640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 -- 172.21.15.38:0/3025077394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 msgr2=0x559b550541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b550541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 -- 172.21.15.38:0/3025077394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 msgr2=0x559b55048470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b55048470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 -- 172.21.15.38:0/3025077394 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b54f65e60 con 0x559b55048a40 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8d7fa640 1 -- 172.21.15.38:0/1766366858 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0f800cfcb0 con 0x7f0f90100fe0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f94c98640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88004ab0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.471+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b55048ec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009230 tx=0x7f0f78009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978e60272a5e68ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.472+0000 7f0f867fc640 1 -- 172.21.15.38:0/3025077394 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78042070 con 0x559b55048a40 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.472+0000 7f0f867fc640 1 -- 172.21.15.38:0/3025077394 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7802c690 con 0x559b55048a40 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.472+0000 7f0f867fc640 1 -- 172.21.15.38:0/3025077394 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78004b60 con 0x559b55048a40 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.472+0000 7f0f993c2740 1 -- 172.21.15.38:0/3025077394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 msgr2=0x559b55048ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.472+0000 7f0f993c2740 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b55048ec0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009230 tx=0x7f0f78009610 comp rx=0 tx=0).stop 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- 172.21.15.38:0/3025077394 shutdown_connections 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b550541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b55048ec0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- 172.21.15.38:0/3025077394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b55048470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- 172.21.15.38:0/3025077394 >> 172.21.15.38:0/3025077394 conn(0x559b5500f4f0 msgr2=0x559b55043490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- 172.21.15.38:0/3025077394 shutdown_connections 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- 172.21.15.38:0/3025077394 wait complete. 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- start start 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 Processor -- start 2024-04-22T00:10:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b5504ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b5504f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559b55054bc0 con 0x559b55049720 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559b54f67d40 con 0x559b55048a40 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.473+0000 7f0f993c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b55054d40 con 0x559b55051db0 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f993c2740 10 client.0 _ll_register_callbacks cb 0x559b54fa3820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36886/0 (socket says 172.21.15.38:36886) 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b5504ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/911927951 learned_addr learned my addr 172.21.15.38:0/911927951 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f94c98640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b5504f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/911927951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 msgr2=0x559b5504f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b5504f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 msgr2=0x559b5504ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b5504ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 -- 172.21.15.38:0/911927951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b54d887e0 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f94c98640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b5504f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b5504ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0f800982a0 tx=0x7f0f8009b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd24cfd1c2fffc08 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80066060 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0f800a4820 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b54f80450 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0f8009e070 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f993c2740 10 client.18492 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.474+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b54de8f70 con 0x559b55049720 2024-04-22T00:10:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559b54f80450 con 0x559b55049720 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f0f8009b320 con 0x559b55049720 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 10 client.18492 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 20 client.18492 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b55061f40 0x559b55064370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 4 client.18492 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b54fb96b0 con 0x559b55061f40 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.475+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b55061f40 0x559b55064370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.476+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b55061f40 0x559b55064370 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb510253999e759d server_cookie=94729d050f7d5642 in_seq=0 out_seq=0 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.476+0000 7f0f6f7fe640 10 client.18492 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.476+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/911927951 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b54fb96b0 con 0x559b55061f40 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.476+0000 7f0f6f7fe640 10 client.18492 handle_command_reply: tid=0 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.476+0000 7f0f993c2740 1 client.18492 shutdown 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 msgr2=0x559b5504f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0f800982a0 tx=0x7f0f8009b980 comp rx=0 tx=0).stop 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 shutdown_connections 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b5504f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b55061f40 0x559b55064370 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55049720 0x559b5504f360 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 --2- 172.21.15.38:0/911927951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55048a40 0x559b5504ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 >> 172.21.15.38:0/911927951 conn(0x559b5500f4f0 msgr2=0x559b550451e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f6f7fe640 0 client.18492 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 shutdown_connections 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 1 -- 172.21.15.38:0/911927951 wait complete. 2024-04-22T00:10:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.477+0000 7f0f993c2740 20 client.18492 trim_cache size 0 max 16384 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 Processor -- start 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 -- start start 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5505dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b55042370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901aeb40 con 0x559b55048a40 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901b0740 con 0x559b55049720 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f993c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901af240 con 0x559b55051db0 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36900/0 (socket says 172.21.15.38:36900) 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 -- 172.21.15.38:0/979220049 learned_addr learned my addr 172.21.15.38:0/979220049 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 -- 172.21.15.38:0/979220049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 msgr2=0x559b55042370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b55042370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 -- 172.21.15.38:0/979220049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 msgr2=0x559b5505dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5505dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.495+0000 7f0f8ffff640 1 -- 172.21.15.38:0/979220049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b54c80170 con 0x559b55048a40 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x559b55049c70 tx=0x7f0f78005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2dc6ab5fe55c5dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f85ffb640 1 -- 172.21.15.38:0/979220049 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78042040 con 0x559b55048a40 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f85ffb640 1 -- 172.21.15.38:0/979220049 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f78004040 con 0x559b55048a40 2024-04-22T00:10:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f85ffb640 1 -- 172.21.15.38:0/979220049 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7804f6f0 con 0x559b55048a40 2024-04-22T00:10:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 -- 172.21.15.38:0/979220049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 msgr2=0x559b55051550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x559b55049c70 tx=0x7f0f78005fd0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 -- 172.21.15.38:0/979220049 shutdown_connections 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55051db0 0x559b55042370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5505dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 --2- 172.21.15.38:0/979220049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55048a40 0x559b55051550 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.496+0000 7f0f993c2740 1 -- 172.21.15.38:0/979220049 >> 172.21.15.38:0/979220049 conn(0x559b5500f4f0 msgr2=0x559b55041c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- 172.21.15.38:0/979220049 shutdown_connections 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- 172.21.15.38:0/979220049 wait complete. 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- start start 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 Processor -- start 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55048a40 0x559b5505a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5504c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901b0070 con 0x559b55051db0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901af940 con 0x559b55049720 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f901acb40 con 0x559b55048a40 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f993c2740 10 client.0 _ll_register_callbacks cb 0x7f0f8000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f94c98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f94c98640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36902/0 (socket says 172.21.15.38:36902) 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.497+0000 7f0f94c98640 1 -- 172.21.15.38:0/4006340464 learned_addr learned my addr 172.21.15.38:0/4006340464 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55048a40 0x559b5505a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f8f7fe640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5504c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 -- 172.21.15.38:0/4006340464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55048a40 msgr2=0x559b5505a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55048a40 0x559b5505a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 -- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 msgr2=0x559b5504c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5504c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 -- 172.21.15.38:0/4006340464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b55052810 con 0x559b55051db0 2024-04-22T00:10:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f94c98640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f0f880041d0 tx=0x7f0f88009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e00f5ffaa6c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88004d20 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0f88011930 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f78004040 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f0f88031050 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f993c2740 10 client.18498 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559b550cca40 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.498+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f78004040 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f0f8801e040 con 0x559b55051db0 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 10 client.18498 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 10 client.18498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 20 client.18498 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b550cd270 0x559b550cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 4 client.18498 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559b54fb3a00 con 0x559b550cd270 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b550cd270 0x559b550cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f6f7fe640 10 client.18498 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.499+0000 7f0f8ffff640 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b550cd270 0x559b550cf690 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6a946404be3c6a4 server_cookie=77e74c4a84e7f045 in_seq=0 out_seq=0 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.500+0000 7f0f6f7fe640 1 -- 172.21.15.38:0/4006340464 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559b54fb3a00 con 0x559b550cd270 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.500+0000 7f0f6f7fe640 10 client.18498 handle_command_reply: tid=0 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.500+0000 7f0f993c2740 1 client.18498 shutdown 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 msgr2=0x559b5504c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f0f880041d0 tx=0x7f0f88009e40 comp rx=0 tx=0).stop 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 shutdown_connections 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559b550cd270 0x559b550cf690 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559b55051db0 0x559b5504c660 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559b55049720 0x559b5504c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 --2- 172.21.15.38:0/4006340464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b55048a40 0x559b5505a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 >> 172.21.15.38:0/4006340464 conn(0x559b5500f4f0 msgr2=0x559b5504f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f6f7fe640 0 client.18498 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 shutdown_connections 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 1 -- 172.21.15.38:0/4006340464 wait complete. 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f993c2740 20 client.18498 trim_cache size 0 max 16384 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "e4b3f296-c4db-4083-b980-deb18af48413", 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:10:54.504 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:54.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 msgr2=0x7f0f7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88004ab0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 msgr2=0x7f0f90198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.501+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000e9f0 tx=0x7f0f8000edf0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 shutdown_connections 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f0f7c0a2d00 0x7f0f7c0a51c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f90106630 0x7f0f9019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f90100fe0 0x7f0f90198630 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 --2- 172.21.15.38:0/1766366858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f90100620 0x7f0f901980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 >> 172.21.15.38:0/1766366858 conn(0x7f0f900776a0 msgr2=0x7f0f9006a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 shutdown_connections 2024-04-22T00:10:54.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.502+0000 7f0f966fb640 1 -- 172.21.15.38:0/1766366858 wait complete. 2024-04-22T00:10:54.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:10:54.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 -- 172.21.15.38:0/302861389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff35410dfd0 msgr2=0x7ff3541103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 --2- 172.21.15.38:0/302861389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff35410dfd0 0x7ff3541103c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009900 tx=0x7ff34c02bcd0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 -- 172.21.15.38:0/302861389 shutdown_connections 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 --2- 172.21.15.38:0/302861389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff35410dfd0 0x7ff3541103c0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 --2- 172.21.15.38:0/302861389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354074410 0x7ff35410da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 --2- 172.21.15.38:0/302861389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff354073af0 0x7ff354073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 -- 172.21.15.38:0/302861389 >> 172.21.15.38:0/302861389 conn(0x7ff3540fd180 msgr2=0x7ff3540ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 -- 172.21.15.38:0/302861389 shutdown_connections 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 -- 172.21.15.38:0/302861389 wait complete. 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.618+0000 7ff358aa7640 1 Processor -- start 2024-04-22T00:10:54.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 -- start start 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 0x7ff35419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff35410dfd0 0x7ff35419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff354113cf0 con 0x7ff354074410 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff354113b70 con 0x7ff354073af0 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff358aa7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff354113e70 con 0x7ff35410dfd0 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36916/0 (socket says 172.21.15.38:36916) 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 -- 172.21.15.38:0/1942548535 learned_addr learned my addr 172.21.15.38:0/1942548535 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff35259c640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 0x7ff35419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff352d9d640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff35410dfd0 0x7ff35419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 -- 172.21.15.38:0/1942548535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff35410dfd0 msgr2=0x7ff35419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff35410dfd0 0x7ff35419eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 -- 172.21.15.38:0/1942548535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 msgr2=0x7ff35419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 0x7ff35419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 -- 172.21.15.38:0/1942548535 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35419f1d0 con 0x7ff354074410 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff35259c640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 0x7ff35419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff351d9b640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ff34800d950 tx=0x7ff34800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.619+0000 7ff3437fe640 1 -- 172.21.15.38:0/1942548535 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348017070 con 0x7ff354074410 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.620+0000 7ff3437fe640 1 -- 172.21.15.38:0/1942548535 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff348009c30 con 0x7ff354074410 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.620+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35419f4c0 con 0x7ff354074410 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.620+0000 7ff3437fe640 1 -- 172.21.15.38:0/1942548535 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34800f6b0 con 0x7ff354074410 2024-04-22T00:10:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.620+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3541a6d50 con 0x7ff354074410 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff3437fe640 1 -- 172.21.15.38:0/1942548535 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff3480109d0 con 0x7ff354074410 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 Processor -- start 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 -- start start 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5623b4a01560 con 0x5623b4a9cf30 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5623b49a9730 con 0x5623b4a9d850 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff35b76c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5623b49fd520 con 0x5623b4aa6430 2024-04-22T00:10:54.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.622+0000 7ff352d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36932/0 (socket says 172.21.15.38:36932) 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 -- 172.21.15.38:0/793016149 learned_addr learned my addr 172.21.15.38:0/793016149 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff3437fe640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 0x7ff33c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff35259c640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff351d9b640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 0x7ff33c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff3437fe640 1 -- 172.21.15.38:0/1942548535 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff3480cf430 con 0x7ff354074410 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 -- 172.21.15.38:0/793016149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 msgr2=0x5623b4aa8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 -- 172.21.15.38:0/793016149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 msgr2=0x5623b4aa5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 -- 172.21.15.38:0/793016149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b47df7d0 con 0x5623b4a9cf30 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff351d9b640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:10:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005ab0 tx=0x7ff34c02cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6c93f6fd1b8ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff340ff9640 1 -- 172.21.15.38:0/793016149 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c005300 con 0x5623b4a9cf30 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff340ff9640 1 -- 172.21.15.38:0/793016149 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff34c0054a0 con 0x5623b4a9cf30 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff340ff9640 1 -- 172.21.15.38:0/793016149 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c003850 con 0x5623b4a9cf30 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.623+0000 7ff352d9d640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 0x7ff33c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5623b4aa8df0 tx=0x7ff34c04a000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 -- 172.21.15.38:0/793016149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 msgr2=0x5623b4a9d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005ab0 tx=0x7ff34c02cc60 comp rx=0 tx=0).stop 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 -- 172.21.15.38:0/793016149 shutdown_connections 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 --2- 172.21.15.38:0/793016149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4a9d310 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 -- 172.21.15.38:0/793016149 >> 172.21.15.38:0/793016149 conn(0x5623b4a97660 msgr2=0x5623b4a97a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 -- 172.21.15.38:0/793016149 shutdown_connections 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.624+0000 7ff35b76c740 1 -- 172.21.15.38:0/793016149 wait complete. 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 -- start start 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 Processor -- start 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5623b4aa9220 con 0x5623b4a9cf30 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5623b49bce00 con 0x5623b4a9d850 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5623b4aa93a0 con 0x5623b4aa6430 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35b76c740 10 client.0 _ll_register_callbacks cb 0x5623b49f7eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36946/0 (socket says 172.21.15.38:36946) 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 -- 172.21.15.38:0/3133932277 learned_addr learned my addr 172.21.15.38:0/3133932277 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff352d9d640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff351d9b640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 -- 172.21.15.38:0/3133932277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 msgr2=0x5623b4aa3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 -- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 msgr2=0x5623b4aa39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.625+0000 7ff35259c640 1 -- 172.21.15.38:0/3133932277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b47153f0 con 0x5623b4a9cf30 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35259c640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff344013a50 tx=0x7ff344013f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564e5bc60b9e385c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344003970 con 0x5623b4a9cf30 2024-04-22T00:10:54.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff344003b10 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff34c0054a0 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff344003d10 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 10 client.18510 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5623b46d50f0 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff34c0054a0 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff34401a070 con 0x5623b4a9cf30 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 10 client.18510 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 10 client.18510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 20 client.18510 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4ab68d0 0x5623b4ab8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 4 client.18510 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.626+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623b4a0eaa0 con 0x5623b4ab68d0 2024-04-22T00:10:54.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.627+0000 7ff351d9b640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4ab68d0 0x5623b4ab8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.627+0000 7ff351d9b640 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4ab68d0 0x5623b4ab8d60 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5f0ef31e20f88dd server_cookie=cff7f111fe0b4b04 in_seq=0 out_seq=0 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.627+0000 7ff339ffb640 10 client.18510 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff339ffb640 1 -- 172.21.15.38:0/3133932277 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623b4a0eaa0 con 0x5623b4ab68d0 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff339ffb640 10 client.18510 handle_command_reply: tid=0 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 client.18510 shutdown 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 msgr2=0x5623b4aa3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff344013a50 tx=0x7ff344013f10 comp rx=0 tx=0).stop 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 shutdown_connections 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4aa3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4ab68d0 0x5623b4ab8d60 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 --2- 172.21.15.38:0/3133932277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa3480 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff339ffb640 0 client.18510 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 >> 172.21.15.38:0/3133932277 conn(0x5623b4a97660 msgr2=0x5623b4a993b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 shutdown_connections 2024-04-22T00:10:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 1 -- 172.21.15.38:0/3133932277 wait complete. 2024-04-22T00:10:54.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.628+0000 7ff35b76c740 20 client.18510 trim_cache size 0 max 16384 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.645+0000 7ff35b76c740 1 Processor -- start 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.645+0000 7ff35b76c740 1 -- start start 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4ab20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541b0da0 con 0x5623b4a9cf30 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541b29a0 con 0x5623b4a9d850 2024-04-22T00:10:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35b76c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541b14a0 con 0x5623b4aa6430 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36956/0 (socket says 172.21.15.38:36956) 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 -- 172.21.15.38:0/2837952494 learned_addr learned my addr 172.21.15.38:0/2837952494 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff352d9d640 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4ab20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 -- 172.21.15.38:0/2837952494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 msgr2=0x5623b4ab20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4ab20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 -- 172.21.15.38:0/2837952494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 msgr2=0x5623b4aa23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 -- 172.21.15.38:0/2837952494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623b4aa9340 con 0x5623b4a9cf30 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff352d9d640 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4ab20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff35259c640 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff344006010 tx=0x7ff344011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27782d0667c8032a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff33bfff640 1 -- 172.21.15.38:0/2837952494 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34402f070 con 0x5623b4a9cf30 2024-04-22T00:10:54.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff33bfff640 1 -- 172.21.15.38:0/2837952494 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff344003970 con 0x5623b4a9cf30 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.646+0000 7ff33bfff640 1 -- 172.21.15.38:0/2837952494 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3440057b0 con 0x5623b4a9cf30 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- 172.21.15.38:0/2837952494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 msgr2=0x5623b4aa5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff344006010 tx=0x7ff344011f80 comp rx=0 tx=0).stop 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- 172.21.15.38:0/2837952494 shutdown_connections 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4aa6430 0x5623b4ab20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4a9d850 0x5623b4aa23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 --2- 172.21.15.38:0/2837952494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa5bb0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- 172.21.15.38:0/2837952494 >> 172.21.15.38:0/2837952494 conn(0x5623b4a97660 msgr2=0x5623b4a966e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- 172.21.15.38:0/2837952494 shutdown_connections 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- 172.21.15.38:0/2837952494 wait complete. 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 -- start start 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.647+0000 7ff35b76c740 1 Processor -- start 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4a9d850 0x5623b4aa0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4aa6430 0x5623b4b1bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541b22d0 con 0x5623b4a9cf30 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541b1ba0 con 0x5623b4aa6430 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541aeda0 con 0x5623b4a9d850 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 10 client.0 _ll_register_callbacks cb 0x7ff344009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36958/0 (socket says 172.21.15.38:36958) 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 -- 172.21.15.38:0/673946623 learned_addr learned my addr 172.21.15.38:0/673946623 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff351d9b640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4a9d850 0x5623b4aa0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff352d9d640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4aa6430 0x5623b4b1bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 -- 172.21.15.38:0/673946623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4a9d850 msgr2=0x5623b4aa0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4a9d850 0x5623b4aa0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 -- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4aa6430 msgr2=0x5623b4b1bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4aa6430 0x5623b4b1bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 -- 172.21.15.38:0/673946623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623b4aa12f0 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35259c640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff34402d040 tx=0x7ff344013700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950cf7c9388e212e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff339ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34402f070 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff3440050c0 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.648+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff344003970 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff344040050 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 10 client.18516 fetch_fsmap learned FSMap version 38 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5623b4b1d460 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff344003970 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7ff34401a030 con 0x5623b4a9cf30 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 10 client.18516 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:10:54.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 10 client.18516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 20 client.18516 populate_metadata read hostname 'smithi038' 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4b1dab0 0x5623b4b1fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 4 client.18516 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623b4a08df0 con 0x5623b4b1dab0 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.649+0000 7ff351d9b640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4b1dab0 0x5623b4b1fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff351d9b640 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4b1dab0 0x5623b4b1fea0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31706530ce8187a server_cookie=33bc32a0424f7c43 in_seq=0 out_seq=0 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff339ffb640 10 client.18516 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff339ffb640 1 -- 172.21.15.38:0/673946623 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5623b4a08df0 con 0x5623b4b1dab0 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff339ffb640 10 client.18516 handle_command_reply: tid=0 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 client.18516 shutdown 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 msgr2=0x5623b4aa04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff34402d040 tx=0x7ff344013700 comp rx=0 tx=0).stop 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 shutdown_connections 2024-04-22T00:10:54.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5623b4aa6430 0x5623b4b1bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623b4a9d850 0x5623b4aa0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5623b4b1dab0 0x5623b4b1fea0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 --2- 172.21.15.38:0/673946623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623b4a9cf30 0x5623b4aa04e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff339ffb640 0 client.18516 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 >> 172.21.15.38:0/673946623 conn(0x5623b4a97660 msgr2=0x5623b4a98a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 shutdown_connections 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 1 -- 172.21.15.38:0/673946623 wait complete. 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff35b76c740 20 client.18516 trim_cache size 0 max 16384 2024-04-22T00:10:54.655 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "e4b3f296-c4db-4083-b980-deb18af48413": { 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "e4b3f296-c4db-4083-b980-deb18af48413", 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:10:54.656 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 msgr2=0x7ff33c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 0x7ff33c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5623b4aa8df0 tx=0x7ff34c04a000 comp rx=0 tx=0).stop 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 msgr2=0x7ff35419a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.650+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7ff34800d950 tx=0x7ff34800de10 comp rx=0 tx=0).stop 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 shutdown_connections 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff33c0a2d00 0x7ff33c0a51c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff35410dfd0 0x7ff35419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff354074410 0x7ff35419a770 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 --2- 172.21.15.38:0/1942548535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff354073af0 0x7ff35419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:10:54.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 >> 172.21.15.38:0/1942548535 conn(0x7ff3540fd180 msgr2=0x7ff3540fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:10:54.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 shutdown_connections 2024-04-22T00:10:54.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:10:54.651+0000 7ff358aa7640 1 -- 172.21.15.38:0/1942548535 wait complete. 2024-04-22T00:10:54.659 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b3f296-c4db-4083-b980-deb18af48413 - {'path': '/', 'tag': 'e4b3f296-c4db-4083-b980-deb18af48413', 'options': 'recursive,force'} 2024-04-22T00:10:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:54 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:10:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:54 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:54 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:10:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:54 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:55.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:54 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:10:55.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:54 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:55.284 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:54 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:10:55.284 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:54 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:55.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:54 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:10:55.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:54 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:55.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:54 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:10:55.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:54 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:10:56.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:55 smithi038 ceph-mon[27972]: pgmap v272: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T00:10:56.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:55 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:10:56.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:55 smithi203 ceph-mon[31431]: pgmap v272: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T00:10:56.283 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:55 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:10:56.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:55 smithi111 ceph-mon[31543]: pgmap v272: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T00:10:56.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:55 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:10:58.330 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:10:58 smithi111 ceph-mon[31543]: pgmap v273: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:10:58.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:10:58 smithi038 ceph-mon[27972]: pgmap v273: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:10:58.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:10:58 smithi203 ceph-mon[31431]: pgmap v273: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:11:00.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:00 smithi111 ceph-mon[31543]: pgmap v274: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:00.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:00 smithi038 ceph-mon[27972]: pgmap v274: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:00.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:00 smithi203 ceph-mon[31431]: pgmap v274: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:02.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:02 smithi111 ceph-mon[31543]: pgmap v275: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:11:02.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:02 smithi038 ceph-mon[27972]: pgmap v275: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:11:02.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:02 smithi203 ceph-mon[31431]: pgmap v275: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:11:04.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:04 smithi111 ceph-mon[31543]: pgmap v276: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T00:11:04.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:04 smithi038 ceph-mon[27972]: pgmap v276: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T00:11:04.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:04 smithi203 ceph-mon[31431]: pgmap v276: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-22T00:11:06.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:06 smithi038 ceph-mon[27972]: pgmap v277: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:11:06.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:06 smithi203 ceph-mon[31431]: pgmap v277: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:11:06.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:06 smithi111 ceph-mon[31543]: pgmap v277: 129 pgs: 129 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:11:07.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:07 smithi038 ceph-mon[27972]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T00:11:07.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:07 smithi203 ceph-mon[31431]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T00:11:07.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:07 smithi111 ceph-mon[31543]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T00:11:08.434 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:08 smithi203 ceph-mon[31431]: pgmap v279: 129 pgs: 25 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 93 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:11:08.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:08 smithi038 ceph-mon[27972]: pgmap v279: 129 pgs: 25 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 93 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:11:08.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:08 smithi111 ceph-mon[31543]: pgmap v279: 129 pgs: 25 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 93 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:11:10.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:10 smithi038 ceph-mon[27972]: pgmap v280: 129 pgs: 58 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 50 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:10.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:10 smithi203 ceph-mon[31431]: pgmap v280: 129 pgs: 58 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 50 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:10.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:10 smithi111 ceph-mon[31543]: pgmap v280: 129 pgs: 58 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 50 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:12.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:12 smithi038 ceph-mon[27972]: pgmap v281: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:12.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:12 smithi203 ceph-mon[31431]: pgmap v281: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:12.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:12 smithi111 ceph-mon[31543]: pgmap v281: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:11:14.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:14 smithi038 ceph-mon[27972]: pgmap v282: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-22T00:11:14.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:14 smithi203 ceph-mon[31431]: pgmap v282: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-22T00:11:14.580 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:14 smithi111 ceph-mon[31543]: pgmap v282: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 29 op/s 2024-04-22T00:11:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:16 smithi038 ceph-mon[27972]: pgmap v283: 129 pgs: 21 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 101 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:11:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:16 smithi203 ceph-mon[31431]: pgmap v283: 129 pgs: 21 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 101 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:11:16.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:16 smithi111 ceph-mon[31543]: pgmap v283: 129 pgs: 21 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 101 active+clean; 15 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:11:18.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:18 smithi038 ceph-mon[27972]: pgmap v284: 129 pgs: 7 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 120 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 187 B/s wr, 12 op/s 2024-04-22T00:11:18.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:18 smithi203 ceph-mon[31431]: pgmap v284: 129 pgs: 7 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 120 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 187 B/s wr, 12 op/s 2024-04-22T00:11:18.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:18 smithi111 ceph-mon[31543]: pgmap v284: 129 pgs: 7 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 120 active+clean; 15 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 187 B/s wr, 12 op/s 2024-04-22T00:11:20.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:20 smithi038 ceph-mon[27972]: pgmap v285: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:11:20.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:20 smithi203 ceph-mon[31431]: pgmap v285: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:11:20.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:20 smithi111 ceph-mon[31543]: pgmap v285: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T00:11:22.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:22 smithi038 ceph-mon[27972]: pgmap v286: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T00:11:22.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:22 smithi038 ceph-mon[27972]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T00:11:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:22 smithi203 ceph-mon[31431]: pgmap v286: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T00:11:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:22 smithi203 ceph-mon[31431]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T00:11:22.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:22 smithi111 ceph-mon[31543]: pgmap v286: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T00:11:22.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:22 smithi111 ceph-mon[31543]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T00:11:24.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:11:24.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:11:24.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:24 smithi038 ceph-mon[27972]: pgmap v288: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:24.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3975042623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100fd0 msgr2=0x7f1acc10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3975042623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100fd0 0x7f1acc10d4e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009cd0 tx=0x7f1ac402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3975042623 shutdown_connections 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3975042623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100fd0 0x7f1acc10d4e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3975042623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc100610 0x7f1acc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3975042623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc1066b0 0x7f1acc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3975042623 >> 172.21.15.38:0/3975042623 conn(0x7f1acc077740 msgr2=0x7f1acc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3975042623 shutdown_connections 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3975042623 wait complete. 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 Processor -- start 2024-04-22T00:11:24.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.781+0000 7f1ad31b8640 1 -- start start 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 0x7f1acc199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc1066b0 0x7f1acc1a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1109f0 con 0x7f1acc100610 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc110870 con 0x7f1acc1066b0 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad31b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc110b70 con 0x7f1acc100fd0 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33106/0 (socket says 172.21.15.38:33106) 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 -- 172.21.15.38:0/3487704636 learned_addr learned my addr 172.21.15.38:0/3487704636 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc1066b0 0x7f1acc1a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 0x7f1acc199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 -- 172.21.15.38:0/3487704636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 msgr2=0x7f1acc199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 0x7f1acc199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 -- 172.21.15.38:0/3487704636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc1066b0 msgr2=0x7f1acc1a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc1066b0 0x7f1acc1a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 -- 172.21.15.38:0/3487704636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc1a4390 con 0x7f1acc100610 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 0x7f1acc199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.782+0000 7f1ad0f54640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9c0 tx=0x7f1ac800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.783+0000 7f1ac1ffb640 1 -- 172.21.15.38:0/3487704636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cd80 con 0x7f1acc100610 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.783+0000 7f1ac1ffb640 1 -- 172.21.15.38:0/3487704636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac80035f0 con 0x7f1acc100610 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.783+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc1a4560 con 0x7f1acc100610 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.783+0000 7f1ac1ffb640 1 -- 172.21.15.38:0/3487704636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8010690 con 0x7f1acc100610 2024-04-22T00:11:24.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.783+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc1a4a00 con 0x7f1acc100610 2024-04-22T00:11:24.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:24 smithi203 ceph-mon[31431]: pgmap v288: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.785+0000 7f1ac1ffb640 1 -- 172.21.15.38:0/3487704636 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f1ac80037a0 con 0x7f1acc100610 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.785+0000 7f1ad5d9e740 1 Processor -- start 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.785+0000 7f1ad5d9e740 1 -- start start 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 0x55cfa1a9e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac1ffb640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 0x7f1ab40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cfa1a02560 con 0x55cfa1aa7430 2024-04-22T00:11:24.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfa19aa730 con 0x55cfa1a9e850 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cfa19fe520 con 0x55cfa1a9df30 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad0f54640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 0x7f1ab40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33110/0 (socket says 172.21.15.38:33110) 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad1755640 1 -- 172.21.15.38:0/1404382366 learned_addr learned my addr 172.21.15.38:0/1404382366 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac1ffb640 1 -- 172.21.15.38:0/3487704636 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1ac80cfb40 con 0x7f1acc100610 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad1755640 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 0x55cfa1a9e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 -- 172.21.15.38:0/1404382366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 msgr2=0x55cfa1a9e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 0x55cfa1a9e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 -- 172.21.15.38:0/1404382366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 msgr2=0x55cfa1aa9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 -- 172.21.15.38:0/1404382366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa17e07d0 con 0x55cfa1a9e850 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad1755640 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 0x55cfa1a9e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ad0f54640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 0x7f1ab40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80980d0 tx=0x7f1ac801bb30 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.786+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa6ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0693f0 tx=0x7f1abc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1e360a8c66d241 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ab2ffd640 1 -- 172.21.15.38:0/1404382366 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc004030 con 0x55cfa1a9e850 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ab2ffd640 1 -- 172.21.15.38:0/1404382366 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc005500 con 0x55cfa1a9e850 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ab2ffd640 1 -- 172.21.15.38:0/1404382366 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc004c00 con 0x55cfa1a9e850 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/1404382366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 msgr2=0x55cfa1aa6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa6ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0693f0 tx=0x7f1abc00f730 comp rx=0 tx=0).stop 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/1404382366 shutdown_connections 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa6ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/1404382366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9df30 0x55cfa1a9e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/1404382366 >> 172.21.15.38:0/1404382366 conn(0x55cfa1a98660 msgr2=0x55cfa1a98a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/1404382366 shutdown_connections 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.787+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/1404382366 wait complete. 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 -- start start 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 Processor -- start 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cfa1aaa220 con 0x55cfa1aa7430 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cfa19bde00 con 0x55cfa1a9df30 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cfa1aaa3a0 con 0x55cfa1a9e850 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.788+0000 7f1ad5d9e740 10 client.0 _ll_register_callbacks cb 0x55cfa19f8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33112/0 (socket says 172.21.15.38:33112) 2024-04-22T00:11:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 -- 172.21.15.38:0/3988340614 learned_addr learned my addr 172.21.15.38:0/3988340614 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 -- 172.21.15.38:0/3988340614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 msgr2=0x55cfa1aa49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 -- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 msgr2=0x55cfa1aa4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 -- 172.21.15.38:0/3988340614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa17163f0 con 0x55cfa1aa7430 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2171f734ff652f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1aabfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4047070 con 0x55cfa1aa7430 2024-04-22T00:11:24.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1ac4003970 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1abc005500 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1ac4039040 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad5d9e740 10 client.18528 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.789+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cfa16d60f0 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc005500 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1ac403f070 con 0x55cfa1aa7430 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 10 client.18528 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 10 client.18528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 20 client.18528 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1ab7700 0x55cfa1ab9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 4 client.18528 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfa1a0faa0 con 0x55cfa1ab7700 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.790+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1ab7700 0x55cfa1ab9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.791+0000 7f1aabfff640 10 client.18528 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.791+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1ab7700 0x55cfa1ab9b60 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c1d9466a37eadcd server_cookie=35154fca3ffe69c9 in_seq=0 out_seq=0 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1aabfff640 1 -- 172.21.15.38:0/3988340614 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfa1a0faa0 con 0x55cfa1ab7700 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1aabfff640 10 client.18528 handle_command_reply: tid=0 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 client.18528 shutdown 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 msgr2=0x55cfa1aa4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4036000 tx=0x7f1ac402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 shutdown_connections 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1ab7700 0x55cfa1ab9b60 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa4ee0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aa49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1aabfff640 0 client.18528 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3988340614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 >> 172.21.15.38:0/3988340614 conn(0x55cfa1a98660 msgr2=0x55cfa1a9a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 shutdown_connections 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3988340614 wait complete. 2024-04-22T00:11:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.792+0000 7f1ad5d9e740 20 client.18528 trim_cache size 0 max 16384 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 Processor -- start 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 -- start start 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1ab30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1aec30 con 0x55cfa1a9df30 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1b0830 con 0x55cfa1aa7430 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1af330 con 0x55cfa1a9e850 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33122/0 (socket says 172.21.15.38:33122) 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 -- 172.21.15.38:0/3479275864 learned_addr learned my addr 172.21.15.38:0/3479275864 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 -- 172.21.15.38:0/3479275864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 msgr2=0x55cfa1ab30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1ab30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1ab30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 -- 172.21.15.38:0/3479275864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1aa7430 msgr2=0x55cfa1aa35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.810+0000 7f1ad1755640 1 -- 172.21.15.38:0/3479275864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfa1a98490 con 0x55cfa1a9df30 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad1755640 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009890 tx=0x7f1ac4009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40f8dcfa2138614 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ab27fc640 1 -- 172.21.15.38:0/3479275864 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4047040 con 0x55cfa1a9df30 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ab27fc640 1 -- 172.21.15.38:0/3479275864 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac40349f0 con 0x55cfa1a9df30 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ab27fc640 1 -- 172.21.15.38:0/3479275864 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac404c7c0 con 0x55cfa1a9df30 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3479275864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 msgr2=0x55cfa1aa6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009890 tx=0x7f1ac4009540 comp rx=0 tx=0).stop 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3479275864 shutdown_connections 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1aa7430 0x55cfa1aa35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1ab30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/3479275864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1a9df30 0x55cfa1aa6b20 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3479275864 >> 172.21.15.38:0/3479275864 conn(0x55cfa1a98660 msgr2=0x55cfa1a96e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3479275864 shutdown_connections 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.811+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/3479275864 wait complete. 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 -- start start 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 Processor -- start 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aaf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aafb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1b0160 con 0x55cfa1aa7430 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1afa30 con 0x55cfa1a9df30 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1acc30 con 0x55cfa1a9e850 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad5d9e740 10 client.0 _ll_register_callbacks cb 0x7f1ac4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad1755640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33126/0 (socket says 172.21.15.38:33126) 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ad1755640 1 -- 172.21.15.38:0/4056688476 learned_addr learned my addr 172.21.15.38:0/4056688476 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.812+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aafb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aaf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 -- 172.21.15.38:0/4056688476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 msgr2=0x55cfa1aafb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aafb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 -- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 msgr2=0x55cfa1aaf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aaf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 -- 172.21.15.38:0/4056688476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfa1aa9820 con 0x55cfa1aa7430 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad1755640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f1ac404c4a0 tx=0x7f1ac402c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d6a9db12fbe939 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1aabfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac404c540 con 0x55cfa1aa7430 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac40349f0 con 0x55cfa1aa7430 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1ac404c6e0 con 0x55cfa1aa7430 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f1ac404c8e0 con 0x55cfa1aa7430 2024-04-22T00:11:24.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad5d9e740 10 client.18534 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55cfa1b1d730 con 0x55cfa1aa7430 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.813+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac40349f0 con 0x55cfa1aa7430 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f1ac403f040 con 0x55cfa1aa7430 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 10 client.18534 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 10 client.18534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 20 client.18534 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1b1dbb0 0x55cfa1b1ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 4 client.18534 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cfa1a09df0 con 0x55cfa1b1dbb0 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1b1dbb0 0x55cfa1b1ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1aabfff640 10 client.18534 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ac3fff640 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1b1dbb0 0x55cfa1b1ffa0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ce4ddc9d7aa1ec server_cookie=1342057e59e5a779 in_seq=0 out_seq=0 2024-04-22T00:11:24.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1aabfff640 1 -- 172.21.15.38:0/4056688476 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cfa1a09df0 con 0x55cfa1b1dbb0 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1aabfff640 10 client.18534 handle_command_reply: tid=0 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.814+0000 7f1ad5d9e740 1 client.18534 shutdown 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 msgr2=0x55cfa1aaab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f1ac404c4a0 tx=0x7f1ac402c770 comp rx=0 tx=0).stop 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 shutdown_connections 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cfa1b1dbb0 0x55cfa1b1ffa0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cfa1aa7430 0x55cfa1aaab70 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cfa1a9e850 0x55cfa1aafb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 --2- 172.21.15.38:0/4056688476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cfa1a9df30 0x55cfa1aaf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1aabfff640 0 client.18534 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 >> 172.21.15.38:0/4056688476 conn(0x55cfa1a98660 msgr2=0x55cfa1aa4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 shutdown_connections 2024-04-22T00:11:24.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 1 -- 172.21.15.38:0/4056688476 wait complete. 2024-04-22T00:11:24.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad5d9e740 20 client.18534 trim_cache size 0 max 16384 2024-04-22T00:11:24.819 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:24.819 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:11:24.819 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:11:24.819 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 msgr2=0x7f1ab40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 0x7f1ab40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80980d0 tx=0x7f1ac801bb30 comp rx=0 tx=0).stop 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 msgr2=0x7f1acc198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9c0 tx=0x7f1ac800ee80 comp rx=0 tx=0).stop 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 shutdown_connections 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f1ab40a2d00 0x7f1ab40a51c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1acc1066b0 0x7f1acc1a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1acc100fd0 0x7f1acc199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 --2- 172.21.15.38:0/3487704636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1acc100610 0x7f1acc198dd0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.815+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 >> 172.21.15.38:0/3487704636 conn(0x7f1acc077740 msgr2=0x7f1acc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.816+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 shutdown_connections 2024-04-22T00:11:24.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.816+0000 7f1ad31b8640 1 -- 172.21.15.38:0/3487704636 wait complete. 2024-04-22T00:11:24.824 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:11:24.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:11:24.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:24 smithi111 ceph-mon[31543]: pgmap v288: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:24.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 -- 172.21.15.38:0/2259335591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 msgr2=0x7feca4107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/2259335591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca4107b00 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fec98009930 tx=0x7fec9802bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:24.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 -- 172.21.15.38:0/2259335591 shutdown_connections 2024-04-22T00:11:24.969 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/2259335591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 0x7feca410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/2259335591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/2259335591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca4107b00 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 -- 172.21.15.38:0/2259335591 >> 172.21.15.38:0/2259335591 conn(0x7feca4078b00 msgr2=0x7feca4078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 -- 172.21.15.38:0/2259335591 shutdown_connections 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.969+0000 7fecaa4c2640 1 -- 172.21.15.38:0/2259335591 wait complete. 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 Processor -- start 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 -- start start 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 0x7feca4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111b80 con 0x7feca4107720 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111a00 con 0x7feca41020d0 2024-04-22T00:11:24.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7fecaa4c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feca4111d00 con 0x7feca4101710 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7feca8a5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7feca8a5f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33136/0 (socket says 172.21.15.38:33136) 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7feca8a5f640 1 -- 172.21.15.38:0/3544698532 learned_addr learned my addr 172.21.15.38:0/3544698532 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7feca3fff640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.970+0000 7feca37fe640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 0x7feca4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 -- 172.21.15.38:0/3544698532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 msgr2=0x7feca4073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 -- 172.21.15.38:0/3544698532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 msgr2=0x7feca4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:24.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 0x7feca4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 -- 172.21.15.38:0/3544698532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca41a6580 con 0x7feca4107720 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca3fff640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca8a5f640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00eaa0 tx=0x7fec9c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00cdb0 con 0x7feca4107720 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca41a6870 con 0x7feca4107720 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9c005020 con 0x7feca4107720 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0116b0 con 0x7feca4107720 2024-04-22T00:11:24.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.971+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41a6dc0 con 0x7feca4107720 2024-04-22T00:11:24.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.972+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca40689b0 con 0x7feca4107720 2024-04-22T00:11:24.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.973+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fec9c005560 con 0x7feca4107720 2024-04-22T00:11:24.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.974+0000 7feca17fa640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 0x7fec8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:24.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.974+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fec9c0cfd00 con 0x7feca4107720 2024-04-22T00:11:24.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.974+0000 7feca3fff640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 0x7fec8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:24.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.975+0000 7feca3fff640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 0x7fec8c0a51c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec98009900 tx=0x7fec9802df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:24.975 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:24.978+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fec9c016030 con 0x7feca4107720 2024-04-22T00:11:25.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.201+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feca4109c80 con 0x7feca4107720 2024-04-22T00:11:25.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.202+0000 7feca17fa640 1 -- 172.21.15.38:0/3544698532 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11941 (secure 0 0 0) 0x7fec9c09bf40 con 0x7feca4107720 2024-04-22T00:11:25.199 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:11:25.200 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:09:42.375340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:11:25.200 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 38 2024-04-22T00:11:25.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 msgr2=0x7fec8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 0x7fec8c0a51c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fec98009900 tx=0x7fec9802df90 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 msgr2=0x7feca40744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00eaa0 tx=0x7fec9c00ef60 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 shutdown_connections 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fec8c0a2d00 0x7fec8c0a51c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feca4107720 0x7feca40744c0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feca41020d0 0x7feca4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 --2- 172.21.15.38:0/3544698532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feca4101710 0x7feca4073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 >> 172.21.15.38:0/3544698532 conn(0x7feca4078b00 msgr2=0x7feca41041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 shutdown_connections 2024-04-22T00:11:25.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.204+0000 7fecaa4c2640 1 -- 172.21.15.38:0/3544698532 wait complete. 2024-04-22T00:11:25.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.320+0000 7fd5d408d640 1 -- 172.21.15.38:0/2321300424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 msgr2=0x7fd5cc101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.320+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2321300424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc101b90 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009930 tx=0x7fd5c002bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- 172.21.15.38:0/2321300424 shutdown_connections 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2321300424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc1020d0 0x7fd5cc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2321300424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc101b90 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2321300424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc107720 0x7fd5cc107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- 172.21.15.38:0/2321300424 >> 172.21.15.38:0/2321300424 conn(0x7fd5cc0fd180 msgr2=0x7fd5cc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- 172.21.15.38:0/2321300424 shutdown_connections 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- 172.21.15.38:0/2321300424 wait complete. 2024-04-22T00:11:25.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 Processor -- start 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- start start 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc1020d0 0x7fd5cc198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc107720 0x7fd5cc19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc111b30 con 0x7fd5cc101710 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1119b0 con 0x7fd5cc1020d0 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.321+0000 7fd5d408d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc111cb0 con 0x7fd5cc107720 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33162/0 (socket says 172.21.15.38:33162) 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2597408815 learned_addr learned my addr 172.21.15.38:0/2597408815 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1628640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc1020d0 0x7fd5cc198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d262a640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc107720 0x7fd5cc19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2597408815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc107720 msgr2=0x7fd5cc19c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc107720 0x7fd5cc19c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2597408815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc1020d0 msgr2=0x7fd5cc198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc1020d0 0x7fd5cc198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2597408815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5cc19d0c0 con 0x7fd5cc101710 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400d950 tx=0x7fd5c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5caffd640 1 -- 172.21.15.38:0/2597408815 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4017070 con 0x7fd5cc101710 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5cc19d3b0 con 0x7fd5cc101710 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5caffd640 1 -- 172.21.15.38:0/2597408815 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4009c30 con 0x7fd5cc101710 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5caffd640 1 -- 172.21.15.38:0/2597408815 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400f6b0 con 0x7fd5cc101710 2024-04-22T00:11:25.322 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.322+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5cc1a4c40 con 0x7fd5cc101710 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.324+0000 7fd5caffd640 1 -- 172.21.15.38:0/2597408815 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd5c40109d0 con 0x7fd5cc101710 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 Processor -- start 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 -- start start 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5caffd640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 0x7fd5b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5357b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd5357dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fd534d2530 con 0x55fd5356da00 2024-04-22T00:11:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fd5347a830 con 0x55fd5356e3b0 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fd534ce4f0 con 0x55fd5357b930 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5caffd640 1 -- 172.21.15.38:0/2597408815 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd5c40cf2c0 con 0x7fd5cc101710 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d262a640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 0x7fd5b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.325+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33164/0 (socket says 172.21.15.38:33164) 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 -- 172.21.15.38:0/1595417602 learned_addr learned my addr 172.21.15.38:0/1595417602 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1628640 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5357b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d262a640 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd5357dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 -- 172.21.15.38:0/1595417602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 msgr2=0x55fd5357dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd5357dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 -- 172.21.15.38:0/1595417602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 msgr2=0x55fd5357b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5357b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 -- 172.21.15.38:0/1595417602 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd5349e2c0 con 0x55fd5356da00 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d262a640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 0x7fd5b40a51c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005fd0 tx=0x7fd5bc013940 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4097960 tx=0x7fd5c409b160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752c8317a5f7bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5b3fff640 1 -- 172.21.15.38:0/1595417602 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4020070 con 0x55fd5356da00 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5b3fff640 1 -- 172.21.15.38:0/1595417602 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4015d80 con 0x55fd5356da00 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.326+0000 7fd5b3fff640 1 -- 172.21.15.38:0/1595417602 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c40a40c0 con 0x55fd5356da00 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- 172.21.15.38:0/1595417602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 msgr2=0x55fd5356dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4097960 tx=0x7fd5c409b160 comp rx=0 tx=0).stop 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- 172.21.15.38:0/1595417602 shutdown_connections 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd5357dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5357b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/1595417602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd5356dde0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- 172.21.15.38:0/1595417602 >> 172.21.15.38:0/1595417602 conn(0x55fd53568750 msgr2=0x55fd53568b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- 172.21.15.38:0/1595417602 shutdown_connections 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- 172.21.15.38:0/1595417602 wait complete. 2024-04-22T00:11:25.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 -- start start 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.327+0000 7fd5d6c71740 1 Processor -- start 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 0x55fd5357b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd53578650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fd5357e690 con 0x55fd5357b930 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55fd5348df10 con 0x55fd5356da00 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fd5357e810 con 0x55fd5356e3b0 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d6c71740 10 client.0 _ll_register_callbacks cb 0x55fd531e5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 0x55fd5357b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33170/0 (socket says 172.21.15.38:33170) 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 -- 172.21.15.38:0/4092244914 learned_addr learned my addr 172.21.15.38:0/4092244914 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d1628640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd53578650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 -- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 msgr2=0x55fd53578650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd53578650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 -- 172.21.15.38:0/4092244914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 msgr2=0x55fd5357b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 0x55fd5357b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 -- 172.21.15.38:0/4092244914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd5345fc80 con 0x55fd5357b930 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 0x55fd5357b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5d262a640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc003f50 tx=0x7fd5bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60160e258f9ef225 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.328+0000 7fd5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc023070 con 0x55fd5357b930 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd5bc00f560 con 0x55fd5357b930 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c4015d80 con 0x55fd5357b930 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc00f760 con 0x55fd5357b930 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 10 client.18552 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd531e63f0 con 0x55fd5357b930 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4015d80 con 0x55fd5357b930 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fd5bc010c40 con 0x55fd5357b930 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 10 client.18552 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 10 client.18552 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 20 client.18552 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd5358b940 0x55fd5358dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 4 client.18552 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd534dfa70 con 0x55fd5358b940 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.329+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd5358b940 0x55fd5358dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.330+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd5358b940 0x55fd5358dd30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=21ed7b3fd4b7c4f4 server_cookie=2fadc4c1b3ea98a3 in_seq=0 out_seq=0 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.330+0000 7fd5b17fa640 10 client.18552 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:25.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5b17fa640 1 -- 172.21.15.38:0/4092244914 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd534dfa70 con 0x55fd5358b940 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5b17fa640 10 client.18552 handle_command_reply: tid=0 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 client.18552 shutdown 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 msgr2=0x55fd53578bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc003f50 tx=0x7fd5bc004ab0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 shutdown_connections 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd5358b940 0x55fd5358dd30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5357b930 0x55fd53578bd0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd53578650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5b17fa640 0 client.18552 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/4092244914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356da00 0x55fd5357b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 >> 172.21.15.38:0/4092244914 conn(0x55fd53568750 msgr2=0x55fd5356a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 shutdown_connections 2024-04-22T00:11:25.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 1 -- 172.21.15.38:0/4092244914 wait complete. 2024-04-22T00:11:25.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.331+0000 7fd5d6c71740 20 client.18552 trim_cache size 0 max 16384 2024-04-22T00:11:25.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.344+0000 7fd5d6c71740 1 Processor -- start 2024-04-22T00:11:25.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.344+0000 7fd5d6c71740 1 -- start start 2024-04-22T00:11:25.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.344+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.348 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.344+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5356a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd53593620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1acaf0 con 0x55fd5356da00 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1ae6f0 con 0x55fd5356e3b0 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1ad1f0 con 0x55fd5357b930 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33176/0 (socket says 172.21.15.38:33176) 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 -- 172.21.15.38:0/3123984815 learned_addr learned my addr 172.21.15.38:0/3123984815 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 -- 172.21.15.38:0/3123984815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 msgr2=0x55fd53593620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd53593620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 -- 172.21.15.38:0/3123984815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 msgr2=0x55fd5356a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5356a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.349 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 -- 172.21.15.38:0/3123984815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd532b07d0 con 0x55fd5356da00 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd5c409b490 tx=0x7fd5c4009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356dfb77b979d2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5b37fe640 1 -- 172.21.15.38:0/3123984815 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4020040 con 0x55fd5356da00 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5b37fe640 1 -- 172.21.15.38:0/3123984815 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4084070 con 0x55fd5356da00 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.345+0000 7fd5b37fe640 1 -- 172.21.15.38:0/3123984815 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c40cca60 con 0x55fd5356da00 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- 172.21.15.38:0/3123984815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 msgr2=0x55fd53587590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd5c409b490 tx=0x7fd5c4009f20 comp rx=0 tx=0).stop 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- 172.21.15.38:0/3123984815 shutdown_connections 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5357b930 0x55fd53593620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5356e3b0 0x55fd5356a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/3123984815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53587590 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- 172.21.15.38:0/3123984815 >> 172.21.15.38:0/3123984815 conn(0x55fd53568750 msgr2=0x55fd53569b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- 172.21.15.38:0/3123984815 shutdown_connections 2024-04-22T00:11:25.350 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- 172.21.15.38:0/3123984815 wait complete. 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 -- start start 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.346+0000 7fd5d6c71740 1 Processor -- start 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd5357f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 0x55fd5357f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1ae020 con 0x55fd5356da00 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1ad8f0 con 0x55fd5357b930 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc1aaaf0 con 0x55fd5356e3b0 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 10 client.0 _ll_register_callbacks cb 0x7fd5bc012050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33182/0 (socket says 172.21.15.38:33182) 2024-04-22T00:11:25.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2345034448 learned_addr learned my addr 172.21.15.38:0/2345034448 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d262a640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 0x55fd5357f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1628640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd5357f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 msgr2=0x55fd5357f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd5357f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2345034448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 msgr2=0x55fd5357f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 0x55fd5357f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 -- 172.21.15.38:0/2345034448 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd5357fcd0 con 0x55fd5356da00 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d262a640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 0x55fd5357f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d1e29640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4097ed0 tx=0x7fd5c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929a6c0065e9db91 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4020050 con 0x55fd5356da00 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd5c4097b70 con 0x55fd5356da00 2024-04-22T00:11:25.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c4084070 con 0x55fd5356da00 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd5c40b4230 con 0x55fd5356da00 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 10 client.18558 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.347+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55fd535f1d50 con 0x55fd5356da00 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4084070 con 0x55fd5356da00 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fd5c401c630 con 0x55fd5356da00 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 10 client.18558 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 10 client.18558 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 20 client.18558 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd535f1d50 0x55fd535f4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 4 client.18558 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd534d9dc0 con 0x55fd535f1d50 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.348+0000 7fd5d1628640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd535f1d50 0x55fd535f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d1628640 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd535f1d50 0x55fd535f4140 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b70fcbea000e524 server_cookie=e58eb53b620151a8 in_seq=0 out_seq=0 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5b17fa640 10 client.18558 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5b17fa640 1 -- 172.21.15.38:0/2345034448 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd534d9dc0 con 0x55fd535f1d50 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5b17fa640 10 client.18558 handle_command_reply: tid=0 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 client.18558 shutdown 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 msgr2=0x55fd53583d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4097ed0 tx=0x7fd5c4005880 comp rx=0 tx=0).stop 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 shutdown_connections 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55fd535f1d50 0x55fd535f4140 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fd5357b930 0x55fd5357f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd5356e3b0 0x55fd5357f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 --2- 172.21.15.38:0/2345034448 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fd5356da00 0x55fd53583d30 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 >> 172.21.15.38:0/2345034448 conn(0x55fd53568750 msgr2=0x55fd53573c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5b17fa640 0 client.18558 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:25.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 shutdown_connections 2024-04-22T00:11:25.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 1 -- 172.21.15.38:0/2345034448 wait complete. 2024-04-22T00:11:25.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d6c71740 20 client.18558 trim_cache size 0 max 16384 2024-04-22T00:11:25.355 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 msgr2=0x7fd5b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 0x7fd5b40a51c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005fd0 tx=0x7fd5bc013940 comp rx=0 tx=0).stop 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 msgr2=0x7fd5cc1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400d950 tx=0x7fd5c400de10 comp rx=0 tx=0).stop 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.349+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 shutdown_connections 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5b40a2d00 0x7fd5b40a51c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.356 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5cc107720 0x7fd5cc19c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5cc1020d0 0x7fd5cc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 --2- 172.21.15.38:0/2597408815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5cc101710 0x7fd5cc1980d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 >> 172.21.15.38:0/2597408815 conn(0x7fd5cc0fd180 msgr2=0x7fd5cc0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 shutdown_connections 2024-04-22T00:11:25.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.350+0000 7fd5d408d640 1 -- 172.21.15.38:0/2597408815 wait complete. 2024-04-22T00:11:25.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:11:25.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:25 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3544698532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:25.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 -- 172.21.15.38:0/4288914366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 msgr2=0x7fcddc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/4288914366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc10e1a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4009900 tx=0x7fcdd402bd60 comp rx=0 tx=0).stop 2024-04-22T00:11:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 -- 172.21.15.38:0/4288914366 shutdown_connections 2024-04-22T00:11:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/4288914366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc10e1a0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/4288914366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/4288914366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 0x7fcddc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 -- 172.21.15.38:0/4288914366 >> 172.21.15.38:0/4288914366 conn(0x7fcddc077740 msgr2=0x7fcddc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 -- 172.21.15.38:0/4288914366 shutdown_connections 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.466+0000 7fcde0fcb640 1 -- 172.21.15.38:0/4288914366 wait complete. 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 Processor -- start 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 -- start start 2024-04-22T00:11:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 0x7fcddc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc111bb0 con 0x7fcddc10bd00 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc111a30 con 0x7fcddc1029c0 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcde0fcb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc111d30 con 0x7fcddc1032e0 2024-04-22T00:11:25.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcddad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcddad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33198/0 (socket says 172.21.15.38:33198) 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcddad9d640 1 -- 172.21.15.38:0/19429008 learned_addr learned my addr 172.21.15.38:0/19429008 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.467+0000 7fcdda59c640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 0x7fcddc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 -- 172.21.15.38:0/19429008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 msgr2=0x7fcddc100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 -- 172.21.15.38:0/19429008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 msgr2=0x7fcddc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 0x7fcddc100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 -- 172.21.15.38:0/19429008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc1a64f0 con 0x7fcddc10bd00 2024-04-22T00:11:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcddad9d640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4004f50 tx=0x7fcdd402c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcdcb7fe640 1 -- 172.21.15.38:0/19429008 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4043020 con 0x7fcddc10bd00 2024-04-22T00:11:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcdcb7fe640 1 -- 172.21.15.38:0/19429008 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd40054e0 con 0x7fcddc10bd00 2024-04-22T00:11:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1a6780 con 0x7fcddc10bd00 2024-04-22T00:11:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcdcb7fe640 1 -- 172.21.15.38:0/19429008 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4003d30 con 0x7fcddc10bd00 2024-04-22T00:11:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.468+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1a6cd0 con 0x7fcddc10bd00 2024-04-22T00:11:25.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.471+0000 7fcdcb7fe640 1 -- 172.21.15.38:0/19429008 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcdd4046020 con 0x7fcddc10bd00 2024-04-22T00:11:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.471+0000 7fcde3bb0740 1 Processor -- start 2024-04-22T00:11:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 -- start start 2024-04-22T00:11:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d124400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d12d520 0x55b51d12f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b51d088530 con 0x55b51d12d520 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b51d030830 con 0x55b51d124940 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b51d0844f0 con 0x55b51d124020 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdcb7fe640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 0x7fcdc40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdd9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d12d520 0x55b51d12f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:46452/0 (socket says 172.21.15.38:46452) 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 -- 172.21.15.38:0/2187575892 learned_addr learned my addr 172.21.15.38:0/2187575892 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcddad9d640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 0x7fcdc40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d124400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 -- 172.21.15.38:0/2187575892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 msgr2=0x55b51d124400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d124400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 -- 172.21.15.38:0/2187575892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d12d520 msgr2=0x55b51d12f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d12d520 0x55b51d12f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdcb7fe640 1 -- 172.21.15.38:0/19429008 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fcdd40f1900 con 0x7fcddc10bd00 2024-04-22T00:11:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.472+0000 7fcdda59c640 1 -- 172.21.15.38:0/2187575892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51d0542c0 con 0x55b51d124940 2024-04-22T00:11:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.473+0000 7fcdda59c640 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00a4b0 tx=0x7fcdcc0096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c3624385c2fc06c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.473+0000 7fcddad9d640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 0x7fcdc40a51f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40ba570 tx=0x7fcdd408fb60 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.473+0000 7fcdc8ff9640 1 -- 172.21.15.38:0/2187575892 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc013790 con 0x55b51d124940 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.473+0000 7fcdc8ff9640 1 -- 172.21.15.38:0/2187575892 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc019070 con 0x55b51d124940 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.473+0000 7fcdc8ff9640 1 -- 172.21.15.38:0/2187575892 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc014e10 con 0x55b51d124940 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- 172.21.15.38:0/2187575892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 msgr2=0x55b51d12cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00a4b0 tx=0x7fcdcc0096d0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- 172.21.15.38:0/2187575892 shutdown_connections 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d12d520 0x55b51d12f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124940 0x55b51d12cfe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/2187575892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d124400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- 172.21.15.38:0/2187575892 >> 172.21.15.38:0/2187575892 conn(0x55b51d11e750 msgr2=0x55b51d11eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- 172.21.15.38:0/2187575892 shutdown_connections 2024-04-22T00:11:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- 172.21.15.38:0/2187575892 wait complete. 2024-04-22T00:11:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 -- start start 2024-04-22T00:11:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.474+0000 7fcde3bb0740 1 Processor -- start 2024-04-22T00:11:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 0x55b51d12a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b51d130310 con 0x55b51d124020 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55b51d043f10 con 0x55b51d12d520 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b51d130490 con 0x55b51d124940 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcde3bb0740 10 client.0 _ll_register_callbacks cb 0x55b51cd9bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdd9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdd9d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33212/0 (socket says 172.21.15.38:33212) 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdd9d9b640 1 -- 172.21.15.38:0/835561557 learned_addr learned my addr 172.21.15.38:0/835561557 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 0x55b51d12a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 -- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 msgr2=0x55b51d12a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 -- 172.21.15.38:0/835561557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 msgr2=0x55b51d12cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 -- 172.21.15.38:0/835561557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51d015c80 con 0x55b51d124940 2024-04-22T00:11:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdda59c640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 0x55b51d12a230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc004a80 tx=0x7fcdcc014710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a8313041cd1dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.475+0000 7fcdc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc013590 con 0x55b51d124940 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc019070 con 0x55b51d124940 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fcdcc013730 con 0x55b51d124940 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc031050 con 0x55b51d124940 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 10 client.27655 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b51cd9c3f0 con 0x55b51d124940 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc019070 con 0x55b51d124940 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fcdcc01e700 con 0x55b51d124940 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 10 client.27655 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 10 client.27655 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 20 client.27655 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d13d4d0 0x55b51d13f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 4 client.27655 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.476+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b51d095a70 con 0x55b51d13d4d0 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.477+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d13d4d0 0x55b51d13f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.477+0000 7fcdc1ffb640 10 client.27655 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.477+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d13d4d0 0x55b51d13f8c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=501f66a7ba9cbbb3 server_cookie=b0ee6c402652cd2d in_seq=0 out_seq=0 2024-04-22T00:11:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/835561557 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b51d095a70 con 0x55b51d13d4d0 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcdc1ffb640 10 client.27655 handle_command_reply: tid=0 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 client.27655 shutdown 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 msgr2=0x55b51d12a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 0x55b51d12a230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc004a80 tx=0x7fcdcc014710 comp rx=0 tx=0).stop 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 shutdown_connections 2024-04-22T00:11:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124940 0x55b51d12a230 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d13d4d0 0x55b51d13f8c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/835561557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124020 0x55b51d12cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcdc1ffb640 0 client.27655 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 >> 172.21.15.38:0/835561557 conn(0x55b51d11e750 msgr2=0x55b51d1204a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 shutdown_connections 2024-04-22T00:11:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 1 -- 172.21.15.38:0/835561557 wait complete. 2024-04-22T00:11:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.478+0000 7fcde3bb0740 20 client.27655 trim_cache size 0 max 16384 2024-04-22T00:11:25.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.505+0000 7fcde3bb0740 1 Processor -- start 2024-04-22T00:11:25.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.505+0000 7fcde3bb0740 1 -- start start 2024-04-22T00:11:25.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d1391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1b0df0 con 0x55b51d124940 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1b29f0 con 0x55b51d12d520 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1b14f0 con 0x55b51d124020 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33226/0 (socket says 172.21.15.38:33226) 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 -- 172.21.15.38:0/178302124 learned_addr learned my addr 172.21.15.38:0/178302124 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d1391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 -- 172.21.15.38:0/178302124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 msgr2=0x55b51d1391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d1391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 -- 172.21.15.38:0/178302124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 msgr2=0x55b51d12cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 -- 172.21.15.38:0/178302124 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51ce667d0 con 0x55b51d124940 2024-04-22T00:11:25.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.506+0000 7fcdda59c640 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009980 tx=0x7fcdcc004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ce6c958f3d9463 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcdc3fff640 1 -- 172.21.15.38:0/178302124 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc02e040 con 0x55b51d124940 2024-04-22T00:11:25.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcdc3fff640 1 -- 172.21.15.38:0/178302124 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc019040 con 0x55b51d124940 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcdc3fff640 1 -- 172.21.15.38:0/178302124 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc013960 con 0x55b51d124940 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 -- 172.21.15.38:0/178302124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 msgr2=0x55b51d12c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009980 tx=0x7fcdcc004b50 comp rx=0 tx=0).stop 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 -- 172.21.15.38:0/178302124 shutdown_connections 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d12d520 0x55b51d12cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d12c960 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/178302124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d124020 0x55b51d1391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.507+0000 7fcde3bb0740 1 -- 172.21.15.38:0/178302124 >> 172.21.15.38:0/178302124 conn(0x55b51d11e750 msgr2=0x55b51d125810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- 172.21.15.38:0/178302124 shutdown_connections 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- 172.21.15.38:0/178302124 wait complete. 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- start start 2024-04-22T00:11:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 Processor -- start 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 0x55b51d1358d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d12d520 0x55b51d126fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1b2320 con 0x55b51d124940 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1b1bf0 con 0x55b51d124020 2024-04-22T00:11:25.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1aedf0 con 0x55b51d12d520 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcde3bb0740 10 client.0 _ll_register_callbacks cb 0x7fcdd000ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcdda59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33236/0 (socket says 172.21.15.38:33236) 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.508+0000 7fcdda59c640 1 -- 172.21.15.38:0/64874580 learned_addr learned my addr 172.21.15.38:0/64874580 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d12d520 0x55b51d126fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 0x55b51d1358d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 -- 172.21.15.38:0/64874580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d12d520 msgr2=0x55b51d126fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d12d520 0x55b51d126fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 -- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 msgr2=0x55b51d1358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 0x55b51d1358d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 -- 172.21.15.38:0/64874580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51d127520 con 0x55b51d124940 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 0x55b51d1358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdda59c640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc000f80 tx=0x7fcdcc014940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9398e22c65f0894e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc044020 con 0x55b51d124940 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fcdcc014da0 con 0x55b51d124940 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc019040 con 0x55b51d124940 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc01e610 con 0x55b51d124940 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcde3bb0740 10 client.18576 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.509+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55b51d1a34a0 con 0x55b51d124940 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdcc019040 con 0x55b51d124940 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fcdcc0405e0 con 0x55b51d124940 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 10 client.18576 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 10 client.18576 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 20 client.18576 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d1a3970 0x55b51d1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 4 client.18576 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b51d08fdc0 con 0x55b51d1a3970 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.510+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d1a3970 0x55b51d1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcdd9d9b640 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d1a3970 0x55b51d1a5d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=71066356f0aff854 server_cookie=d5242a9c9e45d1db in_seq=0 out_seq=0 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcdc1ffb640 10 client.18576 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcdc1ffb640 1 -- 172.21.15.38:0/64874580 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b51d08fdc0 con 0x55b51d1a3970 2024-04-22T00:11:25.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcdc1ffb640 10 client.18576 handle_command_reply: tid=0 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 client.18576 shutdown 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 msgr2=0x55b51d126aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc000f80 tx=0x7fcdcc014940 comp rx=0 tx=0).stop 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 shutdown_connections 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b51d12d520 0x55b51d126fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55b51d1a3970 0x55b51d1a5d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b51d124940 0x55b51d126aa0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 --2- 172.21.15.38:0/64874580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b51d124020 0x55b51d1358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcdc1ffb640 0 client.18576 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 >> 172.21.15.38:0/64874580 conn(0x55b51d11e750 msgr2=0x55b51d11cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 shutdown_connections 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 1 -- 172.21.15.38:0/64874580 wait complete. 2024-04-22T00:11:25.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.511+0000 7fcde3bb0740 20 client.18576 trim_cache size 0 max 16384 2024-04-22T00:11:25.526 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:25.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 msgr2=0x7fcdc40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 0x7fcdc40a51f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40ba570 tx=0x7fcdd408fb60 comp rx=0 tx=0).stop 2024-04-22T00:11:25.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 msgr2=0x7fcddc101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4004f50 tx=0x7fcdd402c840 comp rx=0 tx=0).stop 2024-04-22T00:11:25.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 shutdown_connections 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcdc40a2d30 0x7fcdc40a51f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc10bd00 0x7fcddc101090 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcddc1032e0 0x7fcddc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 --2- 172.21.15.38:0/19429008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcddc1029c0 0x7fcddc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 >> 172.21.15.38:0/19429008 conn(0x7fcddc077740 msgr2=0x7fcddc0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 shutdown_connections 2024-04-22T00:11:25.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.512+0000 7fcde0fcb640 1 -- 172.21.15.38:0/19429008 wait complete. 2024-04-22T00:11:25.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:11:25.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- 172.21.15.38:0/1545493618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 msgr2=0x7f77c010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- 172.21.15.38:0/1545493618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c010d4f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009cd0 tx=0x7f77b802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- 172.21.15.38:0/1545493618 shutdown_connections 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- 172.21.15.38:0/1545493618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c010d4f0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- 172.21.15.38:0/1545493618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0100620 0x7f77c0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- 172.21.15.38:0/1545493618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0106630 0x7f77c0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- 172.21.15.38:0/1545493618 >> 172.21.15.38:0/1545493618 conn(0x7f77c00776a0 msgr2=0x7f77c0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- 172.21.15.38:0/1545493618 shutdown_connections 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- 172.21.15.38:0/1545493618 wait complete. 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 Processor -- start 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- start start 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0100620 0x7f77c0198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0106630 0x7f77c01a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.636+0000 7f77c84df640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0110a40 con 0x7f77c0100fe0 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c84df640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01108c0 con 0x7f77c0106630 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c84df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0110bc0 con 0x7f77c0100620 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33266/0 (socket says 172.21.15.38:33266) 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 -- 172.21.15.38:0/3954069391 learned_addr learned my addr 172.21.15.38:0/3954069391 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 -- 172.21.15.38:0/3954069391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0100620 msgr2=0x7f77c0198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c627b640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0100620 0x7f77c0198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0106630 0x7f77c01a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0100620 0x7f77c0198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 -- 172.21.15.38:0/3954069391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0106630 msgr2=0x7f77c01a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0106630 0x7f77c01a3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 -- 172.21.15.38:0/3954069391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c01a43e0 con 0x7f77c0100fe0 2024-04-22T00:11:25.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f77b000d950 tx=0x7f77b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77b77fe640 1 -- 172.21.15.38:0/3954069391 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0017070 con 0x7f77c0100fe0 2024-04-22T00:11:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77b77fe640 1 -- 172.21.15.38:0/3954069391 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b0009c30 con 0x7f77c0100fe0 2024-04-22T00:11:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77b77fe640 1 -- 172.21.15.38:0/3954069391 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b000f6b0 con 0x7f77c0100fe0 2024-04-22T00:11:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c01a45b0 con 0x7f77c0100fe0 2024-04-22T00:11:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.637+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c01a4a70 con 0x7f77c0100fe0 2024-04-22T00:11:25.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77b77fe640 1 -- 172.21.15.38:0/3954069391 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f77b00109d0 con 0x7f77c0100fe0 2024-04-22T00:11:25.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 Processor -- start 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 -- start start 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d80fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d83910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561ed9cdc530 con 0x561ed9d78020 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561ed9c84830 con 0x561ed9d78940 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.640+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ed9cd84f0 con 0x561ed9d81520 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77b77fe640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 0x7f77ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33272/0 (socket says 172.21.15.38:33272) 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1096173610 learned_addr learned my addr 172.21.15.38:0/1096173610 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d83910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 0x7f77ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c627b640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d80fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77b77fe640 1 -- 172.21.15.38:0/3954069391 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f77b00cf480 con 0x7f77c0100fe0 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1096173610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 msgr2=0x561ed9d83910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d83910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1096173610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 msgr2=0x561ed9d80fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d80fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1096173610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed9ca82c0 con 0x561ed9d78020 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c627b640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d80fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c630 tx=0x7f77b80327f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b52c53b40dce091 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.641+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 0x7f77ac0a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009ca0 tx=0x7f77b8040000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77b4bfe640 1 -- 172.21.15.38:0/1096173610 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8039040 con 0x561ed9d78020 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77b4bfe640 1 -- 172.21.15.38:0/1096173610 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8050070 con 0x561ed9d78020 2024-04-22T00:11:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77b4bfe640 1 -- 172.21.15.38:0/1096173610 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b80346a0 con 0x561ed9d78020 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1096173610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 msgr2=0x561ed9d78400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c630 tx=0x7f77b80327f0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1096173610 shutdown_connections 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d83910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d80fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1096173610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78020 0x561ed9d78400 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1096173610 >> 172.21.15.38:0/1096173610 conn(0x561ed9d72750 msgr2=0x561ed9d72b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1096173610 shutdown_connections 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.642+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1096173610 wait complete. 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 -- start start 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 Processor -- start 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d80fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d7ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561ed9d842b0 con 0x561ed9d78940 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x561ed9c97f10 con 0x561ed9d78020 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ed9d84430 con 0x561ed9d81520 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77cb1a4740 10 client.0 _ll_register_callbacks cb 0x561ed99efbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33278/0 (socket says 172.21.15.38:33278) 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 -- 172.21.15.38:0/2169253504 learned_addr learned my addr 172.21.15.38:0/2169253504 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d7ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c627b640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d80fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 -- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 msgr2=0x561ed9d7ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d7ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 -- 172.21.15.38:0/2169253504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 msgr2=0x561ed9d80fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d80fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.643+0000 7f77c5a7a640 1 -- 172.21.15.38:0/2169253504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed9c69c80 con 0x561ed9d78940 2024-04-22T00:11:25.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f77b0010fd0 tx=0x7f77b0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941ed636988a32bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0020040 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f77b0097d20 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77b8050070 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f77b009b6b0 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 10 client.18588 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561ed99f03f0 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8050070 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f77b00bb040 con 0x561ed9d78940 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 10 client.18588 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 10 client.18588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:11:25.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 20 client.18588 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9d91730 0x561ed9d93b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 4 client.18588 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.644+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ed9ce9a70 con 0x561ed9d91730 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.645+0000 7f77c627b640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9d91730 0x561ed9d93b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.645+0000 7f77a9ffb640 10 client.18588 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.645+0000 7f77c627b640 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9d91730 0x561ed9d93b20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b5aae17cf4dff3e server_cookie=fa0286dc0fc1622d in_seq=0 out_seq=0 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77a9ffb640 1 -- 172.21.15.38:0/2169253504 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ed9ce9a70 con 0x561ed9d91730 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77a9ffb640 10 client.18588 handle_command_reply: tid=0 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 client.18588 shutdown 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 msgr2=0x561ed9d7e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f77b0010fd0 tx=0x7f77b0010d70 comp rx=0 tx=0).stop 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 shutdown_connections 2024-04-22T00:11:25.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9d91730 0x561ed9d93b20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d81520 0x561ed9d7ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d78940 0x561ed9d7e340 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/2169253504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d80fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77a9ffb640 0 client.18588 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 >> 172.21.15.38:0/2169253504 conn(0x561ed9d72750 msgr2=0x561ed9d744a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 shutdown_connections 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 1 -- 172.21.15.38:0/2169253504 wait complete. 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.646+0000 7f77cb1a4740 20 client.18588 trim_cache size 0 max 16384 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.659+0000 7f77cb1a4740 1 Processor -- start 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.659+0000 7f77cb1a4740 1 -- start start 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d8d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78940 0x561ed9d936f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.691 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01aea50 con 0x561ed9d81520 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01b0650 con 0x561ed9d78020 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01af150 con 0x561ed9d78940 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33286/0 (socket says 172.21.15.38:33286) 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1529670574 learned_addr learned my addr 172.21.15.38:0/1529670574 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78940 0x561ed9d936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1529670574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78940 msgr2=0x561ed9d936f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78940 0x561ed9d936f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1529670574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 msgr2=0x561ed9d8d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d8d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 -- 172.21.15.38:0/1529670574 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed9aba7d0 con 0x561ed9d81520 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009bf0 tx=0x7f77b802c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272fe57f5648ff34 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77abfff640 1 -- 172.21.15.38:0/1529670574 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8039040 con 0x561ed9d81520 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77abfff640 1 -- 172.21.15.38:0/1529670574 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8059070 con 0x561ed9d81520 2024-04-22T00:11:25.692 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.660+0000 7f77abfff640 1 -- 172.21.15.38:0/1529670574 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8054760 con 0x561ed9d81520 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1529670574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 msgr2=0x561ed9d89a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009bf0 tx=0x7f77b802c7f0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1529670574 shutdown_connections 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d89a60 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78940 0x561ed9d936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/1529670574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78020 0x561ed9d8d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1529670574 >> 172.21.15.38:0/1529670574 conn(0x561ed9d72750 msgr2=0x561ed9d71850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1529670574 shutdown_connections 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- 172.21.15.38:0/1529670574 wait complete. 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 -- start start 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.661+0000 7f77cb1a4740 1 Processor -- start 2024-04-22T00:11:25.693 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78020 0x561ed9d7a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d7aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01aff80 con 0x561ed9d81520 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01af850 con 0x561ed9d78940 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01aca50 con 0x561ed9d78020 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 10 client.0 _ll_register_callbacks cb 0x7f77b0097990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33292/0 (socket says 172.21.15.38:33292) 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 -- 172.21.15.38:0/3010612829 learned_addr learned my addr 172.21.15.38:0/3010612829 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c627b640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78020 0x561ed9d7a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d7aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 -- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78020 msgr2=0x561ed9d7a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.694 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78020 0x561ed9d7a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 -- 172.21.15.38:0/3010612829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 msgr2=0x561ed9d7aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d7aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 -- 172.21.15.38:0/3010612829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed9d7b6f0 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c5a7a640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d7aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77c6a7c640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f77b804d040 tx=0x7f77b80327f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906edf68fc61af0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8039040 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f77b8004240 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77b8059070 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f77b8004840 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 10 client.18594 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.662+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561ed9dfbff0 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77b8059070 con 0x561ed9d81520 2024-04-22T00:11:25.695 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f77b8034440 con 0x561ed9d81520 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 10 client.18594 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 10 client.18594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 20 client.18594 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9dfc2e0 0x561ed9dfe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 4 client.18594 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ed9ce3dc0 con 0x561ed9dfc2e0 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.663+0000 7f77c627b640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9dfc2e0 0x561ed9dfe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77a9ffb640 10 client.18594 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77c627b640 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9dfc2e0 0x561ed9dfe6d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=97b6e2c670eb880b server_cookie=d6defd813831f8d4 in_seq=0 out_seq=0 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77a9ffb640 1 -- 172.21.15.38:0/3010612829 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ed9ce3dc0 con 0x561ed9dfc2e0 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77a9ffb640 10 client.18594 handle_command_reply: tid=0 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 client.18594 shutdown 2024-04-22T00:11:25.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 msgr2=0x561ed9d7b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f77b804d040 tx=0x7f77b80327f0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 shutdown_connections 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x561ed9dfc2e0 0x561ed9dfe6d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ed9d81520 0x561ed9d7b030 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561ed9d78940 0x561ed9d7aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77a9ffb640 0 client.18594 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 --2- 172.21.15.38:0/3010612829 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ed9d78020 0x561ed9d7a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 >> 172.21.15.38:0/3010612829 conn(0x561ed9d72750 msgr2=0x561ed9d73a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 shutdown_connections 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 1 -- 172.21.15.38:0/3010612829 wait complete. 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77cb1a4740 20 client.18594 trim_cache size 0 max 16384 2024-04-22T00:11:25.697 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.664+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 msgr2=0x7f77ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 0x7f77ac0a51c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009ca0 tx=0x7f77b8040000 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 msgr2=0x7f77c0199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f77b000d950 tx=0x7f77b000de10 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 shutdown_connections 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f77ac0a2d00 0x7f77ac0a51c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77c0106630 0x7f77c01a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77c0100fe0 0x7f77c0199360 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 --2- 172.21.15.38:0/3954069391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77c0100620 0x7f77c0198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 >> 172.21.15.38:0/3954069391 conn(0x7f77c00776a0 msgr2=0x7f77c01030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 shutdown_connections 2024-04-22T00:11:25.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.665+0000 7f77c84df640 1 -- 172.21.15.38:0/3954069391 wait complete. 2024-04-22T00:11:25.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:11:25.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:25 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3544698532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 -- 172.21.15.38:0/1150524937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8107720 msgr2=0x7fc2e8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 --2- 172.21.15.38:0/1150524937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8107720 0x7fc2e8107b00 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009930 tx=0x7fc2d802bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 -- 172.21.15.38:0/1150524937 shutdown_connections 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 --2- 172.21.15.38:0/1150524937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 --2- 172.21.15.38:0/1150524937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8101710 0x7fc2e8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 --2- 172.21.15.38:0/1150524937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8107720 0x7fc2e8107b00 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 -- 172.21.15.38:0/1150524937 >> 172.21.15.38:0/1150524937 conn(0x7fc2e80fd180 msgr2=0x7fc2e80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 -- 172.21.15.38:0/1150524937 shutdown_connections 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 -- 172.21.15.38:0/1150524937 wait complete. 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.807+0000 7fc2eed02640 1 Processor -- start 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 -- start start 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 0x7fc2e819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e8111af0 con 0x7fc2e8101710 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e8111970 con 0x7fc2e81020d0 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eed02640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e8111c70 con 0x7fc2e8107720 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33314/0 (socket says 172.21.15.38:33314) 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 -- 172.21.15.38:0/438289956 learned_addr learned my addr 172.21.15.38:0/438289956 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 0x7fc2e819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 -- 172.21.15.38:0/438289956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 msgr2=0x7fc2e819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 0x7fc2e819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 -- 172.21.15.38:0/438289956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 msgr2=0x7fc2e8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 -- 172.21.15.38:0/438289956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e819cfe0 con 0x7fc2e8101710 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 0x7fc2e819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009230 tx=0x7fc2d802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.808+0000 7fc2e5ffb640 1 -- 172.21.15.38:0/438289956 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8041070 con 0x7fc2e8101710 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.809+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e819d270 con 0x7fc2e8101710 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.809+0000 7fc2e5ffb640 1 -- 172.21.15.38:0/438289956 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d8004450 con 0x7fc2e8101710 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.809+0000 7fc2e5ffb640 1 -- 172.21.15.38:0/438289956 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8034440 con 0x7fc2e8101710 2024-04-22T00:11:25.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.809+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e81a4b60 con 0x7fc2e8101710 2024-04-22T00:11:25.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.811+0000 7fc2e5ffb640 1 -- 172.21.15.38:0/438289956 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fc2d8044070 con 0x7fc2e8101710 2024-04-22T00:11:25.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.811+0000 7fc2f19ca740 1 Processor -- start 2024-04-22T00:11:25.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.811+0000 7fc2f19ca740 1 -- start start 2024-04-22T00:11:25.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe443400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe44e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555bbe3a7530 con 0x555bbe443940 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555bbe34f830 con 0x555bbe44c520 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555bbe3a34f0 con 0x555bbe443020 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33326/0 (socket says 172.21.15.38:33326) 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2e5ffb640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 0x7fc2d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 -- 172.21.15.38:0/265520775 learned_addr learned my addr 172.21.15.38:0/265520775 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe44e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe443400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 0x7fc2d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2e5ffb640 1 -- 172.21.15.38:0/438289956 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fc2d80f12b0 con 0x7fc2e8101710 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 -- 172.21.15.38:0/265520775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 msgr2=0x555bbe443400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe443400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 -- 172.21.15.38:0/265520775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 msgr2=0x555bbe44e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe44e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 -- 172.21.15.38:0/265520775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bbe3732c0 con 0x555bbe443940 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80b9ec0 tx=0x7fc2d80b98c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce4644b795a6f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 0x7fc2d40a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0005c40 tx=0x7fc2e0009200 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.812+0000 7fc2deffd640 1 -- 172.21.15.38:0/265520775 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d804b070 con 0x555bbe443940 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2deffd640 1 -- 172.21.15.38:0/265520775 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d80bdbf0 con 0x555bbe443940 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2deffd640 1 -- 172.21.15.38:0/265520775 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80c6160 con 0x555bbe443940 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 -- 172.21.15.38:0/265520775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 msgr2=0x555bbe44bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80b9ec0 tx=0x7fc2d80b98c0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 -- 172.21.15.38:0/265520775 shutdown_connections 2024-04-22T00:11:25.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe44e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe44bfe0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/265520775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe443400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 -- 172.21.15.38:0/265520775 >> 172.21.15.38:0/265520775 conn(0x555bbe43d750 msgr2=0x555bbe43db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 -- 172.21.15.38:0/265520775 shutdown_connections 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.813+0000 7fc2f19ca740 1 -- 172.21.15.38:0/265520775 wait complete. 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 -- start start 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 Processor -- start 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe44bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 0x555bbe449280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555bbe3a7530 con 0x555bbe44c520 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x555bbe362f10 con 0x555bbe443020 2024-04-22T00:11:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555bbe44f0d0 con 0x555bbe443940 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2f19ca740 10 client.0 _ll_register_callbacks cb 0x555bbe0babe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33332/0 (socket says 172.21.15.38:33332) 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 -- 172.21.15.38:0/2645998968 learned_addr learned my addr 172.21.15.38:0/2645998968 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe44bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 0x555bbe449280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 -- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 msgr2=0x555bbe449280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 0x555bbe449280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 -- 172.21.15.38:0/2645998968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 msgr2=0x555bbe44bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe44bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2ed29f640 1 -- 172.21.15.38:0/2645998968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bbe334c80 con 0x555bbe44c520 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.814+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 0x555bbe449280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0014370 tx=0x7fc2e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b9bc47310c71f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0024030 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fc2e0012970 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2d80bdbf0 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc2e0011e90 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 10 client.18606 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555bbe0bb3f0 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d80bdbf0 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc2e001d050 con 0x555bbe44c520 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 10 client.18606 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 10 client.18606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 20 client.18606 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe45c3b0 0x555bbe45e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 4 client.18606 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:25.821 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.815+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bbe3b4a70 con 0x555bbe45c3b0 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.816+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe45c3b0 0x555bbe45e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.816+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe45c3b0 0x555bbe45e810 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3fca259ae1eb81 server_cookie=cb55b27e211f737b in_seq=0 out_seq=0 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.816+0000 7fc2c7fff640 10 client.18606 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2c7fff640 1 -- 172.21.15.38:0/2645998968 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555bbe3b4a70 con 0x555bbe45c3b0 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2c7fff640 10 client.18606 handle_command_reply: tid=0 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 client.18606 shutdown 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 msgr2=0x555bbe449890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0014370 tx=0x7fc2e0004a40 comp rx=0 tx=0).stop 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 shutdown_connections 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe45c3b0 0x555bbe45e810 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe44c520 0x555bbe449890 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443940 0x555bbe449280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2c7fff640 0 client.18606 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/2645998968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe44bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 >> 172.21.15.38:0/2645998968 conn(0x555bbe43d750 msgr2=0x555bbe43f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 shutdown_connections 2024-04-22T00:11:25.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 1 -- 172.21.15.38:0/2645998968 wait complete. 2024-04-22T00:11:25.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.817+0000 7fc2f19ca740 20 client.18606 trim_cache size 0 max 16384 2024-04-22T00:11:25.833 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:25 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3544698532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:25.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.830+0000 7fc2f19ca740 1 Processor -- start 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 -- start start 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe457e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 0x555bbe44b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81aec80 con 0x555bbe443940 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81b0880 con 0x555bbe443020 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81af380 con 0x555bbe44c520 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33344/0 (socket says 172.21.15.38:33344) 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3026417743 learned_addr learned my addr 172.21.15.38:0/3026417743 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe457e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 0x555bbe44b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3026417743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 msgr2=0x555bbe44b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 0x555bbe44b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3026417743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 msgr2=0x555bbe457e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe457e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3026417743 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bbe1857d0 con 0x555bbe443940 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 0x555bbe44b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe457e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000cce0 tx=0x7fc2d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1d523e0d92f024 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.831+0000 7fc2de7fc640 1 -- 172.21.15.38:0/3026417743 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d001e070 con 0x555bbe443940 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2de7fc640 1 -- 172.21.15.38:0/3026417743 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d000ce90 con 0x555bbe443940 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2de7fc640 1 -- 172.21.15.38:0/3026417743 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0004b20 con 0x555bbe443940 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3026417743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 msgr2=0x555bbe454700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000cce0 tx=0x7fc2d0011810 comp rx=0 tx=0).stop 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3026417743 shutdown_connections 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe44c520 0x555bbe44b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe454700 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3026417743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe443020 0x555bbe457e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3026417743 >> 172.21.15.38:0/3026417743 conn(0x555bbe43d750 msgr2=0x555bbe43c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3026417743 shutdown_connections 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3026417743 wait complete. 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.832+0000 7fc2f19ca740 1 -- start start 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 Processor -- start 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe4ba910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe4c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81b01b0 con 0x555bbe443940 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81afa80 con 0x555bbe44c520 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e81acc80 con 0x555bbe443020 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2f19ca740 10 client.0 _ll_register_callbacks cb 0x7fc2e0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33358/0 (socket says 172.21.15.38:33358) 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3809333155 learned_addr learned my addr 172.21.15.38:0/3809333155 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2ed29f640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe4c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe4ba910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 msgr2=0x555bbe4ba910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe4ba910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3809333155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 msgr2=0x555bbe4c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe4c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 -- 172.21.15.38:0/3809333155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bbe4bb740 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2e7fff640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0005cd0 tx=0x7fc2d00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfcf63b531fff16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.833+0000 7fc2c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d001e040 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fc2d0003c90 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2d000ce90 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc2d000f5d0 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 10 client.18612 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x555bbe4c6f70 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d000ce90 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fc2d0021030 con 0x555bbe443940 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 10 client.18612 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 10 client.18612 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:11:25.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 20 client.18612 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe4c7710 0x555bbe4c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 4 client.18612 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.834+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555bbe3aedc0 con 0x555bbe4c7710 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe4c7710 0x555bbe4c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2eca9e640 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe4c7710 0x555bbe4c9b30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11d736a5492cd04b server_cookie=84dd6203312831b2 in_seq=0 out_seq=0 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2c7fff640 10 client.18612 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2c7fff640 1 -- 172.21.15.38:0/3809333155 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555bbe3aedc0 con 0x555bbe4c7710 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2c7fff640 10 client.18612 handle_command_reply: tid=0 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 client.18612 shutdown 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 msgr2=0x555bbe4bae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0005cd0 tx=0x7fc2d00038a0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 shutdown_connections 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x555bbe4c7710 0x555bbe4c9b30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555bbe44c520 0x555bbe4c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2c7fff640 0 client.18612 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:25.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555bbe443940 0x555bbe4bae50 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 --2- 172.21.15.38:0/3809333155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555bbe443020 0x555bbe4ba910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 >> 172.21.15.38:0/3809333155 conn(0x555bbe43d750 msgr2=0x555bbe443d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 shutdown_connections 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.835+0000 7fc2f19ca740 1 -- 172.21.15.38:0/3809333155 wait complete. 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2f19ca740 20 client.18612 trim_cache size 0 max 16384 2024-04-22T00:11:25.840 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 msgr2=0x7fc2d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 0x7fc2d40a51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0005c40 tx=0x7fc2e0009200 comp rx=0 tx=0).stop 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 msgr2=0x7fc2e8197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009230 tx=0x7fc2d802cc70 comp rx=0 tx=0).stop 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 shutdown_connections 2024-04-22T00:11:25.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fc2d40a2d00 0x7fc2d40a51c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2e8107720 0x7fc2e819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2e81020d0 0x7fc2e8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 --2- 172.21.15.38:0/438289956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e8101710 0x7fc2e8197ff0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 >> 172.21.15.38:0/438289956 conn(0x7fc2e80fd180 msgr2=0x7fc2e80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 shutdown_connections 2024-04-22T00:11:25.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.836+0000 7fc2eed02640 1 -- 172.21.15.38:0/438289956 wait complete. 2024-04-22T00:11:25.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 -- 172.21.15.38:0/2411141824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc106690 msgr2=0x7feadc106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2411141824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc106690 0x7feadc106a70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7feacc009900 tx=0x7feacc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 -- 172.21.15.38:0/2411141824 shutdown_connections 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2411141824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 0x7feadc10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2411141824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc100990 0x7feadc100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2411141824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc106690 0x7feadc106a70 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 -- 172.21.15.38:0/2411141824 >> 172.21.15.38:0/2411141824 conn(0x7feadc076440 msgr2=0x7feadc076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 -- 172.21.15.38:0/2411141824 shutdown_connections 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.951+0000 7feae1e9d640 1 -- 172.21.15.38:0/2411141824 wait complete. 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 Processor -- start 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 -- start start 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 0x7feadc1992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc106690 0x7feadc1a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1109e0 con 0x7feadc100990 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feadc110860 con 0x7feadc106690 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feae1e9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feadc110b60 con 0x7feadc101330 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.952+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33376/0 (socket says 172.21.15.38:33376) 2024-04-22T00:11:25.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 -- 172.21.15.38:0/2727796990 learned_addr learned my addr 172.21.15.38:0/2727796990 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadbfff640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc106690 0x7feadc1a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadaffd640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 0x7feadc1992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 -- 172.21.15.38:0/2727796990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 msgr2=0x7feadc1992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 0x7feadc1992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 -- 172.21.15.38:0/2727796990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc106690 msgr2=0x7feadc1a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc106690 0x7feadc1a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 -- 172.21.15.38:0/2727796990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc1a4360 con 0x7feadc100990 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feadb7fe640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7feacc006010 tx=0x7feacc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7fead8ff9640 1 -- 172.21.15.38:0/2727796990 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc041070 con 0x7feadc100990 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7fead8ff9640 1 -- 172.21.15.38:0/2727796990 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feacc004450 con 0x7feadc100990 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7fead8ff9640 1 -- 172.21.15.38:0/2727796990 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc003e00 con 0x7feadc100990 2024-04-22T00:11:25.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feadc1a4530 con 0x7feadc100990 2024-04-22T00:11:25.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.953+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feadc1a49a0 con 0x7feadc100990 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7fead8ff9640 1 -- 172.21.15.38:0/2727796990 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7feacc044070 con 0x7feadc100990 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 Processor -- start 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 -- start start 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad7321c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7321ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ad73173530 con 0x55ad7320ed50 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ad7311b830 con 0x55ad7321c930 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feae4b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ad7316f4f0 con 0x55ad7320f670 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7fead8ff9640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 0x7feac40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feadbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feadbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33388/0 (socket says 172.21.15.38:33388) 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feadbfff640 1 -- 172.21.15.38:0/375697579 learned_addr learned my addr 172.21.15.38:0/375697579 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.956+0000 7feadbfff640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 0x7feac40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7fead8ff9640 1 -- 172.21.15.38:0/2727796990 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7feacc0f12e0 con 0x7feadc100990 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadb7fe640 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad7321c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadaffd640 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7321ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 -- 172.21.15.38:0/375697579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 msgr2=0x55ad7321c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad7321c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 -- 172.21.15.38:0/375697579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 msgr2=0x55ad7321ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7321ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 -- 172.21.15.38:0/375697579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad7313f2c0 con 0x55ad7320ed50 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fead4012d00 tx=0x7fead4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15288265586ff932 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feadbfff640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 0x7feac40a51c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fead4010c40 tx=0x7fead400c000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feac1ffb640 1 -- 172.21.15.38:0/375697579 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4024070 con 0x55ad7320ed50 2024-04-22T00:11:25.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feac1ffb640 1 -- 172.21.15.38:0/375697579 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead4009d20 con 0x55ad7320ed50 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.957+0000 7feac1ffb640 1 -- 172.21.15.38:0/375697579 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40053e0 con 0x55ad7320ed50 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- 172.21.15.38:0/375697579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 msgr2=0x55ad7320f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fead4012d00 tx=0x7fead4010740 comp rx=0 tx=0).stop 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- 172.21.15.38:0/375697579 shutdown_connections 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7321ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad7321c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 --2- 172.21.15.38:0/375697579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad7320f130 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- 172.21.15.38:0/375697579 >> 172.21.15.38:0/375697579 conn(0x55ad73209750 msgr2=0x55ad73209b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- 172.21.15.38:0/375697579 shutdown_connections 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- 172.21.15.38:0/375697579 wait complete. 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 -- start start 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.958+0000 7feae4b67740 1 Processor -- start 2024-04-22T00:11:25.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7320ed50 0x55ad7321c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 0x55ad73219d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ad7321f690 con 0x55ad7320f670 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55ad7312ef10 con 0x55ad7320ed50 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ad7321f810 con 0x55ad7321c930 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feae4b67740 10 client.0 _ll_register_callbacks cb 0x55ad72e86be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33392/0 (socket says 172.21.15.38:33392) 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 -- 172.21.15.38:0/3639033488 learned_addr learned my addr 172.21.15.38:0/3639033488 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadb7fe640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7320ed50 0x55ad7321c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadbfff640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 0x55ad73219d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 -- 172.21.15.38:0/3639033488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 msgr2=0x55ad73219d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 0x55ad73219d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 -- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7320ed50 msgr2=0x55ad7321c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7320ed50 0x55ad7321c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 -- 172.21.15.38:0/3639033488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad73100c80 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadbfff640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 0x55ad73219d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feadaffd640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fead0013710 tx=0x7fead0013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bee31aff649d85d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feabaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0011dd0 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.959+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fead0004030 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ad73180a70 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fead0004270 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 10 client.18624 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ad72e873f0 con 0x55ad7320f670 2024-04-22T00:11:25.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ad73180a70 con 0x55ad7320f670 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fead001a070 con 0x55ad7320f670 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 10 client.18624 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 10 client.18624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 20 client.18624 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad7322cad0 0x55ad7322ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 4 client.18624 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad7317adc0 con 0x55ad7322cad0 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.960+0000 7feadb7fe640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad7322cad0 0x55ad7322ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.961+0000 7feadb7fe640 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad7322cad0 0x55ad7322ef10 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63555aa6e41115c3 server_cookie=8afe232739757863 in_seq=0 out_seq=0 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.961+0000 7feabaffd640 10 client.18624 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feabaffd640 1 -- 172.21.15.38:0/3639033488 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad7317adc0 con 0x55ad7322cad0 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feabaffd640 10 client.18624 handle_command_reply: tid=0 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 client.18624 shutdown 2024-04-22T00:11:25.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 msgr2=0x55ad732196e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fead0013710 tx=0x7fead0013b10 comp rx=0 tx=0).stop 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 shutdown_connections 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7321c930 0x55ad73219d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad7322cad0 0x55ad7322ef10 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad732196e0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 --2- 172.21.15.38:0/3639033488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7320ed50 0x55ad7321c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feabaffd640 0 client.18624 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 >> 172.21.15.38:0/3639033488 conn(0x55ad73209750 msgr2=0x55ad7320b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 shutdown_connections 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 1 -- 172.21.15.38:0/3639033488 wait complete. 2024-04-22T00:11:25.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.962+0000 7feae4b67740 20 client.18624 trim_cache size 0 max 16384 2024-04-22T00:11:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.975+0000 7feae4b67740 1 Processor -- start 2024-04-22T00:11:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.975+0000 7feae4b67740 1 -- start start 2024-04-22T00:11:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 0x55ad73228590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7320b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1ac960 con 0x55ad7320f670 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1ae560 con 0x55ad7321c930 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feae4b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1ad060 con 0x55ad7320ed50 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33396/0 (socket says 172.21.15.38:33396) 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 -- 172.21.15.38:0/1910226060 learned_addr learned my addr 172.21.15.38:0/1910226060 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadb7fe640 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 0x55ad73228590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 -- 172.21.15.38:0/1910226060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 msgr2=0x55ad73228590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 0x55ad73228590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 -- 172.21.15.38:0/1910226060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 msgr2=0x55ad7320b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7320b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.979 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 -- 172.21.15.38:0/1910226060 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad7321a080 con 0x55ad7320f670 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadb7fe640 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 0x55ad73228590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feadaffd640 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fead000f660 tx=0x7fead000f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82812d4b5386ca16 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feac17fa640 1 -- 172.21.15.38:0/1910226060 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead000f8d0 con 0x55ad7320f670 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feac17fa640 1 -- 172.21.15.38:0/1910226060 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead000fa70 con 0x55ad7320f670 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.976+0000 7feac17fa640 1 -- 172.21.15.38:0/1910226060 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead002a860 con 0x55ad7320f670 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- 172.21.15.38:0/1910226060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 msgr2=0x55ad73218fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fead000f660 tx=0x7fead000f450 comp rx=0 tx=0).stop 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- 172.21.15.38:0/1910226060 shutdown_connections 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad7320b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320f670 0x55ad73218fc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 --2- 172.21.15.38:0/1910226060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320ed50 0x55ad73228590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- 172.21.15.38:0/1910226060 >> 172.21.15.38:0/1910226060 conn(0x55ad73209750 msgr2=0x55ad73214390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- 172.21.15.38:0/1910226060 shutdown_connections 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- 172.21.15.38:0/1910226060 wait complete. 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 -- start start 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.977+0000 7feae4b67740 1 Processor -- start 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad73220060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad732205a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1ade90 con 0x55ad7320ed50 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1ad760 con 0x55ad7321c930 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feadc1aa960 con 0x55ad7320f670 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 10 client.0 _ll_register_callbacks cb 0x7fead0009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33400/0 (socket says 172.21.15.38:33400) 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 -- 172.21.15.38:0/43890771 learned_addr learned my addr 172.21.15.38:0/43890771 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:25.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadaffd640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad73220060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadbfff640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad732205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 -- 172.21.15.38:0/43890771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 msgr2=0x55ad73220060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad73220060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 -- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 msgr2=0x55ad732205a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad732205a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 -- 172.21.15.38:0/43890771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad72f517d0 con 0x55ad7320ed50 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadbfff640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad732205a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feadb7fe640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7feacc0b9f50 tx=0x7feacc049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb98bb9e622c2b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feabaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc04b040 con 0x55ad7320ed50 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7feacc0bb880 con 0x55ad7320ed50 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.978+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead000fa70 con 0x55ad7320ed50 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7feacc0bdc80 con 0x55ad7320ed50 2024-04-22T00:11:25.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 10 client.18630 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ad73292cd0 con 0x55ad7320ed50 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead000fa70 con 0x55ad7320ed50 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7feacc0ee920 con 0x55ad7320ed50 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 10 client.18630 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 10 client.18630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 20 client.18630 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad73293350 0x55ad73295770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 4 client.18630 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad730f86e0 con 0x55ad73293350 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.979+0000 7feadaffd640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad73293350 0x55ad73295770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feabaffd640 10 client.18630 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feadaffd640 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad73293350 0x55ad73295770 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=593861f1a46e955 server_cookie=4d73ea6c34785d43 in_seq=0 out_seq=0 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feabaffd640 1 -- 172.21.15.38:0/43890771 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad730f86e0 con 0x55ad73293350 2024-04-22T00:11:25.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feabaffd640 10 client.18630 handle_command_reply: tid=0 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 client.18630 shutdown 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 msgr2=0x55ad73224cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7feacc0b9f50 tx=0x7feacc049000 comp rx=0 tx=0).stop 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 shutdown_connections 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ad7321c930 0x55ad732205a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad7320f670 0x55ad73220060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55ad73293350 0x55ad73295770 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 --2- 172.21.15.38:0/43890771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad7320ed50 0x55ad73224cf0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feabaffd640 0 client.18630 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 >> 172.21.15.38:0/43890771 conn(0x55ad73209750 msgr2=0x55ad73214390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 shutdown_connections 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 1 -- 172.21.15.38:0/43890771 wait complete. 2024-04-22T00:11:25.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.980+0000 7feae4b67740 20 client.18630 trim_cache size 0 max 16384 2024-04-22T00:11:25.985 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:25.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 msgr2=0x7feac40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 0x7feac40a51c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fead4010c40 tx=0x7fead400c000 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 msgr2=0x7feadc198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7feacc006010 tx=0x7feacc02cb30 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 shutdown_connections 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7feac40a2d00 0x7feac40a51c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feadc106690 0x7feadc1a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feadc101330 0x7feadc1992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 --2- 172.21.15.38:0/2727796990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feadc100990 0x7feadc198da0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 >> 172.21.15.38:0/2727796990 conn(0x7feadc076440 msgr2=0x7feadc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 shutdown_connections 2024-04-22T00:11:25.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:25.981+0000 7feae1e9d640 1 -- 172.21.15.38:0/2727796990 wait complete. 2024-04-22T00:11:26.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:26 smithi038 ceph-mon[27972]: pgmap v289: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:26.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:26 smithi203 ceph-mon[31431]: pgmap v289: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:26.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:26 smithi111 ceph-mon[31543]: pgmap v289: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:26.989 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:11:26.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:11:26.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:11:27.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- 172.21.15.38:0/2519356869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934107720 msgr2=0x7fa934107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 --2- 172.21.15.38:0/2519356869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934107720 0x7fa934107b00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fa930009930 tx=0x7fa93002bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- 172.21.15.38:0/2519356869 shutdown_connections 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 --2- 172.21.15.38:0/2519356869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9341020d0 0x7fa93410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 --2- 172.21.15.38:0/2519356869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa934101710 0x7fa934101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 --2- 172.21.15.38:0/2519356869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934107720 0x7fa934107b00 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- 172.21.15.38:0/2519356869 >> 172.21.15.38:0/2519356869 conn(0x7fa9340fd180 msgr2=0x7fa9340ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- 172.21.15.38:0/2519356869 shutdown_connections 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- 172.21.15.38:0/2519356869 wait complete. 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 Processor -- start 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.112+0000 7fa93abf5640 1 -- start start 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 0x7fa934073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa934107720 0x7fa9340744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa934111b80 con 0x7fa934101710 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa934111a00 con 0x7fa934107720 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa934111d00 con 0x7fa9341020d0 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33420/0 (socket says 172.21.15.38:33420) 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 -- 172.21.15.38:0/4200260888 learned_addr learned my addr 172.21.15.38:0/4200260888 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa92bfff640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 0x7fa934073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa939192640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa934107720 0x7fa9340744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 -- 172.21.15.38:0/4200260888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 msgr2=0x7fa934073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 0x7fa934073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 -- 172.21.15.38:0/4200260888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa934107720 msgr2=0x7fa9340744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa934107720 0x7fa9340744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 -- 172.21.15.38:0/4200260888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9341a6520 con 0x7fa934101710 2024-04-22T00:11:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa92bfff640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 0x7fa934073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa938991640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fa930009900 tx=0x7fa930004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa929ffb640 1 -- 172.21.15.38:0/4200260888 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930043020 con 0x7fa934101710 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa929ffb640 1 -- 172.21.15.38:0/4200260888 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9300054e0 con 0x7fa934101710 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9341a67b0 con 0x7fa934101710 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.113+0000 7fa929ffb640 1 -- 172.21.15.38:0/4200260888 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930003d00 con 0x7fa934101710 2024-04-22T00:11:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.114+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9341a6d00 con 0x7fa934101710 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.116+0000 7fa929ffb640 1 -- 172.21.15.38:0/4200260888 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fa930046020 con 0x7fa934101710 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.116+0000 7fa93d7d9740 1 Processor -- start 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.116+0000 7fa93d7d9740 1 -- start start 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713344a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713344aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa929ffb640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 0x7fa91c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5571333ae170 con 0x557133453db0 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x5571333567c0 con 0x55713344aa40 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571333aa130 con 0x55713344b720 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa92bfff640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 0x7fa91c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33422/0 (socket says 172.21.15.38:33422) 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 -- 172.21.15.38:0/2607807273 learned_addr learned my addr 172.21.15.38:0/2607807273 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa929ffb640 1 -- 172.21.15.38:0/4200260888 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fa9300f19f0 con 0x7fa934101710 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa938991640 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713344aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713344a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 -- 172.21.15.38:0/2607807273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 msgr2=0x55713344a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713344a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 -- 172.21.15.38:0/2607807273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 msgr2=0x55713344aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713344aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 -- 172.21.15.38:0/2607807273 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557133367e60 con 0x557133453db0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa92bfff640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 0x7fa91c0a51f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa9240058c0 tx=0x7fa924005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.117+0000 7fa939192640 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa92c014a20 tx=0x7fa92c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c5039151c7da1c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa91affd640 1 -- 172.21.15.38:0/2607807273 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c014270 con 0x557133453db0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa91affd640 1 -- 172.21.15.38:0/2607807273 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c00f700 con 0x557133453db0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa91affd640 1 -- 172.21.15.38:0/2607807273 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c005280 con 0x557133453db0 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2607807273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 msgr2=0x5571334561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fa92c014a20 tx=0x7fa92c009600 comp rx=0 tx=0).stop 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2607807273 shutdown_connections 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x5571334561a0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713344aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2607807273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713344a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2607807273 >> 172.21.15.38:0/2607807273 conn(0x5571334114f0 msgr2=0x557133445490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2607807273 shutdown_connections 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.118+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2607807273 wait complete. 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 -- start start 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 Processor -- start 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x557133450db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 0x557133451830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557133456c20 con 0x55713344b720 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557133369d40 con 0x55713344aa40 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557133456da0 con 0x557133453db0 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa93d7d9740 10 client.0 _ll_register_callbacks cb 0x5571333a5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33436/0 (socket says 172.21.15.38:33436) 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa92bfff640 1 -- 172.21.15.38:0/2537601452 learned_addr learned my addr 172.21.15.38:0/2537601452 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa938991640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x557133450db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.119+0000 7fa939192640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 0x557133451830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 -- 172.21.15.38:0/2537601452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 msgr2=0x557133451830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 0x557133451830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 -- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 msgr2=0x557133450db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x557133450db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 -- 172.21.15.38:0/2537601452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55713318a7e0 con 0x55713344b720 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa938991640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x557133450db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa939192640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 0x557133451830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa92bfff640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa9240116f0 tx=0x7fa9240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cddb0df89a3896 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa913fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924022070 con 0x55713344b720 2024-04-22T00:11:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fa92401e860 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa92c00f700 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa92402e050 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa93d7d9740 10 client.18642 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5571331eaf70 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.120+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c00f700 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa924010c80 con 0x55713344b720 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 10 client.18642 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 10 client.18642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 20 client.18642 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557133463f10 0x557133466350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 4 client.18642 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557133382450 con 0x557133463f10 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa938991640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557133463f10 0x557133466350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa938991640 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557133463f10 0x557133466350 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45d61c2ec08d829 server_cookie=57ebb35e86ec9599 in_seq=0 out_seq=0 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.121+0000 7fa913fff640 10 client.18642 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa913fff640 1 -- 172.21.15.38:0/2537601452 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557133382450 con 0x557133463f10 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa913fff640 10 client.18642 handle_command_reply: tid=0 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 client.18642 shutdown 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 msgr2=0x5571334512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa9240116f0 tx=0x7fa9240046c0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 shutdown_connections 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557133453db0 0x557133451830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557133463f10 0x557133466350 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55713344b720 0x5571334512f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/2537601452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x557133450db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa913fff640 0 client.18642 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 >> 172.21.15.38:0/2537601452 conn(0x5571334114f0 msgr2=0x5571334471e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 shutdown_connections 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.122+0000 7fa93d7d9740 1 -- 172.21.15.38:0/2537601452 wait complete. 2024-04-22T00:11:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.123+0000 7fa93d7d9740 20 client.18642 trim_cache size 0 max 16384 2024-04-22T00:11:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.140+0000 7fa93d7d9740 1 Processor -- start 2024-04-22T00:11:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.140+0000 7fa93d7d9740 1 -- start start 2024-04-22T00:11:27.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x5571334534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x557133445630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341b0df0 con 0x557133453db0 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341b29f0 con 0x55713344aa40 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341b14f0 con 0x55713344b720 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33438/0 (socket says 172.21.15.38:33438) 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 -- 172.21.15.38:0/3021432861 learned_addr learned my addr 172.21.15.38:0/3021432861 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa938991640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x5571334534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa92bfff640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x557133445630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 -- 172.21.15.38:0/3021432861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 msgr2=0x557133445630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x557133445630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 -- 172.21.15.38:0/3021432861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 msgr2=0x5571334534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x5571334534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 -- 172.21.15.38:0/3021432861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557133082170 con 0x557133453db0 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa92bfff640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x557133445630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa938991640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x5571334534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa939192640 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa92c005c40 tx=0x7fa92c0094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fb091e0e3e5b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa91a7fc640 1 -- 172.21.15.38:0/3021432861 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c014270 con 0x557133453db0 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa91a7fc640 1 -- 172.21.15.38:0/3021432861 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c003e50 con 0x557133453db0 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.141+0000 7fa91a7fc640 1 -- 172.21.15.38:0/3021432861 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c020680 con 0x557133453db0 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- 172.21.15.38:0/3021432861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 msgr2=0x55713345fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa92c005c40 tx=0x7fa92c0094c0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- 172.21.15.38:0/3021432861 shutdown_connections 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x55713345fb20 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x557133445630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/3021432861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x5571334534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- 172.21.15.38:0/3021432861 >> 172.21.15.38:0/3021432861 conn(0x5571334114f0 msgr2=0x5571334444f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- 172.21.15.38:0/3021432861 shutdown_connections 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- 172.21.15.38:0/3021432861 wait complete. 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 -- start start 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.142+0000 7fa93d7d9740 1 Processor -- start 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713345c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713345c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341b2320 con 0x557133453db0 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341b1bf0 con 0x55713344aa40 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341aedf0 con 0x55713344b720 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa93d7d9740 10 client.0 _ll_register_callbacks cb 0x7fa924015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:27.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33452/0 (socket says 172.21.15.38:33452) 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 -- 172.21.15.38:0/1869412059 learned_addr learned my addr 172.21.15.38:0/1869412059 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa92bfff640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713345c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa938991640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713345c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 -- 172.21.15.38:0/1869412059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 msgr2=0x55713345c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713345c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 -- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 msgr2=0x55713345c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713345c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 -- 172.21.15.38:0/1869412059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557133457580 con 0x557133453db0 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa92bfff640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713345c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa938991640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713345c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:27.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa939192640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00fd90 tx=0x7fa92c014280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32496a801ae4982 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.143+0000 7fa913fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c00f6d0 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fa92c00f870 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa92c003e50 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa92c03c050 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 10 client.18648 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5571334cad00 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c003e50 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fa92c00c070 con 0x557133453db0 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 10 client.18648 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 10 client.18648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 20 client.18648 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5571334cb170 0x5571334cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 4 client.18648 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:11:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5571333bb6b0 con 0x5571334cb170 2024-04-22T00:11:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.144+0000 7fa938991640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5571334cb170 0x5571334cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.145+0000 7fa913fff640 10 client.18648 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.145+0000 7fa938991640 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5571334cb170 0x5571334cd560 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd2c9c4b030a1d7 server_cookie=cfba5302e459ed6f in_seq=0 out_seq=0 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa913fff640 1 -- 172.21.15.38:0/1869412059 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5571333bb6b0 con 0x5571334cb170 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa913fff640 10 client.18648 handle_command_reply: tid=0 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 client.18648 shutdown 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 msgr2=0x557133456fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00fd90 tx=0x7fa92c014280 comp rx=0 tx=0).stop 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 shutdown_connections 2024-04-22T00:11:27.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5571334cb170 0x5571334cd560 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557133453db0 0x557133456fc0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55713344b720 0x55713345c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa913fff640 0 client.18648 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 --2- 172.21.15.38:0/1869412059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55713344aa40 0x55713345c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 >> 172.21.15.38:0/1869412059 conn(0x5571334114f0 msgr2=0x557133446780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 shutdown_connections 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.146+0000 7fa93d7d9740 1 -- 172.21.15.38:0/1869412059 wait complete. 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93d7d9740 20 client.18648 trim_cache size 0 max 16384 2024-04-22T00:11:27.150 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:27.151 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:11:27.151 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "df2473c2-5df1-4df0-93a1-9597a7bda691", 2024-04-22T00:11:27.151 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:11:27.151 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 msgr2=0x7fa91c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 0x7fa91c0a51f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa9240058c0 tx=0x7fa924005850 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 msgr2=0x7fa934073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fa930009900 tx=0x7fa930004430 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 shutdown_connections 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fa91c0a2d30 0x7fa91c0a51f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa934107720 0x7fa9340744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9341020d0 0x7fa934073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 --2- 172.21.15.38:0/4200260888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa934101710 0x7fa934073a40 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 >> 172.21.15.38:0/4200260888 conn(0x7fa9340fd180 msgr2=0x7fa9340fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 shutdown_connections 2024-04-22T00:11:27.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.147+0000 7fa93abf5640 1 -- 172.21.15.38:0/4200260888 wait complete. 2024-04-22T00:11:27.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:11:27.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:11:27.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- 172.21.15.38:0/2652290431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1032e0 msgr2=0x7fd96c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2652290431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1032e0 0x7fd96c10b7c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009900 tx=0x7fd95c02bd60 comp rx=0 tx=0).stop 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- 172.21.15.38:0/2652290431 shutdown_connections 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2652290431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c10bd00 0x7fd96c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2652290431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1032e0 0x7fd96c10b7c0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2652290431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c1029c0 0x7fd96c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- 172.21.15.38:0/2652290431 >> 172.21.15.38:0/2652290431 conn(0x7fd96c077740 msgr2=0x7fd96c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- 172.21.15.38:0/2652290431 shutdown_connections 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- 172.21.15.38:0/2652290431 wait complete. 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 Processor -- start 2024-04-22T00:11:27.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.263+0000 7fd9741e5640 1 -- start start 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c1032e0 0x7fd96c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c10bd00 0x7fd96c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c111b50 con 0x7fd96c1029c0 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1119d0 con 0x7fd96c1032e0 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c111cd0 con 0x7fd96c10bd00 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33472/0 (socket says 172.21.15.38:33472) 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 -- 172.21.15.38:0/2870066184 learned_addr learned my addr 172.21.15.38:0/2870066184 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971780640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c1032e0 0x7fd96c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 -- 172.21.15.38:0/2870066184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c10bd00 msgr2=0x7fd96c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd972782640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c10bd00 0x7fd96c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c10bd00 0x7fd96c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 -- 172.21.15.38:0/2870066184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c1032e0 msgr2=0x7fd96c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c1032e0 0x7fd96c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 -- 172.21.15.38:0/2870066184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c19cf30 con 0x7fd96c1029c0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd971f81640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd96800c920 tx=0x7fd96800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd962ffd640 1 -- 172.21.15.38:0/2870066184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968007c70 con 0x7fd96c1029c0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd962ffd640 1 -- 172.21.15.38:0/2870066184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd968007e10 con 0x7fd96c1029c0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96c19d220 con 0x7fd96c1029c0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.264+0000 7fd962ffd640 1 -- 172.21.15.38:0/2870066184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968018890 con 0x7fd96c1029c0 2024-04-22T00:11:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.265+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96c1a4ab0 con 0x7fd96c1029c0 2024-04-22T00:11:27.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd962ffd640 1 -- 172.21.15.38:0/2870066184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd968018a70 con 0x7fd96c1029c0 2024-04-22T00:11:27.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 Processor -- start 2024-04-22T00:11:27.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 -- start start 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801022f30 0x556801023310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801023850 0x55680102bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556800f87560 con 0x55680102c430 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556800f2f730 con 0x556801022f30 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.267+0000 7fd976dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556800f83520 con 0x556801023850 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd962ffd640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 0x7fd9580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33476/0 (socket says 172.21.15.38:33476) 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 -- 172.21.15.38:0/4027226536 learned_addr learned my addr 172.21.15.38:0/4027226536 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971f81640 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801023850 0x55680102bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd972782640 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801022f30 0x556801023310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd972782640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 0x7fd9580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd962ffd640 1 -- 172.21.15.38:0/2870066184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd9680cf930 con 0x7fd96c1029c0 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 -- 172.21.15.38:0/4027226536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801023850 msgr2=0x55680102bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801023850 0x55680102bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 -- 172.21.15.38:0/4027226536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801022f30 msgr2=0x556801023310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801022f30 0x556801023310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 -- 172.21.15.38:0/4027226536 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556800d657d0 con 0x55680102c430 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd971780640 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02c370 tx=0x7fd95c02c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f2314dcf404889 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd972782640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 0x7fd9580a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd964012750 tx=0x7fd964005c60 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd957fff640 1 -- 172.21.15.38:0/4027226536 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c043070 con 0x55680102c430 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd957fff640 1 -- 172.21.15.38:0/4027226536 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c004200 con 0x55680102c430 2024-04-22T00:11:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.268+0000 7fd957fff640 1 -- 172.21.15.38:0/4027226536 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c004f80 con 0x55680102c430 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 -- 172.21.15.38:0/4027226536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 msgr2=0x55680102e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02c370 tx=0x7fd95c02c830 comp rx=0 tx=0).stop 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 -- 172.21.15.38:0/4027226536 shutdown_connections 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55680102c430 0x55680102e820 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801023850 0x55680102bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 --2- 172.21.15.38:0/4027226536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801022f30 0x556801023310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 -- 172.21.15.38:0/4027226536 >> 172.21.15.38:0/4027226536 conn(0x55680101d660 msgr2=0x55680101da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 -- 172.21.15.38:0/4027226536 shutdown_connections 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.269+0000 7fd976dce740 1 -- 172.21.15.38:0/4027226536 wait complete. 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 -- start start 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 Processor -- start 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x556801029430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.275 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801029eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55680102f1c0 con 0x556801023850 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x556800f42e00 con 0x55680102c430 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55680102f340 con 0x556801022f30 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd976dce740 10 client.0 _ll_register_callbacks cb 0x556800f7deb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33486/0 (socket says 172.21.15.38:33486) 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd971780640 1 -- 172.21.15.38:0/2570500194 learned_addr learned my addr 172.21.15.38:0/2570500194 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.270+0000 7fd971f81640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x556801029430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 -- 172.21.15.38:0/2570500194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 msgr2=0x556801029430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd972782640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801029eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x556801029430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 -- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 msgr2=0x556801029eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801029eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 -- 172.21.15.38:0/2570500194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556800c9b3f0 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd971780640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd95c005ab0 tx=0x7fd95c02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ed6a8942283cfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c043050 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd95c02cdc0 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c004200 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd95c004f80 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 10 client.18660 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x556800c5b0f0 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c004200 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fd95c048450 con 0x556801023850 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 10 client.18660 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 10 client.18660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 20 client.18660 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55680103c5d0 0x55680103e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 4 client.18660 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.271+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556800f94aa0 con 0x55680103c5d0 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.272+0000 7fd971f81640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55680103c5d0 0x55680103e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.272+0000 7fd971f81640 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55680103c5d0 0x55680103e9c0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f2d50093348d8c server_cookie=3c6794f2f72a2880 in_seq=0 out_seq=0 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.272+0000 7fd9557fa640 10 client.18660 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd9557fa640 1 -- 172.21.15.38:0/2570500194 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556800f94aa0 con 0x55680103c5d0 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd9557fa640 10 client.18660 handle_command_reply: tid=0 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 client.18660 shutdown 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 msgr2=0x556801029970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd95c005ab0 tx=0x7fd95c02bd60 comp rx=0 tx=0).stop 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 shutdown_connections 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801029eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55680103c5d0 0x55680103e9c0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x556801029970 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 --2- 172.21.15.38:0/2570500194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x556801029430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 >> 172.21.15.38:0/2570500194 conn(0x55680101d660 msgr2=0x55680101f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd9557fa640 0 client.18660 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 shutdown_connections 2024-04-22T00:11:27.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 1 -- 172.21.15.38:0/2570500194 wait complete. 2024-04-22T00:11:27.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.273+0000 7fd976dce740 20 client.18660 trim_cache size 0 max 16384 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.291+0000 7fd976dce740 1 Processor -- start 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 -- start start 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x55680102bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801043f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1aea50 con 0x556801023850 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1b0650 con 0x55680102c430 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd976dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1af150 con 0x556801022f30 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33492/0 (socket says 172.21.15.38:33492) 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 -- 172.21.15.38:0/2100468523 learned_addr learned my addr 172.21.15.38:0/2100468523 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971f81640 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x55680102bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 -- 172.21.15.38:0/2100468523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 msgr2=0x55680102bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd972782640 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801043f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x55680102bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 -- 172.21.15.38:0/2100468523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 msgr2=0x556801043f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801043f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 -- 172.21.15.38:0/2100468523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556801023680 con 0x556801023850 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd971780640 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02c2a0 tx=0x7fd95c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4d72462fe03c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd9577fe640 1 -- 172.21.15.38:0/2100468523 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c043070 con 0x556801023850 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd9577fe640 1 -- 172.21.15.38:0/2100468523 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c02cbc0 con 0x556801023850 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.292+0000 7fd9577fe640 1 -- 172.21.15.38:0/2100468523 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0054e0 con 0x556801023850 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- 172.21.15.38:0/2100468523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 msgr2=0x5568010380b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02c2a0 tx=0x7fd95c000bc0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- 172.21.15.38:0/2100468523 shutdown_connections 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55680102c430 0x556801043f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801023850 0x5568010380b0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 --2- 172.21.15.38:0/2100468523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556801022f30 0x55680102bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- 172.21.15.38:0/2100468523 >> 172.21.15.38:0/2100468523 conn(0x55680101d660 msgr2=0x55680101da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- 172.21.15.38:0/2100468523 shutdown_connections 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- 172.21.15.38:0/2100468523 wait complete. 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.293+0000 7fd976dce740 1 -- start start 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 Processor -- start 2024-04-22T00:11:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 0x55680102fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55680102c430 0x556801030310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1aff80 con 0x556801022f30 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1af850 con 0x556801023850 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1aca50 con 0x55680102c430 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd976dce740 10 client.0 _ll_register_callbacks cb 0x7fd96800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33494/0 (socket says 172.21.15.38:33494) 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 -- 172.21.15.38:0/1129334474 learned_addr learned my addr 172.21.15.38:0/1129334474 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd972782640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55680102c430 0x556801030310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971780640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 0x55680102fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 -- 172.21.15.38:0/1129334474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55680102c430 msgr2=0x556801030310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55680102c430 0x556801030310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 -- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 msgr2=0x55680102fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 0x55680102fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971f81640 1 -- 172.21.15.38:0/1129334474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556801030b40 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.294+0000 7fd971780640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 0x55680102fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd971f81640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ccda0 tx=0x7fd96806eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d8481c856dc6ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96800fb30 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd968098130 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c02cbc0 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd9680ccf70 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 10 client.18666 fetch_fsmap learned FSMap version 38 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5568010a28d0 con 0x556801022f30 2024-04-22T00:11:27.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c02cbc0 con 0x556801022f30 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7fd96809af10 con 0x556801022f30 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 10 client.18666 fetch_fsmap finished waiting for FSMap version 38 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 10 client.18666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 20 client.18666 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5568010a2f50 0x5568010a5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 4 client.18666 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.295+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556800f8edf0 con 0x5568010a2f50 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd971780640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5568010a2f50 0x5568010a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd971780640 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5568010a2f50 0x5568010a5360 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=553b2657e7b97b76 server_cookie=b3f1902ea09e3eb5 in_seq=0 out_seq=0 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd9557fa640 10 client.18666 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd9557fa640 1 -- 172.21.15.38:0/1129334474 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556800f8edf0 con 0x5568010a2f50 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd9557fa640 10 client.18666 handle_command_reply: tid=0 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd976dce740 1 client.18666 shutdown 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 msgr2=0x5568010349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.296+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd9680ccda0 tx=0x7fd96806eb30 comp rx=0 tx=0).stop 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 shutdown_connections 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55680102c430 0x556801030310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556801023850 0x55680102fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x5568010a2f50 0x5568010a5360 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 --2- 172.21.15.38:0/1129334474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556801022f30 0x5568010349f0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9557fa640 0 client.18666 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 >> 172.21.15.38:0/1129334474 conn(0x55680101d660 msgr2=0x55680101be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 shutdown_connections 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 1 -- 172.21.15.38:0/1129334474 wait complete. 2024-04-22T00:11:27.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd976dce740 20 client.18666 trim_cache size 0 max 16384 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "df2473c2-5df1-4df0-93a1-9597a7bda691": { 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "df2473c2-5df1-4df0-93a1-9597a7bda691", 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:11:27.301 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:27.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 msgr2=0x7fd9580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 0x7fd9580a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd964012750 tx=0x7fd964005c60 comp rx=0 tx=0).stop 2024-04-22T00:11:27.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 msgr2=0x7fd96c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:27.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd96800c920 tx=0x7fd96800cde0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 shutdown_connections 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd9580a2d00 0x7fd9580a51c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd96c10bd00 0x7fd96c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c1032e0 0x7fd96c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 --2- 172.21.15.38:0/2870066184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd96c1029c0 0x7fd96c197f90 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 >> 172.21.15.38:0/2870066184 conn(0x7fd96c077740 msgr2=0x7fd96c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 shutdown_connections 2024-04-22T00:11:27.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:27.297+0000 7fd9741e5640 1 -- 172.21.15.38:0/2870066184 wait complete. 2024-04-22T00:11:27.305 INFO:tasks.cephfs.filesystem:scrub status for tag:df2473c2-5df1-4df0-93a1-9597a7bda691 - {'path': '/', 'tag': 'df2473c2-5df1-4df0-93a1-9597a7bda691', 'options': 'recursive,force'} 2024-04-22T00:11:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:28 smithi038 ceph-mon[27972]: pgmap v290: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:28 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:11:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:28 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:28 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:11:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:28 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:28 smithi203 ceph-mon[31431]: pgmap v290: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:28 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:11:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:28 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:28 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:11:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:28 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:28 smithi111 ceph-mon[31543]: pgmap v290: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:28 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:11:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:28 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:28 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:11:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:28 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:11:30.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:30 smithi038 ceph-mon[27972]: pgmap v291: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:30.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:30 smithi203 ceph-mon[31431]: pgmap v291: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:30.830 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:30 smithi111 ceph-mon[31543]: pgmap v291: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:11:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:31 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:11:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:11:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:11:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:11:31.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:11:31.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:31 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:11:31.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:11:31.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:11:31.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:11:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:11:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:31 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:11:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:11:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:11:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:11:32.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:32 smithi038 ceph-mon[27972]: pgmap v292: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:11:32.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:32 smithi203 ceph-mon[31431]: pgmap v292: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:11:32.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:32 smithi111 ceph-mon[31543]: pgmap v292: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T00:11:34.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:34 smithi038 ceph-mon[27972]: pgmap v293: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-22T00:11:34.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:34 smithi203 ceph-mon[31431]: pgmap v293: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-22T00:11:34.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:34 smithi111 ceph-mon[31543]: pgmap v293: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-22T00:11:36.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:36 smithi038 ceph-mon[27972]: pgmap v294: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:36.783 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:36 smithi203 ceph-mon[31431]: pgmap v294: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:36.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:36 smithi111 ceph-mon[31543]: pgmap v294: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:38.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:38 smithi038 ceph-mon[27972]: pgmap v295: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-22T00:11:38.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:38 smithi203 ceph-mon[31431]: pgmap v295: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-22T00:11:38.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:38 smithi111 ceph-mon[31543]: pgmap v295: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-22T00:11:40.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:40 smithi203 ceph-mon[31431]: pgmap v296: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:40.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:40 smithi111 ceph-mon[31543]: pgmap v296: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:40.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:40 smithi038 ceph-mon[27972]: pgmap v296: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-22T00:11:42.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:42 smithi203 ceph-mon[31431]: pgmap v297: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-22T00:11:42.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:42 smithi111 ceph-mon[31543]: pgmap v297: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-22T00:11:42.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:42 smithi038 ceph-mon[27972]: pgmap v297: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-22T00:11:44.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:44 smithi203 ceph-mon[31431]: pgmap v298: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:11:44.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:44 smithi111 ceph-mon[31543]: pgmap v298: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:11:44.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:44 smithi038 ceph-mon[27972]: pgmap v298: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-22T00:11:46.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:46 smithi203 ceph-mon[31431]: pgmap v299: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:46.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:46 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:46.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:46 smithi111 ceph-mon[31543]: pgmap v299: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:46.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:46 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:46.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:46 smithi038 ceph-mon[27972]: pgmap v299: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:46.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:46 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:48.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:48 smithi203 ceph-mon[31431]: pgmap v300: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:48.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:48 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:48.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:48 smithi111 ceph-mon[31543]: pgmap v300: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:48.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:48 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:48.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:48 smithi038 ceph-mon[27972]: pgmap v300: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:48.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:48 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:50.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:50 smithi203 ceph-mon[31431]: pgmap v301: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:50.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:50 smithi111 ceph-mon[31543]: pgmap v301: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:50.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:50 smithi038 ceph-mon[27972]: pgmap v301: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:52.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:52 smithi203 ceph-mon[31431]: pgmap v302: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:11:52.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:52 smithi111 ceph-mon[31543]: pgmap v302: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:11:52.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:52 smithi038 ceph-mon[27972]: pgmap v302: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:11:54.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:54 smithi203 ceph-mon[31431]: pgmap v303: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:11:54.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:54 smithi111 ceph-mon[31543]: pgmap v303: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:11:54.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:54 smithi038 ceph-mon[27972]: pgmap v303: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:11:56.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:56 smithi203 ceph-mon[31431]: pgmap v304: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:56.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:56 smithi111 ceph-mon[31543]: pgmap v304: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:56.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:56 smithi038 ceph-mon[27972]: pgmap v304: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:11:57.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:11:57.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:11:57.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 -- 172.21.15.38:0/4050607045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810100620 msgr2=0x7fe810100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 --2- 172.21.15.38:0/4050607045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810100620 0x7fe810100aa0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fe804009930 tx=0x7fe80402bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:57.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 -- 172.21.15.38:0/4050607045 shutdown_connections 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 --2- 172.21.15.38:0/4050607045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100fe0 0x7fe81010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 --2- 172.21.15.38:0/4050607045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810100620 0x7fe810100aa0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 --2- 172.21.15.38:0/4050607045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810106630 0x7fe810106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.427+0000 7fe816a27640 1 -- 172.21.15.38:0/4050607045 >> 172.21.15.38:0/4050607045 conn(0x7fe8100776a0 msgr2=0x7fe810077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.428+0000 7fe816a27640 1 -- 172.21.15.38:0/4050607045 shutdown_connections 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.428+0000 7fe816a27640 1 -- 172.21.15.38:0/4050607045 wait complete. 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.428+0000 7fe816a27640 1 Processor -- start 2024-04-22T00:11:57.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.428+0000 7fe816a27640 1 -- start start 2024-04-22T00:11:57.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 0x7fe8101980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 0x7fe810198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe810110a40 con 0x7fe810106630 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101108c0 con 0x7fe810100fe0 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe810110bc0 con 0x7fe810100620 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59102/0 (socket says 172.21.15.38:59102) 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 -- 172.21.15.38:0/2696020966 learned_addr learned my addr 172.21.15.38:0/2696020966 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80ffff640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 0x7fe8101980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 0x7fe810198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 -- 172.21.15.38:0/2696020966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 msgr2=0x7fe8101980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 0x7fe8101980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 -- 172.21.15.38:0/2696020966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 msgr2=0x7fe810198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 0x7fe810198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 -- 172.21.15.38:0/2696020966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81019d0c0 con 0x7fe810106630 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 0x7fe810198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80ffff640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 0x7fe8101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe814fc4640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe80800eb10 tx=0x7fe80800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80d7fa640 1 -- 172.21.15.38:0/2696020966 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808004b20 con 0x7fe810106630 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80d7fa640 1 -- 172.21.15.38:0/2696020966 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe808004cc0 con 0x7fe810106630 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81019d3b0 con 0x7fe810106630 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.429+0000 7fe80d7fa640 1 -- 172.21.15.38:0/2696020966 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808010760 con 0x7fe810106630 2024-04-22T00:11:57.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.430+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8101a4c40 con 0x7fe810106630 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.432+0000 7fe80d7fa640 1 -- 172.21.15.38:0/2696020966 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fe808005020 con 0x7fe810106630 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 Processor -- start 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 -- start start 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44e7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44f2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cbf444b560 con 0x55cbf44e7850 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cbf43f3730 con 0x55cbf44e6f30 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe819610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4447520 con 0x55cbf44f0430 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59114/0 (socket says 172.21.15.38:59114) 2024-04-22T00:11:57.437 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 -- 172.21.15.38:0/455370627 learned_addr learned my addr 172.21.15.38:0/455370627 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80d7fa640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 0x7fe7f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe814fc4640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 0x7fe7f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 -- 172.21.15.38:0/455370627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 msgr2=0x55cbf44f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 -- 172.21.15.38:0/455370627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 msgr2=0x55cbf44e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44e7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80ffff640 1 -- 172.21.15.38:0/455370627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf42297d0 con 0x55cbf44e7850 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.433+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe80d7fa640 1 -- 172.21.15.38:0/2696020966 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fe8080cfff0 con 0x7fe810106630 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe80ffff640 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fe80000dc40 tx=0x7fe80000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbadceb2f106ca91 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.438 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe814fc4640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 0x7fe7f80a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe808098b30 tx=0x7fe8080a0040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe7f67fc640 1 -- 172.21.15.38:0/455370627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800005020 con 0x55cbf44e7850 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe7f67fc640 1 -- 172.21.15.38:0/455370627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8000147e0 con 0x55cbf44e7850 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.434+0000 7fe7f67fc640 1 -- 172.21.15.38:0/455370627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8000049e0 con 0x55cbf44e7850 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- 172.21.15.38:0/455370627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 msgr2=0x55cbf44efef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fe80000dc40 tx=0x7fe80000f730 comp rx=0 tx=0).stop 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- 172.21.15.38:0/455370627 shutdown_connections 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44efef0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 --2- 172.21.15.38:0/455370627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- 172.21.15.38:0/455370627 >> 172.21.15.38:0/455370627 conn(0x55cbf44e1660 msgr2=0x55cbf44e1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- 172.21.15.38:0/455370627 shutdown_connections 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- 172.21.15.38:0/455370627 wait complete. 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.435+0000 7fe819610740 1 -- start start 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 Processor -- start 2024-04-22T00:11:57.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44ed970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55cbf44f3220 con 0x55cbf44e6f30 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55cbf4406e00 con 0x55cbf44e7850 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cbf44f33a0 con 0x55cbf44f0430 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe819610740 10 client.0 _ll_register_callbacks cb 0x55cbf4441eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59116/0 (socket says 172.21.15.38:59116) 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1238848913 learned_addr learned my addr 172.21.15.38:0/1238848913 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80ffff640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1238848913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 msgr2=0x55cbf44edeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44edeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 msgr2=0x55cbf44ed970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44ed970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.436+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1238848913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf415f3f0 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fe804036040 tx=0x7fe80402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdfdf7946153bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804046070 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fe8040037f0 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8000147e0 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe804004490 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 10 client.18678 fetch_fsmap learned FSMap version 40 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cbf411f0f0 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8000147e0 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe804034c40 con 0x55cbf44e6f30 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 10 client.18678 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T00:11:57.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 10 client.18678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 20 client.18678 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf4500630 0x55cbf4502a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 4 client.18678 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.437+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbf4458aa0 con 0x55cbf4500630 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.438+0000 7fe80ffff640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf4500630 0x55cbf4502a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.438+0000 7fe80ffff640 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf4500630 0x55cbf4502a40 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2716f8780e04e14 server_cookie=e0b60b481c68d372 in_seq=0 out_seq=0 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.438+0000 7fe7ef7fe640 10 client.18678 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/1238848913 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbf4458aa0 con 0x55cbf4500630 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe7ef7fe640 10 client.18678 handle_command_reply: tid=0 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 client.18678 shutdown 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 msgr2=0x55cbf44ed430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fe804036040 tx=0x7fe80402bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 shutdown_connections 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44f0430 0x55cbf44edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44ed970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf4500630 0x55cbf4502a40 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 --2- 172.21.15.38:0/1238848913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e6f30 0x55cbf44ed430 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe7ef7fe640 0 client.18678 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 >> 172.21.15.38:0/1238848913 conn(0x55cbf44e1660 msgr2=0x55cbf44e33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 shutdown_connections 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 1 -- 172.21.15.38:0/1238848913 wait complete. 2024-04-22T00:11:57.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.439+0000 7fe819610740 20 client.18678 trim_cache size 0 max 16384 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 Processor -- start 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 -- start start 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e1660 0x55cbf44efb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101ac9e0 con 0x55cbf44e7850 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101ae580 con 0x55cbf44e6f30 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe819610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101ad0e0 con 0x55cbf44e1660 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59128/0 (socket says 172.21.15.38:59128) 2024-04-22T00:11:57.460 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.457+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1603468380 learned_addr learned my addr 172.21.15.38:0/1603468380 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e1660 0x55cbf44efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80ffff640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1603468380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e1660 msgr2=0x55cbf44efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e1660 0x55cbf44efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1603468380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 msgr2=0x55cbf44fc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44fc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 -- 172.21.15.38:0/1603468380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf44eb980 con 0x55cbf44e7850 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80ffff640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fe804005c40 tx=0x7fe804005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3449db26e9c225 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe7f5ffb640 1 -- 172.21.15.38:0/1603468380 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804046070 con 0x55cbf44e7850 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe7f5ffb640 1 -- 172.21.15.38:0/1603468380 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe804041680 con 0x55cbf44e7850 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.458+0000 7fe7f5ffb640 1 -- 172.21.15.38:0/1603468380 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804003d90 con 0x55cbf44e7850 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- 172.21.15.38:0/1603468380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 msgr2=0x55cbf44e31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.461 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fe804005c40 tx=0x7fe804005a20 comp rx=0 tx=0).stop 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- 172.21.15.38:0/1603468380 shutdown_connections 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e7850 0x55cbf44e31a0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e6f30 0x55cbf44fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 --2- 172.21.15.38:0/1603468380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e1660 0x55cbf44efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- 172.21.15.38:0/1603468380 >> 172.21.15.38:0/1603468380 conn(0x55cbf44e7fa0 msgr2=0x55cbf44e2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- 172.21.15.38:0/1603468380 shutdown_connections 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- 172.21.15.38:0/1603468380 wait complete. 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 -- start start 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.459+0000 7fe819610740 1 Processor -- start 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e6f30 0x55cbf44f4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101adeb0 con 0x55cbf44e1660 2024-04-22T00:11:57.462 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101ad7e0 con 0x55cbf44e7850 2024-04-22T00:11:57.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8101aa9e0 con 0x55cbf44e6f30 2024-04-22T00:11:57.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 10 client.0 _ll_register_callbacks cb 0x7fe8080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:57.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59134/0 (socket says 172.21.15.38:59134) 2024-04-22T00:11:57.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 -- 172.21.15.38:0/3958874423 learned_addr learned my addr 172.21.15.38:0/3958874423 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80ffff640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e6f30 0x55cbf44f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 -- 172.21.15.38:0/3958874423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e6f30 msgr2=0x55cbf44f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e6f30 0x55cbf44f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 -- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 msgr2=0x55cbf44f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44f4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 -- 172.21.15.38:0/3958874423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf4565ce0 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe80f7fe640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe80402cfd0 tx=0x7fe804003a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a045949fef2d4631 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe7ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804046040 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fe804041980 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.460+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe804041680 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe804041b80 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 10 client.18684 fetch_fsmap learned FSMap version 40 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cbf4566fd0 con 0x55cbf44e1660 2024-04-22T00:11:57.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe804041680 con 0x55cbf44e1660 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 40) ==== 8370+0+0 (secure 0 0 0) 0x7fe804034440 con 0x55cbf44e1660 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 10 client.18684 fetch_fsmap finished waiting for FSMap version 40 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 10 client.18684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 20 client.18684 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf44f0430 0x55cbf4569640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 4 client.18684 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbf4452df0 con 0x55cbf44f0430 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.461+0000 7fe80ffff640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf44f0430 0x55cbf4569640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe7ef7fe640 10 client.18684 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe80ffff640 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf44f0430 0x55cbf4569640 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ff4e2298c9faa6 server_cookie=d81922dbd41e15bf in_seq=0 out_seq=0 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe7ef7fe640 1 -- 172.21.15.38:0/3958874423 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cbf4452df0 con 0x55cbf44f0430 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe7ef7fe640 10 client.18684 handle_command_reply: tid=0 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 client.18684 shutdown 2024-04-22T00:11:57.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 msgr2=0x55cbf44f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe80402cfd0 tx=0x7fe804003a30 comp rx=0 tx=0).stop 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 shutdown_connections 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cbf44e7850 0x55cbf44f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbf44e6f30 0x55cbf44f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55cbf44f0430 0x55cbf4569640 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 --2- 172.21.15.38:0/3958874423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cbf44e1660 0x55cbf44f4150 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe7ef7fe640 0 client.18684 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 >> 172.21.15.38:0/3958874423 conn(0x55cbf44e7fa0 msgr2=0x55cbf44e3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 shutdown_connections 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.462+0000 7fe819610740 1 -- 172.21.15.38:0/3958874423 wait complete. 2024-04-22T00:11:57.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe819610740 20 client.18684 trim_cache size 0 max 16384 2024-04-22T00:11:57.469 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:11:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:11:57.469 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 msgr2=0x7fe7f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 0x7fe7f80a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe808098b30 tx=0x7fe8080a0040 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 msgr2=0x7fe81019c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fe80800eb10 tx=0x7fe80800c2d0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 shutdown_connections 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fe7f80a2d00 0x7fe7f80a51c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe810106630 0x7fe81019c990 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe810100fe0 0x7fe810198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 --2- 172.21.15.38:0/2696020966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe810100620 0x7fe8101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 >> 172.21.15.38:0/2696020966 conn(0x7fe8100776a0 msgr2=0x7fe81006a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 shutdown_connections 2024-04-22T00:11:57.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.463+0000 7fe816a27640 1 -- 172.21.15.38:0/2696020966 wait complete. 2024-04-22T00:11:57.471 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:11:57.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:11:57.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- 172.21.15.38:0/2147580013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 msgr2=0x7ff0e010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2147580013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e010e5a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009ca0 tx=0x7ff0d802c180 comp rx=0 tx=0).stop 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- 172.21.15.38:0/2147580013 shutdown_connections 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2147580013 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e010e5a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2147580013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2147580013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 0x7ff0e0107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- 172.21.15.38:0/2147580013 >> 172.21.15.38:0/2147580013 conn(0x7ff0e00788d0 msgr2=0x7ff0e0078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- 172.21.15.38:0/2147580013 shutdown_connections 2024-04-22T00:11:57.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- 172.21.15.38:0/2147580013 wait complete. 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 Processor -- start 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.578+0000 7ff0e7177640 1 -- start start 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 0x7ff0e019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e0111b10 con 0x7ff0e0102160 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e0111990 con 0x7ff0e0107750 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e0111c90 con 0x7ff0e01017c0 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59146/0 (socket says 172.21.15.38:59146) 2024-04-22T00:11:57.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 -- 172.21.15.38:0/2327881170 learned_addr learned my addr 172.21.15.38:0/2327881170 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e4f13640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e5714640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 0x7ff0e019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 -- 172.21.15.38:0/2327881170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 msgr2=0x7ff0e0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 -- 172.21.15.38:0/2327881170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 msgr2=0x7ff0e019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 0x7ff0e019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 -- 172.21.15.38:0/2327881170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e019cfc0 con 0x7ff0e0102160 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e4f13640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d7fff640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d000cd80 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d0005500 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0010690 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.579+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e019d2b0 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.580+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e01a4b40 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.581+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e00689b0 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.581+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff0d00046c0 con 0x7ff0e0102160 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.582+0000 7ff0d5ffb640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 0x7ff0c80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.582+0000 7ff0e4f13640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 0x7ff0c80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.582+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff0d00cfc10 con 0x7ff0e0102160 2024-04-22T00:11:57.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.583+0000 7ff0e4f13640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 0x7ff0c80a5240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc005e00 tx=0x7ff0dc005c70 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.587+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff0d009c4a0 con 0x7ff0e0102160 2024-04-22T00:11:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.809+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0e0199320 con 0x7ff0e0102160 2024-04-22T00:11:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.810+0000 7ff0d5ffb640 1 -- 172.21.15.38:0/2327881170 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11936 (secure 0 0 0) 0x7ff0d009bda0 con 0x7ff0e0102160 2024-04-22T00:11:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:11:57.808 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:11:56.566478+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,24487,24460,24448,24455]},"id":1}]} 2024-04-22T00:11:57.808 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 41 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 msgr2=0x7ff0c80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 0x7ff0c80a5240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc005e00 tx=0x7ff0dc005c70 comp rx=0 tx=0).stop 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 msgr2=0x7ff0e0198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000edf0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 shutdown_connections 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff0c80a2d80 0x7ff0c80a5240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0e0107750 0x7ff0e019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff0e0102160 0x7ff0e0198510 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 --2- 172.21.15.38:0/2327881170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0e01017c0 0x7ff0e0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.812+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 >> 172.21.15.38:0/2327881170 conn(0x7ff0e00788d0 msgr2=0x7ff0e01003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.813+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 shutdown_connections 2024-04-22T00:11:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.813+0000 7ff0e7177640 1 -- 172.21.15.38:0/2327881170 wait complete. 2024-04-22T00:11:57.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 -- 172.21.15.38:0/53095494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c107720 msgr2=0x7f8f3c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 --2- 172.21.15.38:0/53095494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c107720 0x7f8f3c107b00 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009930 tx=0x7f8f3002bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 -- 172.21.15.38:0/53095494 shutdown_connections 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 --2- 172.21.15.38:0/53095494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 0x7f8f3c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 --2- 172.21.15.38:0/53095494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c069d40 0x7f8f3c06a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 --2- 172.21.15.38:0/53095494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c107720 0x7f8f3c107b00 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 -- 172.21.15.38:0/53095494 >> 172.21.15.38:0/53095494 conn(0x7f8f3c076440 msgr2=0x7f8f3c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 -- 172.21.15.38:0/53095494 shutdown_connections 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.928+0000 7f8f42955640 1 -- 172.21.15.38:0/53095494 wait complete. 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 Processor -- start 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 -- start start 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 0x7f8f3c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 0x7f8f3c19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c111b30 con 0x7f8f3c069d40 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1119b0 con 0x7f8f3c107720 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f42955640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c111cb0 con 0x7f8f3c06a700 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59150/0 (socket says 172.21.15.38:59150) 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.929+0000 7f8f3bfff640 1 -- 172.21.15.38:0/3718239776 learned_addr learned my addr 172.21.15.38:0/3718239776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 0x7f8f3c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 0x7f8f3c19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 -- 172.21.15.38:0/3718239776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 msgr2=0x7f8f3c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 0x7f8f3c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 -- 172.21.15.38:0/3718239776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 msgr2=0x7f8f3c19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 0x7f8f3c19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 -- 172.21.15.38:0/3718239776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c19f220 con 0x7f8f3c069d40 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 0x7f8f3c19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009230 tx=0x7f8f3002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f397fa640 1 -- 172.21.15.38:0/3718239776 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30041070 con 0x7f8f3c069d40 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3c19f4b0 con 0x7f8f3c069d40 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f397fa640 1 -- 172.21.15.38:0/3718239776 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f30004450 con 0x7f8f3c069d40 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f397fa640 1 -- 172.21.15.38:0/3718239776 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30034440 con 0x7f8f3c069d40 2024-04-22T00:11:57.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.930+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f3c1a6da0 con 0x7f8f3c069d40 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.932+0000 7f8f397fa640 1 -- 172.21.15.38:0/3718239776 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8f30044070 con 0x7f8f3c069d40 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 Processor -- start 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 -- start start 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b020 0x55e44025b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e4401bf530 con 0x55e440264520 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e440167830 con 0x55e44025b940 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f4560e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e4401bb4f0 con 0x55e44025b020 2024-04-22T00:11:57.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f397fa640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 0x7f8f240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59160/0 (socket says 172.21.15.38:59160) 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/618456904 learned_addr learned my addr 172.21.15.38:0/618456904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b020 0x55e44025b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f397fa640 1 -- 172.21.15.38:0/3718239776 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8f300f11a0 con 0x7f8f3c069d40 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 0x7f8f240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/618456904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b020 msgr2=0x55e44025b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b020 0x55e44025b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/618456904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 msgr2=0x55e440263fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.933+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/618456904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e44018b2c0 con 0x55e440264520 2024-04-22T00:11:57.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00bc90 tx=0x7f8f2c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb102458b5478e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 0x7f8f240a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009fd0 tx=0x7f8f34005bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f227fc640 1 -- 172.21.15.38:0/618456904 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c005220 con 0x55e440264520 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f227fc640 1 -- 172.21.15.38:0/618456904 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c00be40 con 0x55e440264520 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.934+0000 7f8f227fc640 1 -- 172.21.15.38:0/618456904 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c011730 con 0x55e440264520 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- 172.21.15.38:0/618456904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 msgr2=0x55e440266910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00bc90 tx=0x7f8f2c0107a0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- 172.21.15.38:0/618456904 shutdown_connections 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440266910 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- 172.21.15.38:0/618456904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b020 0x55e44025b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- 172.21.15.38:0/618456904 >> 172.21.15.38:0/618456904 conn(0x55e440255750 msgr2=0x55e440255b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- 172.21.15.38:0/618456904 shutdown_connections 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- 172.21.15.38:0/618456904 wait complete. 2024-04-22T00:11:57.939 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 -- start start 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 Processor -- start 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e440263e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.935+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 0x55e440261210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e440267310 con 0x55e440264520 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e44017af10 con 0x55e44025b020 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e440267490 con 0x55e44025b940 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 10 client.0 _ll_register_callbacks cb 0x55e43fed2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59162/0 (socket says 172.21.15.38:59162) 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 -- 172.21.15.38:0/929079009 learned_addr learned my addr 172.21.15.38:0/929079009 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 0x55e440261210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e440263e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 -- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 msgr2=0x55e440261210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 0x55e440261210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 -- 172.21.15.38:0/929079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 msgr2=0x55e440263e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e440263e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 -- 172.21.15.38:0/929079009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e44014cc80 con 0x55e440264520 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 0x55e440261210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e440263e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8f34012f60 tx=0x7f8f34004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15820297beecb87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34025070 con 0x55e440264520 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8f3400fc80 con 0x55e440264520 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e4401cca70 con 0x55e440264520 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.936+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8f34033050 con 0x55e440264520 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 10 client.18702 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:57.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e43fed33f0 con 0x55e440264520 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e4401cca70 con 0x55e440264520 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f8f3401e040 con 0x55e440264520 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 10 client.18702 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 10 client.18702 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 20 client.18702 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e440274600 0x55e440276a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 4 client.18702 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4401c6dc0 con 0x55e440274600 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.937+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e440274600 0x55e440276a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e440274600 0x55e440276a80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=269934080de9cd34 server_cookie=a64ee99cf45d962a in_seq=0 out_seq=0 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f1b7fe640 10 client.18702 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/929079009 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e4401c6dc0 con 0x55e440274600 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f1b7fe640 10 client.18702 handle_command_reply: tid=0 2024-04-22T00:11:57.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f4560e740 1 client.18702 shutdown 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 msgr2=0x55e440261770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.938+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8f34012f60 tx=0x7f8f34004ab0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 shutdown_connections 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e440274600 0x55e440276a80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e440264520 0x55e440261770 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e44025b940 0x55e440261210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f1b7fe640 0 client.18702 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 --2- 172.21.15.38:0/929079009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e440263e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 >> 172.21.15.38:0/929079009 conn(0x55e440255750 msgr2=0x55e4402574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 shutdown_connections 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 1 -- 172.21.15.38:0/929079009 wait complete. 2024-04-22T00:11:57.943 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.939+0000 7f8f4560e740 20 client.18702 trim_cache size 0 max 16384 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 Processor -- start 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 -- start start 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e440263e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1b0df0 con 0x55e44025b020 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1b2960 con 0x55e44025b940 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f4560e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1b14f0 con 0x55e440264520 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f3bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59166/0 (socket says 172.21.15.38:59166) 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.952+0000 7f8f3bfff640 1 -- 172.21.15.38:0/824706638 learned_addr learned my addr 172.21.15.38:0/824706638 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e440263e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 -- 172.21.15.38:0/824706638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 msgr2=0x55e440263e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e440263e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 -- 172.21.15.38:0/824706638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 msgr2=0x55e440263940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 -- 172.21.15.38:0/824706638 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e440261ae0 con 0x55e44025b020 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8f30000c00 tx=0x7f8f30088070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa9e590a595ebeb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f21ffb640 1 -- 172.21.15.38:0/824706638 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3004b040 con 0x55e44025b020 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f21ffb640 1 -- 172.21.15.38:0/824706638 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f300bdb40 con 0x55e44025b020 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.953+0000 7f8f21ffb640 1 -- 172.21.15.38:0/824706638 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f300ee7e0 con 0x55e44025b020 2024-04-22T00:11:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- 172.21.15.38:0/824706638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 msgr2=0x55e440270200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8f30000c00 tx=0x7f8f30088070 comp rx=0 tx=0).stop 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- 172.21.15.38:0/824706638 shutdown_connections 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e440263e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b940 0x55e440263940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- 172.21.15.38:0/824706638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b020 0x55e440270200 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- 172.21.15.38:0/824706638 >> 172.21.15.38:0/824706638 conn(0x55e440255750 msgr2=0x55e440254850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- 172.21.15.38:0/824706638 shutdown_connections 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- 172.21.15.38:0/824706638 wait complete. 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- start start 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 Processor -- start 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e44026cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e4402685b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1b2290 con 0x55e44025b940 2024-04-22T00:11:57.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1b1bc0 con 0x55e44025b020 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c1aedf0 con 0x55e440264520 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.954+0000 7f8f4560e740 10 client.0 _ll_register_callbacks cb 0x7f8f34009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59180/0 (socket says 172.21.15.38:59180) 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/1407797231 learned_addr learned my addr 172.21.15.38:0/1407797231 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f40ef2640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e4402685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e44026cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 msgr2=0x55e4402685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e4402685b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/1407797231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 msgr2=0x55e44026cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e44026cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 -- 172.21.15.38:0/1407797231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43ff9d7d0 con 0x55e44025b940 2024-04-22T00:11:57.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f3b7fe640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x55e4402568e0 tx=0x7f8f2c0058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794093ea185d4442 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c005550 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8f2c004020 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f300bdb40 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8f2c031050 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f4560e740 10 client.18708 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.955+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e4402daa90 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f300bdb40 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f8f2c025070 con 0x55e44025b940 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 10 client.18708 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 10 client.18708 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 20 client.18708 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e4402daf20 0x55e4402dd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:57.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 4 client.18708 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4401446e0 con 0x55e4402daf20 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e4402daf20 0x55e4402dd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f3bfff640 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e4402daf20 0x55e4402dd340 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9378d868f7aeac2a server_cookie=9d9d0a13eb8f92ff in_seq=0 out_seq=0 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.956+0000 7f8f1b7fe640 10 client.18708 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f1b7fe640 1 -- 172.21.15.38:0/1407797231 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4401446e0 con 0x55e4402daf20 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f1b7fe640 10 client.18708 handle_command_reply: tid=0 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 client.18708 shutdown 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 msgr2=0x55e440267fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x55e4402568e0 tx=0x7f8f2c0058c0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 shutdown_connections 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x55e4402daf20 0x55e4402dd340 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e440264520 0x55e4402685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e44025b940 0x55e440267fa0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f1b7fe640 0 client.18708 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 --2- 172.21.15.38:0/1407797231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e44025b020 0x55e44026cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 >> 172.21.15.38:0/1407797231 conn(0x55e440255750 msgr2=0x55e440253fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 shutdown_connections 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 1 -- 172.21.15.38:0/1407797231 wait complete. 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f4560e740 20 client.18708 trim_cache size 0 max 16384 2024-04-22T00:11:57.961 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 msgr2=0x7f8f240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 0x7f8f240a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009fd0 tx=0x7f8f34005bd0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 msgr2=0x7f8f3c19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.957+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009230 tx=0x7f8f3002cc70 comp rx=0 tx=0).stop 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 shutdown_connections 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8f240a2d00 0x7f8f240a51c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c107720 0x7f8f3c19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f3c06a700 0x7f8f3c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 --2- 172.21.15.38:0/3718239776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f3c069d40 0x7f8f3c19a230 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:57.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 >> 172.21.15.38:0/3718239776 conn(0x7f8f3c076440 msgr2=0x7f8f3c10cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:57.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 shutdown_connections 2024-04-22T00:11:57.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:57.958+0000 7f8f42955640 1 -- 172.21.15.38:0/3718239776 wait complete. 2024-04-22T00:11:57.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1431587656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c106630 msgr2=0x7f9f1c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1431587656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c106630 0x7f9f1c106a10 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009930 tx=0x7f9f0402bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1431587656 shutdown_connections 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1431587656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c100fe0 0x7f9f1c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1431587656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100620 0x7f9f1c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1431587656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c106630 0x7f9f1c106a10 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1431587656 >> 172.21.15.38:0/1431587656 conn(0x7f9f1c0776a0 msgr2=0x7f9f1c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1431587656 shutdown_connections 2024-04-22T00:11:58.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.073+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1431587656 wait complete. 2024-04-22T00:11:58.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 Processor -- start 2024-04-22T00:11:58.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 -- start start 2024-04-22T00:11:58.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100fe0 0x7f9f1c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c106630 0x7f9f1c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c110a40 con 0x7f9f1c100620 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1108c0 con 0x7f9f1c106630 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f21cb5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c110bc0 con 0x7f9f1c100fe0 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.074+0000 7f9f1b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59202/0 (socket says 172.21.15.38:59202) 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 -- 172.21.15.38:0/1563867524 learned_addr learned my addr 172.21.15.38:0/1563867524 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c106630 0x7f9f1c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100fe0 0x7f9f1c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 -- 172.21.15.38:0/1563867524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100fe0 msgr2=0x7f9f1c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100fe0 0x7f9f1c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 -- 172.21.15.38:0/1563867524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c106630 msgr2=0x7f9f1c19c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c106630 0x7f9f1c19c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 -- 172.21.15.38:0/1563867524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c19d0e0 con 0x7f9f1c100620 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009230 tx=0x7f9f0402cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f18ff9640 1 -- 172.21.15.38:0/1563867524 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04041070 con 0x7f9f1c100620 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1c19d370 con 0x7f9f1c100620 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f18ff9640 1 -- 172.21.15.38:0/1563867524 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f04004450 con 0x7f9f1c100620 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f18ff9640 1 -- 172.21.15.38:0/1563867524 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04034440 con 0x7f9f1c100620 2024-04-22T00:11:58.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.075+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1c1a4c60 con 0x7f9f1c100620 2024-04-22T00:11:58.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f18ff9640 1 -- 172.21.15.38:0/1563867524 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9f04044070 con 0x7f9f1c100620 2024-04-22T00:11:58.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 Processor -- start 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 -- start start 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e59d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d940 0x55e46e5a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e46e501530 con 0x55e46e5a6520 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e46e4a9830 con 0x55e46e59d940 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f2497a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e46e4fd4f0 con 0x55e46e59d020 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f18ff9640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 0x7f9f080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59218/0 (socket says 172.21.15.38:59218) 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1affd640 1 -- 172.21.15.38:0/1379762865 learned_addr learned my addr 172.21.15.38:0/1379762865 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e59d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 0x7f9f080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f18ff9640 1 -- 172.21.15.38:0/1563867524 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9f040f1000 con 0x7f9f1c100620 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.078+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d940 0x55e46e5a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 -- 172.21.15.38:0/1379762865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 msgr2=0x55e46e59d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e59d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 -- 172.21.15.38:0/1379762865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d940 msgr2=0x55e46e5a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d940 0x55e46e5a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 -- 172.21.15.38:0/1379762865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46e4cd2c0 con 0x55e46e5a6520 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009ca0 tx=0x7f9f0c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df635f5e6c93df7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 0x7f9f080a51c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009fd0 tx=0x7f9f14009600 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f11ffb640 1 -- 172.21.15.38:0/1379762865 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c01e070 con 0x55e46e5a6520 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f11ffb640 1 -- 172.21.15.38:0/1379762865 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c009e50 con 0x55e46e5a6520 2024-04-22T00:11:58.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.079+0000 7f9f11ffb640 1 -- 172.21.15.38:0/1379762865 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00e8e0 con 0x55e46e5a6520 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- 172.21.15.38:0/1379762865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 msgr2=0x55e46e5a8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009ca0 tx=0x7f9f0c011810 comp rx=0 tx=0).stop 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- 172.21.15.38:0/1379762865 shutdown_connections 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e5a6520 0x55e46e5a8910 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d940 0x55e46e5a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1379762865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e59d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- 172.21.15.38:0/1379762865 >> 172.21.15.38:0/1379762865 conn(0x55e46e597750 msgr2=0x55e46e597b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- 172.21.15.38:0/1379762865 shutdown_connections 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- 172.21.15.38:0/1379762865 wait complete. 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 -- start start 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.080+0000 7f9f2497a740 1 Processor -- start 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e5a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e5a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e46e5a9310 con 0x55e46e59d940 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55e46e4bcf10 con 0x55e46e59d020 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e46e5a9490 con 0x55e46e5a6520 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 10 client.0 _ll_register_callbacks cb 0x55e46e214be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59230/0 (socket says 172.21.15.38:59230) 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 -- 172.21.15.38:0/3971574719 learned_addr learned my addr 172.21.15.38:0/3971574719 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e5a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e5a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 -- 172.21.15.38:0/3971574719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 msgr2=0x55e46e5a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e5a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 -- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 msgr2=0x55e46e5a5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e5a5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 -- 172.21.15.38:0/3971574719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46e48ec80 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e5a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e5a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f1affd640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009c70 tx=0x7f9f0c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a9ac37ca2cda86 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9efaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c029030 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9f0c004020 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.081+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f0c009e50 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9f0c033050 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 10 client.18720 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e46e2153f0 con 0x55e46e59d940 2024-04-22T00:11:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c009e50 con 0x55e46e59d940 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f9f0c00e6f0 con 0x55e46e59d940 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 10 client.18720 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 10 client.18720 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 20 client.18720 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e5b6550 0x55e46e5b8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 4 client.18720 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e46e50ea70 con 0x55e46e5b6550 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.082+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e5b6550 0x55e46e5b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.083+0000 7f9efaffd640 10 client.18720 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.083+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e5b6550 0x55e46e5b8940 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef65480c872900cf server_cookie=e2cc1e5f828f60e1 in_seq=0 out_seq=0 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.083+0000 7f9efaffd640 1 -- 172.21.15.38:0/3971574719 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e46e50ea70 con 0x55e46e5b6550 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.083+0000 7f9efaffd640 10 client.18720 handle_command_reply: tid=0 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 client.18720 shutdown 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 msgr2=0x55e46e5a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009c70 tx=0x7f9f0c005720 comp rx=0 tx=0).stop 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 shutdown_connections 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e5a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e5b6550 0x55e46e5b8940 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e5a3250 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 --2- 172.21.15.38:0/3971574719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e5a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9efaffd640 0 client.18720 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:58.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 >> 172.21.15.38:0/3971574719 conn(0x55e46e597750 msgr2=0x55e46e5994a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 shutdown_connections 2024-04-22T00:11:58.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 1 -- 172.21.15.38:0/3971574719 wait complete. 2024-04-22T00:11:58.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.084+0000 7f9f2497a740 20 client.18720 trim_cache size 0 max 16384 2024-04-22T00:11:58.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 Processor -- start 2024-04-22T00:11:58.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 -- start start 2024-04-22T00:11:58.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e5b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e5a6520 0x55e46e5a5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1aebc0 con 0x55e46e59d940 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1b0730 con 0x55e46e5a6520 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.097+0000 7f9f2497a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1af2c0 con 0x55e46e59d020 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59238/0 (socket says 172.21.15.38:59238) 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 -- 172.21.15.38:0/2633102068 learned_addr learned my addr 172.21.15.38:0/2633102068 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e5b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 -- 172.21.15.38:0/2633102068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 msgr2=0x55e46e5b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e5b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 -- 172.21.15.38:0/2633102068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e5a6520 msgr2=0x55e46e5a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e5a6520 0x55e46e5a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 -- 172.21.15.38:0/2633102068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46e5a3b60 con 0x55e46e59d940 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e5b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f1affd640 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00ea90 tx=0x7f9f0c009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35de54ea57b945d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f117fa640 1 -- 172.21.15.38:0/2633102068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c029070 con 0x55e46e59d940 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f117fa640 1 -- 172.21.15.38:0/2633102068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c00ed00 con 0x55e46e59d940 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.098+0000 7f9f117fa640 1 -- 172.21.15.38:0/2633102068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c01e040 con 0x55e46e59d940 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- 172.21.15.38:0/2633102068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 msgr2=0x55e46e596850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00ea90 tx=0x7f9f0c009af0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- 172.21.15.38:0/2633102068 shutdown_connections 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e5a6520 0x55e46e5a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e596850 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- 172.21.15.38:0/2633102068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e59d020 0x55e46e5b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- 172.21.15.38:0/2633102068 >> 172.21.15.38:0/2633102068 conn(0x55e46e597750 msgr2=0x55e46e597b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- 172.21.15.38:0/2633102068 shutdown_connections 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- 172.21.15.38:0/2633102068 wait complete. 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- start start 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 Processor -- start 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e6149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e615460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1b0060 con 0x55e46e59d940 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1af990 con 0x55e46e59d020 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.099+0000 7f9f2497a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c1acbc0 con 0x55e46e5a6520 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f2497a740 10 client.0 _ll_register_callbacks cb 0x7f9f0c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59252/0 (socket says 172.21.15.38:59252) 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 -- 172.21.15.38:0/1131928471 learned_addr learned my addr 172.21.15.38:0/1131928471 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1bfff640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e615460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e6149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 -- 172.21.15.38:0/1131928471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 msgr2=0x55e46e615460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e615460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 -- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 msgr2=0x55e46e6149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e6149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 -- 172.21.15.38:0/1131928471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46e59d770 con 0x55e46e59d940 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e6149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f1affd640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x55e46e5972d0 tx=0x7f9f0c009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eeaf27369fbf3b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9efaffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c01e070 con 0x55e46e59d940 2024-04-22T00:11:58.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9f0c0035f0 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f0c00ed00 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9f0c029040 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f2497a740 10 client.18726 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.100+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e46e2df7d0 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c00ed00 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f9f0c037740 con 0x55e46e59d940 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 10 client.18726 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 10 client.18726 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 20 client.18726 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e620f80 0x55e46e6233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 4 client.18726 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e46e508dc0 con 0x55e46e620f80 2024-04-22T00:11:58.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e620f80 0x55e46e6233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9efaffd640 10 client.18726 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9f1b7fe640 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e620f80 0x55e46e6233a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7307ab60aceba9a1 server_cookie=6903287cf8a35f18 in_seq=0 out_seq=0 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9efaffd640 1 -- 172.21.15.38:0/1131928471 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e46e508dc0 con 0x55e46e620f80 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.101+0000 7f9efaffd640 10 client.18726 handle_command_reply: tid=0 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 client.18726 shutdown 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 msgr2=0x55e46e614f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x55e46e5972d0 tx=0x7f9f0c009c60 comp rx=0 tx=0).stop 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 shutdown_connections 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e46e5a6520 0x55e46e615460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55e46e620f80 0x55e46e6233a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e46e59d940 0x55e46e614f20 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 --2- 172.21.15.38:0/1131928471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e46e59d020 0x55e46e6149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9efaffd640 0 client.18726 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 >> 172.21.15.38:0/1131928471 conn(0x55e46e597750 msgr2=0x55e46e596850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 shutdown_connections 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 1 -- 172.21.15.38:0/1131928471 wait complete. 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f2497a740 20 client.18726 trim_cache size 0 max 16384 2024-04-22T00:11:58.108 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 msgr2=0x7f9f080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 0x7f9f080a51c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009fd0 tx=0x7f9f14009600 comp rx=0 tx=0).stop 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 msgr2=0x7f9f1c1980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009230 tx=0x7f9f0402cba0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 shutdown_connections 2024-04-22T00:11:58.109 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f080a2d00 0x7f9f080a51c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f1c106630 0x7f9f1c19c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f1c100fe0 0x7f9f1c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 --2- 172.21.15.38:0/1563867524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f1c100620 0x7f9f1c1980f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.102+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 >> 172.21.15.38:0/1563867524 conn(0x7f9f1c0776a0 msgr2=0x7f9f1c06a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.103+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 shutdown_connections 2024-04-22T00:11:58.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.103+0000 7f9f21cb5640 1 -- 172.21.15.38:0/1563867524 wait complete. 2024-04-22T00:11:58.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:11:58.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 -- 172.21.15.38:0/1568393637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f826810bd70 msgr2=0x7f826810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 --2- 172.21.15.38:0/1568393637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f826810bd70 0x7f826810e1a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f8260009900 tx=0x7f826002bcd0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 -- 172.21.15.38:0/1568393637 shutdown_connections 2024-04-22T00:11:58.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 --2- 172.21.15.38:0/1568393637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f826810bd70 0x7f826810e1a0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.218 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 --2- 172.21.15.38:0/1568393637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681032c0 0x7f826810b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 --2- 172.21.15.38:0/1568393637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681029a0 0x7f8268102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 -- 172.21.15.38:0/1568393637 >> 172.21.15.38:0/1568393637 conn(0x7f82680769c0 msgr2=0x7f8268076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 -- 172.21.15.38:0/1568393637 shutdown_connections 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 -- 172.21.15.38:0/1568393637 wait complete. 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.218+0000 7f826d8df640 1 Processor -- start 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 -- start start 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.219 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 0x7f82681984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826810bd70 0x7f826819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8268111b00 con 0x7f82681029a0 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8268111980 con 0x7f82681032c0 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f826d8df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8268111c80 con 0x7f826810bd70 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59270/0 (socket says 172.21.15.38:59270) 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 -- 172.21.15.38:0/1269497414 learned_addr learned my addr 172.21.15.38:0/1269497414 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f82677fe640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826810bd70 0x7f826819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f82667fc640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 0x7f82681984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 -- 172.21.15.38:0/1269497414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826810bd70 msgr2=0x7f826819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826810bd70 0x7f826819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 -- 172.21.15.38:0/1269497414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 msgr2=0x7f82681984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 0x7f82681984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f8266ffd640 1 -- 172.21.15.38:0/1269497414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826819cf90 con 0x7f82681029a0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.219+0000 7f82667fc640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 0x7f82681984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f8266ffd640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f825800ebc0 tx=0x7f825800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f826c8dd640 1 -- 172.21.15.38:0/1269497414 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258010630 con 0x7f82681029a0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f826c8dd640 1 -- 172.21.15.38:0/1269497414 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8258010cb0 con 0x7f82681029a0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826819d280 con 0x7f82681029a0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f826c8dd640 1 -- 172.21.15.38:0/1269497414 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258018bd0 con 0x7f82681029a0 2024-04-22T00:11:58.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.220+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82681a4b10 con 0x7f82681029a0 2024-04-22T00:11:58.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f826c8dd640 1 -- 172.21.15.38:0/1269497414 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f82580107d0 con 0x7f82681029a0 2024-04-22T00:11:58.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 Processor -- start 2024-04-22T00:11:58.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 -- start start 2024-04-22T00:11:58.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 0x55d51fbab400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab940 0x55d51fbb3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d51fb0f530 con 0x55d51fbb4520 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d51fab7830 con 0x55d51fbab020 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.222+0000 7f82705a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d51fb0b4f0 con 0x55d51fbab940 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59276/0 (socket says 172.21.15.38:59276) 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 -- 172.21.15.38:0/2340086784 learned_addr learned my addr 172.21.15.38:0/2340086784 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab940 0x55d51fbb3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f826c8dd640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 0x7f82500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82677fe640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 0x55d51fbab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 -- 172.21.15.38:0/2340086784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab940 msgr2=0x55d51fbb3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab940 0x55d51fbb3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 -- 172.21.15.38:0/2340086784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 msgr2=0x55d51fbab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 0x55d51fbab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 -- 172.21.15.38:0/2340086784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d51fadb2c0 con 0x55d51fbb4520 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82677fe640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 0x55d51fbab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82677fe640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 0x7f82500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f826c8dd640 1 -- 172.21.15.38:0/1269497414 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f82580cfeb0 con 0x7f82681029a0 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f82667fc640 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f825c009b90 tx=0x7f825c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ded6bd624e5da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f824dffb640 1 -- 172.21.15.38:0/2340086784 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c016070 con 0x55d51fbb4520 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f824dffb640 1 -- 172.21.15.38:0/2340086784 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c005500 con 0x55d51fbb4520 2024-04-22T00:11:58.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.223+0000 7f824dffb640 1 -- 172.21.15.38:0/2340086784 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c011470 con 0x55d51fbb4520 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- 172.21.15.38:0/2340086784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 msgr2=0x55d51fbb6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82677fe640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 0x7f82500a51c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f826002efd0 tx=0x7f826002cf90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f825c009b90 tx=0x7f825c00e6c0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- 172.21.15.38:0/2340086784 shutdown_connections 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbb4520 0x55d51fbb6910 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab940 0x55d51fbb3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 --2- 172.21.15.38:0/2340086784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbab020 0x55d51fbab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- 172.21.15.38:0/2340086784 >> 172.21.15.38:0/2340086784 conn(0x55d51fba5750 msgr2=0x55d51fba5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- 172.21.15.38:0/2340086784 shutdown_connections 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- 172.21.15.38:0/2340086784 wait complete. 2024-04-22T00:11:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.224+0000 7f82705a5740 1 -- start start 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 Processor -- start 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbb3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d51fbb7310 con 0x55d51fbab940 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d51facaf10 con 0x55d51fbb4520 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d51fbb7490 con 0x55d51fbab020 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82705a5740 10 client.0 _ll_register_callbacks cb 0x55d51f822be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59284/0 (socket says 172.21.15.38:59284) 2024-04-22T00:11:58.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 -- 172.21.15.38:0/2312841146 learned_addr learned my addr 172.21.15.38:0/2312841146 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82677fe640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 -- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 msgr2=0x55d51fbb3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbb3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 -- 172.21.15.38:0/2312841146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 msgr2=0x55d51fbb1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 -- 172.21.15.38:0/2312841146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d51fa9cc80 con 0x55d51fbab940 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f82667fc640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f825c004c10 tx=0x7f825c005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a08c44b432f274 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f8246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.225+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c016040 con 0x55d51fbab940 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f825c009c80 con 0x55d51fbab940 2024-04-22T00:11:58.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825c005500 con 0x55d51fbab940 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f825c009760 con 0x55d51fbab940 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 10 client.18738 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d51f8233f0 con 0x55d51fbab940 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c005500 con 0x55d51fbab940 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f825c00f040 con 0x55d51fbab940 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 10 client.18738 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 10 client.18738 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 20 client.18738 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fbc43b0 0x55d51fbc67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 4 client.18738 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d51fb1ca70 con 0x55d51fbc43b0 2024-04-22T00:11:58.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.226+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fbc43b0 0x55d51fbc67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.227+0000 7f8246ffd640 10 client.18738 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.227+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fbc43b0 0x55d51fbc67a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c21028e05d99c84c server_cookie=2dc581e94c958e7 in_seq=0 out_seq=0 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.227+0000 7f8246ffd640 1 -- 172.21.15.38:0/2312841146 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d51fb1ca70 con 0x55d51fbc43b0 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.227+0000 7f8246ffd640 10 client.18738 handle_command_reply: tid=0 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.227+0000 7f82705a5740 1 client.18738 shutdown 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 msgr2=0x55d51fbb11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f825c004c10 tx=0x7f825c005f60 comp rx=0 tx=0).stop 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 shutdown_connections 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fbc43b0 0x55d51fbc67a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb11f0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f8246ffd640 0 client.18738 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:58.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 --2- 172.21.15.38:0/2312841146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbb3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 >> 172.21.15.38:0/2312841146 conn(0x55d51fba5750 msgr2=0x55d51fba74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 shutdown_connections 2024-04-22T00:11:58.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 1 -- 172.21.15.38:0/2312841146 wait complete. 2024-04-22T00:11:58.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.228+0000 7f82705a5740 20 client.18738 trim_cache size 0 max 16384 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 Processor -- start 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 -- start start 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbc01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fba46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82681aec10 con 0x55d51fbab940 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82681b0780 con 0x55d51fbb4520 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.241+0000 7f82705a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82681af310 con 0x55d51fbab020 2024-04-22T00:11:58.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59286/0 (socket says 172.21.15.38:59286) 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 -- 172.21.15.38:0/3764982644 learned_addr learned my addr 172.21.15.38:0/3764982644 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f8266ffd640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbc01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82677fe640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fba46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 -- 172.21.15.38:0/3764982644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 msgr2=0x55d51fbc01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbc01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 -- 172.21.15.38:0/3764982644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 msgr2=0x55d51fba46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fba46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 -- 172.21.15.38:0/3764982644 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d51fbab770 con 0x55d51fbab940 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82677fe640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fba46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f82667fc640 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f825c003c50 tx=0x7f825c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b72c8fd8c0a2c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f824d7fa640 1 -- 172.21.15.38:0/3764982644 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c016070 con 0x55d51fbab940 2024-04-22T00:11:58.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f824d7fa640 1 -- 172.21.15.38:0/3764982644 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c0048d0 con 0x55d51fbab940 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.242+0000 7f824d7fa640 1 -- 172.21.15.38:0/3764982644 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c003850 con 0x55d51fbab940 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- 172.21.15.38:0/3764982644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 msgr2=0x55d51fbb0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f825c003c50 tx=0x7f825c00e6c0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- 172.21.15.38:0/3764982644 shutdown_connections 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fba46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb0350 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 --2- 172.21.15.38:0/3764982644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbc01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- 172.21.15.38:0/3764982644 >> 172.21.15.38:0/3764982644 conn(0x55d51fba5750 msgr2=0x55d51fba3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- 172.21.15.38:0/3764982644 shutdown_connections 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- 172.21.15.38:0/3764982644 wait complete. 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 -- start start 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.243+0000 7f82705a5740 1 Processor -- start 2024-04-22T00:11:58.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbbca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82681b00b0 con 0x55d51fbab940 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f82681af9e0 con 0x55d51fbb4520 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82681acc10 con 0x55d51fbab020 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 10 client.0 _ll_register_callbacks cb 0x7f8258098360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59288/0 (socket says 172.21.15.38:59288) 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 -- 172.21.15.38:0/2044492837 learned_addr learned my addr 172.21.15.38:0/2044492837 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82677fe640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbbca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 -- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 msgr2=0x55d51fbbca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbbca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 -- 172.21.15.38:0/2044492837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 msgr2=0x55d51fbb8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 -- 172.21.15.38:0/2044492837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d51f8ed7d0 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbbca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82677fe640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82667fc640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f825c0092e0 tx=0x7f825c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1667676163f52db server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c037070 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f825c0035f0 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825c0048d0 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f825c043050 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 10 client.18744 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.244+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d51fc2a3d0 con 0x55d51fbab940 2024-04-22T00:11:58.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c0048d0 con 0x55d51fbab940 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f825c031070 con 0x55d51fbab940 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 10 client.18744 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 10 client.18744 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 20 client.18744 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fc2aa60 0x55d51fc2ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 4 client.18744 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d51fb16dc0 con 0x55d51fc2aa60 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.245+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fc2aa60 0x55d51fc2ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f8246ffd640 10 client.18744 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f8266ffd640 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fc2aa60 0x55d51fc2ce80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27ebb654a8709e2c server_cookie=925813fd5fb2e16f in_seq=0 out_seq=0 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f8246ffd640 1 -- 172.21.15.38:0/2044492837 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d51fb16dc0 con 0x55d51fc2aa60 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f8246ffd640 10 client.18744 handle_command_reply: tid=0 2024-04-22T00:11:58.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 client.18744 shutdown 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 msgr2=0x55d51fbb7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f825c0092e0 tx=0x7f825c009b50 comp rx=0 tx=0).stop 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 shutdown_connections 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55d51fc2aa60 0x55d51fc2ce80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d51fbb4520 0x55d51fbb8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f8246ffd640 0 client.18744 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d51fbab940 0x55d51fbb7dc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 --2- 172.21.15.38:0/2044492837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d51fbab020 0x55d51fbbca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 >> 172.21.15.38:0/2044492837 conn(0x55d51fba5750 msgr2=0x55d51fba3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 shutdown_connections 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 1 -- 172.21.15.38:0/2044492837 wait complete. 2024-04-22T00:11:58.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.246+0000 7f82705a5740 20 client.18744 trim_cache size 0 max 16384 2024-04-22T00:11:58.251 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:58.251 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 msgr2=0x7f82500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 0x7f82500a51c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f826002efd0 tx=0x7f826002cf90 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 msgr2=0x7f8268197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f825800ebc0 tx=0x7f825800c6a0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 shutdown_connections 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f82500a2d00 0x7f82500a51c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f826810bd70 0x7f826819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82681032c0 0x7f82681984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 --2- 172.21.15.38:0/1269497414 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82681029a0 0x7f8268197fa0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 >> 172.21.15.38:0/1269497414 conn(0x7f82680769c0 msgr2=0x7f82680fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 shutdown_connections 2024-04-22T00:11:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.247+0000 7f826d8df640 1 -- 172.21.15.38:0/1269497414 wait complete. 2024-04-22T00:11:58.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 -- 172.21.15.38:0/1751258669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 msgr2=0x7fdd7c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1751258669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c1011f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009900 tx=0x7fdd6c02bcd0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 -- 172.21.15.38:0/1751258669 shutdown_connections 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1751258669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd7c101730 0x7fdd7c10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1751258669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c1011f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1751258669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c106660 0x7fdd7c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.362+0000 7fdd837a1640 1 -- 172.21.15.38:0/1751258669 >> 172.21.15.38:0/1751258669 conn(0x7fdd7c076c00 msgr2=0x7fdd7c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- 172.21.15.38:0/1751258669 shutdown_connections 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- 172.21.15.38:0/1751258669 wait complete. 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 Processor -- start 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- start start 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c101730 0x7fdd7c102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd7c106660 0x7fdd7c102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c110a30 con 0x7fdd7c100d70 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1108b0 con 0x7fdd7c106660 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd837a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c110bb0 con 0x7fdd7c101730 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd8153d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd8153d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59308/0 (socket says 172.21.15.38:59308) 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.363+0000 7fdd8153d640 1 -- 172.21.15.38:0/1609689541 learned_addr learned my addr 172.21.15.38:0/1609689541 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c101730 0x7fdd7c102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 -- 172.21.15.38:0/1609689541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c101730 msgr2=0x7fdd7c102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c101730 0x7fdd7c102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 -- 172.21.15.38:0/1609689541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd7c106660 msgr2=0x7fdd7c102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd7c106660 0x7fdd7c102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 -- 172.21.15.38:0/1609689541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7c1a6520 con 0x7fdd7c100d70 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd8153d640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800d770 tx=0x7fdd7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd727fc640 1 -- 172.21.15.38:0/1609689541 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78017070 con 0x7fdd7c100d70 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd7c1a6810 con 0x7fdd7c100d70 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd727fc640 1 -- 172.21.15.38:0/1609689541 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd78005500 con 0x7fdd7c100d70 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd7c1a6d60 con 0x7fdd7c100d70 2024-04-22T00:11:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.364+0000 7fdd727fc640 1 -- 172.21.15.38:0/1609689541 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78020060 con 0x7fdd7c100d70 2024-04-22T00:11:58.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.366+0000 7fdd727fc640 1 -- 172.21.15.38:0/1609689541 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdd78005020 con 0x7fdd7c100d70 2024-04-22T00:11:58.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 Processor -- start 2024-04-22T00:11:58.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 -- start start 2024-04-22T00:11:58.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e7e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e86fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d515de2530 con 0x55d515e87520 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d515d8a830 con 0x55d515e7e940 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8638f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d515dde4f0 con 0x55d515e7e020 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd727fc640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 0x7fdd680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59324/0 (socket says 172.21.15.38:59324) 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 -- 172.21.15.38:0/1303335625 learned_addr learned my addr 172.21.15.38:0/1303335625 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 0x7fdd680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd727fc640 1 -- 172.21.15.38:0/1609689541 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fdd780cf250 con 0x7fdd7c100d70 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd8153d640 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e86fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e7e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 -- 172.21.15.38:0/1303335625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 msgr2=0x55d515e7e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e7e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 -- 172.21.15.38:0/1303335625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 msgr2=0x55d515e86fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e86fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.367+0000 7fdd80d3c640 1 -- 172.21.15.38:0/1303335625 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d515dae2c0 con 0x55d515e87520 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c02c210 tx=0x7fdd6c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0568e81c426fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 0x7fdd680a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd74005fd0 tx=0x7fdd740094b0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd677fe640 1 -- 172.21.15.38:0/1303335625 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c004240 con 0x55d515e87520 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd677fe640 1 -- 172.21.15.38:0/1303335625 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd6c02ce30 con 0x55d515e87520 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd677fe640 1 -- 172.21.15.38:0/1303335625 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0048c0 con 0x55d515e87520 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd8638f740 1 -- 172.21.15.38:0/1303335625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 msgr2=0x55d515e89910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.368+0000 7fdd8638f740 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c02c210 tx=0x7fdd6c005e70 comp rx=0 tx=0).stop 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- 172.21.15.38:0/1303335625 shutdown_connections 2024-04-22T00:11:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e87520 0x55d515e89910 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e86fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- 172.21.15.38:0/1303335625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e7e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- 172.21.15.38:0/1303335625 >> 172.21.15.38:0/1303335625 conn(0x55d515e78750 msgr2=0x55d515e78b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- 172.21.15.38:0/1303335625 shutdown_connections 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- 172.21.15.38:0/1303335625 wait complete. 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- start start 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 Processor -- start 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e84230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 0x55d515e847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d515e8a2b0 con 0x55d515e7e020 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d515d9df10 con 0x55d515e7e940 2024-04-22T00:11:58.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.369+0000 7fdd8638f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d515e8a430 con 0x55d515e87520 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8638f740 10 client.0 _ll_register_callbacks cb 0x55d515af5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59332/0 (socket says 172.21.15.38:59332) 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 -- 172.21.15.38:0/168284108 learned_addr learned my addr 172.21.15.38:0/168284108 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e84230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 0x55d515e847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 -- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 msgr2=0x55d515e847b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 0x55d515e847b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 -- 172.21.15.38:0/168284108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 msgr2=0x55d515e84230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e84230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 -- 172.21.15.38:0/168284108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d515d6fc80 con 0x55d515e7e020 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 0x55d515e847b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8153d640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdd7809b080 tx=0x7fdd78004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe27c0134ef6ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780a3ff0 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fdd780979f0 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd6c02ce30 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdd78005310 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8638f740 10 client.18756 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.370+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d515af63f0 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd6c02ce30 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fdd7809a7d0 con 0x55d515e7e020 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 10 client.18756 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 10 client.18756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 20 client.18756 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515e97480 0x55d515e99910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 4 client.18756 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d515defa70 con 0x55d515e97480 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515e97480 0x55d515e99910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515e97480 0x55d515e99910 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe9890b22a290263 server_cookie=c48ea6eed0aba68c in_seq=0 out_seq=0 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.371+0000 7fdd64ff9640 10 client.18756 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd64ff9640 1 -- 172.21.15.38:0/168284108 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d515defa70 con 0x55d515e97480 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd64ff9640 10 client.18756 handle_command_reply: tid=0 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 client.18756 shutdown 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 msgr2=0x55d515e86ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fdd7809b080 tx=0x7fdd78004800 comp rx=0 tx=0).stop 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 shutdown_connections 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515e97480 0x55d515e99910 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e87520 0x55d515e847b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e7e940 0x55d515e84230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd64ff9640 0 client.18756 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:58.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 --2- 172.21.15.38:0/168284108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e020 0x55d515e86ea0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 >> 172.21.15.38:0/168284108 conn(0x55d515e78750 msgr2=0x55d515e7a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.372+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 shutdown_connections 2024-04-22T00:11:58.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.373+0000 7fdd8638f740 1 -- 172.21.15.38:0/168284108 wait complete. 2024-04-22T00:11:58.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.373+0000 7fdd8638f740 20 client.18756 trim_cache size 0 max 16384 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 Processor -- start 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 -- start start 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e93170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e86960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1b0da0 con 0x55d515e7e940 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1b2910 con 0x55d515e87520 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd8638f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1b14a0 con 0x55d515e7e020 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59334/0 (socket says 172.21.15.38:59334) 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.386+0000 7fdd80d3c640 1 -- 172.21.15.38:0/4010677321 learned_addr learned my addr 172.21.15.38:0/4010677321 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 -- 172.21.15.38:0/4010677321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 msgr2=0x55d515e93170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e93170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 -- 172.21.15.38:0/4010677321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 msgr2=0x55d515e86960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e86960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 -- 172.21.15.38:0/4010677321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d515bc07d0 con 0x55d515e7e940 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x55d515e798e0 tx=0x7fdd6c02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e4209a58a3379e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd66ffd640 1 -- 172.21.15.38:0/4010677321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c02c950 con 0x55d515e7e940 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd66ffd640 1 -- 172.21.15.38:0/4010677321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd6c053080 con 0x55d515e7e940 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd66ffd640 1 -- 172.21.15.38:0/4010677321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0346a0 con 0x55d515e7e940 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 -- 172.21.15.38:0/4010677321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 msgr2=0x55d515e7f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x55d515e798e0 tx=0x7fdd6c02bcd0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 -- 172.21.15.38:0/4010677321 shutdown_connections 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e86960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e7f810 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 --2- 172.21.15.38:0/4010677321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e93170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.387+0000 7fdd8638f740 1 -- 172.21.15.38:0/4010677321 >> 172.21.15.38:0/4010677321 conn(0x55d515e78750 msgr2=0x55d515e78b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- 172.21.15.38:0/4010677321 shutdown_connections 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- 172.21.15.38:0/4010677321 wait complete. 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- start start 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 Processor -- start 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e80ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e81950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1b2240 con 0x55d515e7e940 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1b1b70 con 0x55d515e87520 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7c1aeda0 con 0x55d515e7e020 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd8638f740 10 client.0 _ll_register_callbacks cb 0x7fdd78005780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd80d3c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59336/0 (socket says 172.21.15.38:59336) 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.388+0000 7fdd80d3c640 1 -- 172.21.15.38:0/2716207224 learned_addr learned my addr 172.21.15.38:0/2716207224 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd8153d640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e80ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd81d3e640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e81950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 -- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 msgr2=0x55d515e80ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e80ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 -- 172.21.15.38:0/2716207224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 msgr2=0x55d515e81950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e81950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 -- 172.21.15.38:0/2716207224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d515efca70 con 0x55d515e7e940 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd80d3c640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c02c580 tx=0x7fdd6c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cea750895aa3163 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c02c950 con 0x55d515e7e940 2024-04-22T00:11:58.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd6c053080 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fdd6c02caf0 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdd6c02ccf0 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd8638f740 10 client.18762 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d515efdd40 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.389+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd6c053080 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7fdd6c050070 con 0x55d515e7e940 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 10 client.18762 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 10 client.18762 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 20 client.18762 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515efe010 0x55d515f00400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 4 client.18762 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d515de9dc0 con 0x55d515efe010 2024-04-22T00:11:58.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8153d640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515efe010 0x55d515f00400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd64ff9640 10 client.18762 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.390+0000 7fdd8153d640 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515efe010 0x55d515f00400 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7434cb2f9b3dc9ac server_cookie=39f8a7327f6feadc in_seq=0 out_seq=0 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd64ff9640 1 -- 172.21.15.38:0/2716207224 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d515de9dc0 con 0x55d515efe010 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd64ff9640 10 client.18762 handle_command_reply: tid=0 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 client.18762 shutdown 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 msgr2=0x55d515e81410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c02c580 tx=0x7fdd6c0056d0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 shutdown_connections 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d515efe010 0x55d515f00400 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d515e87520 0x55d515e81950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d515e7e940 0x55d515e81410 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd64ff9640 0 client.18762 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:11:58.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 --2- 172.21.15.38:0/2716207224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d515e7e020 0x55d515e80ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 >> 172.21.15.38:0/2716207224 conn(0x55d515e78750 msgr2=0x55d515e77850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 shutdown_connections 2024-04-22T00:11:58.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 1 -- 172.21.15.38:0/2716207224 wait complete. 2024-04-22T00:11:58.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd8638f740 20 client.18762 trim_cache size 0 max 16384 2024-04-22T00:11:58.395 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 msgr2=0x7fdd680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 0x7fdd680a51f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdd74005fd0 tx=0x7fdd740094b0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 msgr2=0x7fdd7c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800d770 tx=0x7fdd7800dc30 comp rx=0 tx=0).stop 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 shutdown_connections 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdd680a2d30 0x7fdd680a51f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd7c106660 0x7fdd7c102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd7c101730 0x7fdd7c102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 --2- 172.21.15.38:0/1609689541 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd7c100d70 0x7fdd7c102210 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.391+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 >> 172.21.15.38:0/1609689541 conn(0x7fdd7c076c00 msgr2=0x7fdd7c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.392+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 shutdown_connections 2024-04-22T00:11:58.397 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.392+0000 7fdd837a1640 1 -- 172.21.15.38:0/1609689541 wait complete. 2024-04-22T00:11:58.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- 172.21.15.38:0/1430392416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c06a7c0 msgr2=0x7f5f6c10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1430392416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c10e5f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009cd0 tx=0x7f5f6402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- 172.21.15.38:0/1430392416 shutdown_connections 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1430392416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c10e5f0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1430392416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c069e00 0x7f5f6c06a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1430392416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c107730 0x7f5f6c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- 172.21.15.38:0/1430392416 >> 172.21.15.38:0/1430392416 conn(0x7f5f6c0776a0 msgr2=0x7f5f6c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- 172.21.15.38:0/1430392416 shutdown_connections 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- 172.21.15.38:0/1430392416 wait complete. 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 Processor -- start 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 -- start start 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.507+0000 7f5f72ced640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 0x7f5f6c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c111b40 con 0x7f5f6c069e00 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1119c0 con 0x7f5f6c107730 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c111cc0 con 0x7f5f6c06a7c0 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59342/0 (socket says 172.21.15.38:59342) 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 -- 172.21.15.38:0/1483986149 learned_addr learned my addr 172.21.15.38:0/1483986149 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f7128a640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 0x7f5f6c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 -- 172.21.15.38:0/1483986149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c06a7c0 msgr2=0x7f5f6c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 -- 172.21.15.38:0/1483986149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 msgr2=0x7f5f6c19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 0x7f5f6c19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 -- 172.21.15.38:0/1483986149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c19cf20 con 0x7f5f6c069e00 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f7128a640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 0x7f5f6c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00d950 tx=0x7f5f5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f69ffb640 1 -- 172.21.15.38:0/1483986149 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c017030 con 0x7f5f6c069e00 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c19d210 con 0x7f5f6c069e00 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f69ffb640 1 -- 172.21.15.38:0/1483986149 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c004020 con 0x7f5f6c069e00 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f69ffb640 1 -- 172.21.15.38:0/1483986149 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0047a0 con 0x7f5f6c069e00 2024-04-22T00:11:58.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.508+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c1a4aa0 con 0x7f5f6c069e00 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f69ffb640 1 -- 172.21.15.38:0/1483986149 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5f5c0041d0 con 0x7f5f6c069e00 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 Processor -- start 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 -- start start 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f69ffb640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 0x7f5f540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 0x557b53a59400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59940 0x557b53a61fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f7128a640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 0x7f5f540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557b539bd530 con 0x557b53a62520 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557b53965830 con 0x557b53a59020 2024-04-22T00:11:58.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f69ffb640 1 -- 172.21.15.38:0/1483986149 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f5f5c0cf210 con 0x7f5f6c069e00 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.511+0000 7f5f759b3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557b539b94f0 con 0x557b53a59940 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f7128a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f7128a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59358/0 (socket says 172.21.15.38:59358) 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f7128a640 1 -- 172.21.15.38:0/3364930500 learned_addr learned my addr 172.21.15.38:0/3364930500 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59940 0x557b53a61fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 0x557b53a59400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f7128a640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 0x7f5f540a51c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5f64036000 tx=0x7f5f64032d70 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 -- 172.21.15.38:0/3364930500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59940 msgr2=0x557b53a61fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59940 0x557b53a61fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 -- 172.21.15.38:0/3364930500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 msgr2=0x557b53a64910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 -- 172.21.15.38:0/3364930500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b539892c0 con 0x557b53a59020 2024-04-22T00:11:58.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f7128a640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 0x557b53a59400 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0cc5d0 tx=0x7f5f5c09afa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a9a9c5fceb6b0b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f52ffd640 1 -- 172.21.15.38:0/3364930500 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c09f750 con 0x557b53a59020 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f52ffd640 1 -- 172.21.15.38:0/3364930500 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c0a4010 con 0x557b53a59020 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.512+0000 7f5f52ffd640 1 -- 172.21.15.38:0/3364930500 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0a3de0 con 0x557b53a59020 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- 172.21.15.38:0/3364930500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 msgr2=0x557b53a59400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 0x557b53a59400 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0cc5d0 tx=0x7f5f5c09afa0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- 172.21.15.38:0/3364930500 shutdown_connections 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a64910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59940 0x557b53a61fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3364930500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59020 0x557b53a59400 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- 172.21.15.38:0/3364930500 >> 172.21.15.38:0/3364930500 conn(0x557b53a53750 msgr2=0x557b53a53b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- 172.21.15.38:0/3364930500 shutdown_connections 2024-04-22T00:11:58.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- 172.21.15.38:0/3364930500 wait complete. 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 -- start start 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.513+0000 7f5f759b3740 1 Processor -- start 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a5f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557b53a65310 con 0x557b53a59020 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x557b53978f10 con 0x557b53a59940 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557b53a65490 con 0x557b53a62520 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f759b3740 10 client.0 _ll_register_callbacks cb 0x557b536d0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59366/0 (socket says 172.21.15.38:59366) 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 -- 172.21.15.38:0/2898197316 learned_addr learned my addr 172.21.15.38:0/2898197316 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f7128a640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a5f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 -- 172.21.15.38:0/2898197316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 msgr2=0x557b53a5f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a5f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 -- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 msgr2=0x557b53a5f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 -- 172.21.15.38:0/2898197316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b5394ac80 con 0x557b53a59020 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f70a89640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0ccbd0 tx=0x7f5f5c09b010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00a9749ce7e61c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.514+0000 7f5f4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b3070 con 0x557b53a59020 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f5f5c0a3d90 con 0x557b53a59020 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f5c0a4010 con 0x557b53a59020 2024-04-22T00:11:58.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f5f5c097a10 con 0x557b53a59020 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 10 client.18774 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557b536d13f0 con 0x557b53a59020 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c0a4010 con 0x557b53a59020 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f5f5c09a260 con 0x557b53a59020 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 10 client.18774 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 10 client.18774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 20 client.18774 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53a72460 0x557b53a74850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 4 client.18774 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b539caa70 con 0x557b53a72460 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.515+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53a72460 0x557b53a74850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.516+0000 7f5f6bfff640 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53a72460 0x557b53a74850 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb69dc27f86f07c server_cookie=d8e4d7324c487b11 in_seq=0 out_seq=0 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.516+0000 7f5f4bfff640 10 client.18774 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:58.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f4bfff640 1 -- 172.21.15.38:0/2898197316 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b539caa70 con 0x557b53a72460 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f4bfff640 10 client.18774 handle_command_reply: tid=0 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 client.18774 shutdown 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 msgr2=0x557b53a61ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0ccbd0 tx=0x7f5f5c09b010 comp rx=0 tx=0).stop 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 shutdown_connections 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a5f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53a72460 0x557b53a74850 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 --2- 172.21.15.38:0/2898197316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a61ee0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f4bfff640 0 client.18774 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 >> 172.21.15.38:0/2898197316 conn(0x557b53a53750 msgr2=0x557b53a554a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 shutdown_connections 2024-04-22T00:11:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 1 -- 172.21.15.38:0/2898197316 wait complete. 2024-04-22T00:11:58.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.517+0000 7f5f759b3740 20 client.18774 trim_cache size 0 max 16384 2024-04-22T00:11:58.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 Processor -- start 2024-04-22T00:11:58.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 -- start start 2024-04-22T00:11:58.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a6aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1aec40 con 0x557b53a59020 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1b07b0 con 0x557b53a59940 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.530+0000 7f5f759b3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1af340 con 0x557b53a62520 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59370/0 (socket says 172.21.15.38:59370) 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 -- 172.21.15.38:0/1446752564 learned_addr learned my addr 172.21.15.38:0/1446752564 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f7128a640 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 -- 172.21.15.38:0/1446752564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 msgr2=0x557b53a619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 -- 172.21.15.38:0/1446752564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 msgr2=0x557b53a6aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a6aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 -- 172.21.15.38:0/1446752564 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b5379b7d0 con 0x557b53a59020 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f70a89640 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0cc680 tx=0x7f5f5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be60fba0a8d7a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f527fc640 1 -- 172.21.15.38:0/1446752564 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b3070 con 0x557b53a59020 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f527fc640 1 -- 172.21.15.38:0/1446752564 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c0a08e0 con 0x557b53a59020 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.531+0000 7f5f527fc640 1 -- 172.21.15.38:0/1446752564 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c09f750 con 0x557b53a59020 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- 172.21.15.38:0/1446752564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 msgr2=0x557b53a6e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0cc680 tx=0x7f5f5c009780 comp rx=0 tx=0).stop 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- 172.21.15.38:0/1446752564 shutdown_connections 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a62520 0x557b53a619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a6aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 --2- 172.21.15.38:0/1446752564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a59020 0x557b53a6e1d0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- 172.21.15.38:0/1446752564 >> 172.21.15.38:0/1446752564 conn(0x557b53a53750 msgr2=0x557b53a52850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- 172.21.15.38:0/1446752564 shutdown_connections 2024-04-22T00:11:58.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- 172.21.15.38:0/1446752564 wait complete. 2024-04-22T00:11:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 -- start start 2024-04-22T00:11:58.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.532+0000 7f5f759b3740 1 Processor -- start 2024-04-22T00:11:58.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59020 0x557b53a5b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.535 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1b00e0 con 0x557b53a62520 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1afa10 con 0x557b53a59940 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1acc40 con 0x557b53a59020 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f759b3740 10 client.0 _ll_register_callbacks cb 0x7f5f64037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59376/0 (socket says 172.21.15.38:59376) 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 -- 172.21.15.38:0/3225746161 learned_addr learned my addr 172.21.15.38:0/3225746161 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f6bfff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59020 0x557b53a5b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 -- 172.21.15.38:0/3225746161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59020 msgr2=0x557b53a5b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59020 0x557b53a5b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 -- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 msgr2=0x557b53a5ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 -- 172.21.15.38:0/3225746161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b53a5c850 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f7128a640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5f6402c630 tx=0x7f5f6402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea37df824bc189b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.533+0000 7f5f4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64053060 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f5f64004510 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f5c0a08e0 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f5f6405a050 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f759b3740 10 client.18780 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557b53adcdb0 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c0a08e0 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.535+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f5f64034440 con 0x557b53a62520 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 10 client.18780 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:58.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 10 client.18780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 20 client.18780 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53add530 0x557b53adf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 4 client.18780 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b539c4dc0 con 0x557b53add530 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53add530 0x557b53adf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f4bfff640 10 client.18780 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f70a89640 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53add530 0x557b53adf950 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a110ac621a81f5 server_cookie=2723158444276f99 in_seq=0 out_seq=0 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f4bfff640 1 -- 172.21.15.38:0/3225746161 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b539c4dc0 con 0x557b53add530 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f4bfff640 10 client.18780 handle_command_reply: tid=0 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 1 client.18780 shutdown 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 msgr2=0x557b53a5c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.536+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5f6402c630 tx=0x7f5f6402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 shutdown_connections 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x557b53add530 0x557b53adf950 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557b53a62520 0x557b53a5c020 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557b53a59940 0x557b53a5ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f4bfff640 0 client.18780 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 --2- 172.21.15.38:0/3225746161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557b53a59020 0x557b53a5b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 >> 172.21.15.38:0/3225746161 conn(0x557b53a53750 msgr2=0x557b53a54a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 shutdown_connections 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 1 -- 172.21.15.38:0/3225746161 wait complete. 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f759b3740 20 client.18780 trim_cache size 0 max 16384 2024-04-22T00:11:58.539 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:11:58.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 msgr2=0x7f5f540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 0x7f5f540a51c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5f64036000 tx=0x7f5f64032d70 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 msgr2=0x7f5f6c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00d950 tx=0x7f5f5c00de10 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 shutdown_connections 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5f540a2d00 0x7f5f540a51c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f6c107730 0x7f5f6c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f6c06a7c0 0x7f5f6c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 --2- 172.21.15.38:0/1483986149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f6c069e00 0x7f5f6c197f80 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.537+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 >> 172.21.15.38:0/1483986149 conn(0x7f5f6c0776a0 msgr2=0x7f5f6c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.538+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 shutdown_connections 2024-04-22T00:11:58.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:58.538+0000 7f5f72ced640 1 -- 172.21.15.38:0/1483986149 wait complete. 2024-04-22T00:11:58.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:58 smithi038 ceph-mon[27972]: pgmap v305: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:58.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:58 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:58.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:11:58 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2327881170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:58.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:58 smithi203 ceph-mon[31431]: pgmap v305: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:58.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:58 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:58.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:11:58 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2327881170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:58.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:58 smithi111 ceph-mon[31543]: pgmap v305: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-22T00:11:58.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:58 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:11:58.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:11:58 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2327881170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:11:59.546 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T00:11:59.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:11:59.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T00:11:59.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 -- 172.21.15.38:0/598202100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c069dc0 msgr2=0x7faa1c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 --2- 172.21.15.38:0/598202100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c069dc0 0x7faa1c06a240 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7faa04009930 tx=0x7faa0402bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:59.669 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 -- 172.21.15.38:0/598202100 shutdown_connections 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 --2- 172.21.15.38:0/598202100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 0x7faa1c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 --2- 172.21.15.38:0/598202100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c069dc0 0x7faa1c06a240 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 --2- 172.21.15.38:0/598202100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c107730 0x7faa1c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 -- 172.21.15.38:0/598202100 >> 172.21.15.38:0/598202100 conn(0x7faa1c0776a0 msgr2=0x7faa1c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 -- 172.21.15.38:0/598202100 shutdown_connections 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.669+0000 7faa21120640 1 -- 172.21.15.38:0/598202100 wait complete. 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.670+0000 7faa21120640 1 Processor -- start 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.670+0000 7faa21120640 1 -- start start 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.670+0000 7faa21120640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 0x7faa1c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.670+0000 7faa21120640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 0x7faa1c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c111b40 con 0x7faa1c107730 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1119c0 con 0x7faa1c06a780 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c111cc0 con 0x7faa1c069dc0 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59388/0 (socket says 172.21.15.38:59388) 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 -- 172.21.15.38:0/2044817322 learned_addr learned my addr 172.21.15.38:0/2044817322 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 0x7faa1c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa13fff640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 0x7faa1c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 -- 172.21.15.38:0/2044817322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 msgr2=0x7faa1c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 0x7faa1c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 -- 172.21.15.38:0/2044817322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 msgr2=0x7faa1c1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 0x7faa1c1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 -- 172.21.15.38:0/2044817322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c19cf90 con 0x7faa1c107730 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 0x7faa1c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa1b59e640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7faa1400eb10 tx=0x7faa1400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa18d99640 1 -- 172.21.15.38:0/2044817322 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14004b20 con 0x7faa1c107730 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa18d99640 1 -- 172.21.15.38:0/2044817322 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa14004cc0 con 0x7faa1c107730 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1c19d280 con 0x7faa1c107730 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa18d99640 1 -- 172.21.15.38:0/2044817322 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14010760 con 0x7faa1c107730 2024-04-22T00:11:59.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.671+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa1c1a4b10 con 0x7faa1c107730 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa18d99640 1 -- 172.21.15.38:0/2044817322 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7faa14005020 con 0x7faa1c107730 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 Processor -- start 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 -- start start 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb4720 0x559032bb3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559032b17170 con 0x559032bbcdb0 2024-04-22T00:11:59.679 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559032abf7c0 con 0x559032bb3a40 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.674+0000 7faa23ddf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559032b13130 con 0x559032bb4720 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59394/0 (socket says 172.21.15.38:59394) 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 -- 172.21.15.38:0/1419713453 learned_addr learned my addr 172.21.15.38:0/1419713453 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb4720 0x559032bb3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 -- 172.21.15.38:0/1419713453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb4720 msgr2=0x559032bb3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa18d99640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 0x7faa080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb4720 0x559032bb3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 -- 172.21.15.38:0/1419713453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 msgr2=0x559032bb3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 -- 172.21.15.38:0/1419713453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559032ad0e60 con 0x559032bbcdb0 2024-04-22T00:11:59.680 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa1b59e640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 0x7faa080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa13fff640 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7faa04005940 tx=0x7faa040058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cefe6dee0a8f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.675+0000 7faa18d99640 1 -- 172.21.15.38:0/2044817322 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7faa140cfec0 con 0x7faa1c107730 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa117fa640 1 -- 172.21.15.38:0/1419713453 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04049070 con 0x559032bbcdb0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa117fa640 1 -- 172.21.15.38:0/1419713453 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0402cc50 con 0x559032bbcdb0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa117fa640 1 -- 172.21.15.38:0/1419713453 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04003c60 con 0x559032bbcdb0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 -- 172.21.15.38:0/1419713453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 msgr2=0x559032bbf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa1b59e640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 0x7faa080a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa1409bd50 tx=0x7faa1408f040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7faa04005940 tx=0x7faa040058b0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 -- 172.21.15.38:0/1419713453 shutdown_connections 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bbf1a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1419713453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb4720 0x559032bb3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.676+0000 7faa23ddf740 1 -- 172.21.15.38:0/1419713453 >> 172.21.15.38:0/1419713453 conn(0x559032b7a4f0 msgr2=0x559032bae490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- 172.21.15.38:0/1419713453 shutdown_connections 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- 172.21.15.38:0/1419713453 wait complete. 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- start start 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 Processor -- start 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bba300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bba840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559032bbfbc0 con 0x559032bb3a40 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x559032ad2d40 con 0x559032bb4720 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559032bbfd40 con 0x559032bbcdb0 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa23ddf740 10 client.0 _ll_register_callbacks cb 0x559032b0e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59404/0 (socket says 172.21.15.38:59404) 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.677+0000 7faa13fff640 1 -- 172.21.15.38:0/1218648692 learned_addr learned my addr 172.21.15.38:0/1218648692 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bba300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 -- 172.21.15.38:0/1218648692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 msgr2=0x559032bba840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bba840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 -- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 msgr2=0x559032bba300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bba300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 -- 172.21.15.38:0/1218648692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590328f37e0 con 0x559032bb3a40 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa13fff640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7faa04047040 tx=0x7faa0402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ce213e60994df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04049040 con 0x559032bb3a40 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7faa04043680 con 0x559032bb3a40 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0402cc50 con 0x559032bb3a40 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7faa04059050 con 0x559032bb3a40 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa23ddf740 10 client.18792 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:59.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559032953f70 con 0x559032bb3a40 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0402cc50 con 0x559032bb3a40 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7faa0404da50 con 0x559032bb3a40 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa23ddf740 10 client.18792 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.678+0000 7faa23ddf740 10 client.18792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa23ddf740 20 client.18792 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032bccd20 0x559032bcf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa23ddf740 4 client.18792 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559032aeb450 con 0x559032bccd20 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032bccd20 0x559032bcf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032bccd20 0x559032bcf110 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4bce18cc607d4e0 server_cookie=9ed812de7c1fbe4b in_seq=0 out_seq=0 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.679+0000 7faa027fc640 10 client.18792 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa027fc640 1 -- 172.21.15.38:0/1218648692 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559032aeb450 con 0x559032bccd20 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa027fc640 10 client.18792 handle_command_reply: tid=0 2024-04-22T00:11:59.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 client.18792 shutdown 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 msgr2=0x559032bb9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7faa04047040 tx=0x7faa0402bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 shutdown_connections 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bba840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bba300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032bccd20 0x559032bcf110 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 --2- 172.21.15.38:0/1218648692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb3a40 0x559032bb9dc0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa027fc640 0 client.18792 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 >> 172.21.15.38:0/1218648692 conn(0x559032b7a4f0 msgr2=0x559032bb01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 shutdown_connections 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 1 -- 172.21.15.38:0/1218648692 wait complete. 2024-04-22T00:11:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.680+0000 7faa23ddf740 20 client.18792 trim_cache size 0 max 16384 2024-04-22T00:11:59.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.698+0000 7faa23ddf740 1 Processor -- start 2024-04-22T00:11:59.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.698+0000 7faa23ddf740 1 -- start start 2024-04-22T00:11:59.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb3a40 0x559032bbc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bd2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1aea80 con 0x559032bbcdb0 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b05f0 con 0x559032bb4720 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa23ddf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1af180 con 0x559032bb3a40 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59412/0 (socket says 172.21.15.38:59412) 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 -- 172.21.15.38:0/4126038250 learned_addr learned my addr 172.21.15.38:0/4126038250 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bd2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 -- 172.21.15.38:0/4126038250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb3a40 msgr2=0x559032bbc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb3a40 0x559032bbc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 -- 172.21.15.38:0/4126038250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 msgr2=0x559032bd2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bd2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 -- 172.21.15.38:0/4126038250 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590327eb170 con 0x559032bbcdb0 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.699+0000 7faa13fff640 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7faa04005ab0 tx=0x7faa04005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4d5581d51c9e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa10ff9640 1 -- 172.21.15.38:0/4126038250 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04062070 con 0x559032bbcdb0 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa10ff9640 1 -- 172.21.15.38:0/4126038250 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0404d440 con 0x559032bbcdb0 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa10ff9640 1 -- 172.21.15.38:0/4126038250 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04049040 con 0x559032bbcdb0 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 -- 172.21.15.38:0/4126038250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 msgr2=0x559032bc8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7faa04005ab0 tx=0x7faa04005d50 comp rx=0 tx=0).stop 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 -- 172.21.15.38:0/4126038250 shutdown_connections 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bbcdb0 0x559032bc8710 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb4720 0x559032bd2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 --2- 172.21.15.38:0/4126038250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bb3a40 0x559032bbc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 -- 172.21.15.38:0/4126038250 >> 172.21.15.38:0/4126038250 conn(0x559032b7a4f0 msgr2=0x559032bc8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 -- 172.21.15.38:0/4126038250 shutdown_connections 2024-04-22T00:11:59.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.700+0000 7faa23ddf740 1 -- 172.21.15.38:0/4126038250 wait complete. 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 -- start start 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 Processor -- start 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bb6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1aff20 con 0x559032bb4720 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1af850 con 0x559032bb3a40 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1aca80 con 0x559032bbcdb0 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa23ddf740 10 client.0 _ll_register_callbacks cb 0x7faa04047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa1ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa1ad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59414/0 (socket says 172.21.15.38:59414) 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa1ad9d640 1 -- 172.21.15.38:0/409301593 learned_addr learned my addr 172.21.15.38:0/409301593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa13fff640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bb6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.701+0000 7faa13fff640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 -- 172.21.15.38:0/409301593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 msgr2=0x559032bb6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bb6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 -- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 msgr2=0x559032bb7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 -- 172.21.15.38:0/409301593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559032bb68f0 con 0x559032bb4720 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa1ad9d640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x559032bb9620 tx=0x7faa0c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf281f344993b3f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c01d340 con 0x559032bb4720 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7faa0c01d4e0 con 0x559032bb4720 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559032b246b0 con 0x559032bb4720 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7faa0c02c050 con 0x559032bb4720 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 10 client.18798 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:59.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559032c37b30 con 0x559032bb4720 2024-04-22T00:11:59.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559032b246b0 con 0x559032bb4720 2024-04-22T00:11:59.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7faa0c011440 con 0x559032bb4720 2024-04-22T00:11:59.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 10 client.18798 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 10 client.18798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 20 client.18798 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032c38250 0x559032c3a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 4 client.18798 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.702+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559032b1ea00 con 0x559032c38250 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.703+0000 7faa13fff640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032c38250 0x559032c3a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.703+0000 7faa027fc640 10 client.18798 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.703+0000 7faa13fff640 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032c38250 0x559032c3a640 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8806359dff3db0be server_cookie=191ba13d657f6faf in_seq=0 out_seq=0 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa027fc640 1 -- 172.21.15.38:0/409301593 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559032b1ea00 con 0x559032c38250 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa027fc640 10 client.18798 handle_command_reply: tid=0 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 client.18798 shutdown 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 msgr2=0x559032bb79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x559032bb9620 tx=0x7faa0c004410 comp rx=0 tx=0).stop 2024-04-22T00:11:59.710 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 shutdown_connections 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559032bbcdb0 0x559032bb6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x559032c38250 0x559032c3a640 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559032bb4720 0x559032bb79d0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 --2- 172.21.15.38:0/409301593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559032bb3a40 0x559032bb7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa027fc640 0 client.18798 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 >> 172.21.15.38:0/409301593 conn(0x559032b7a4f0 msgr2=0x559032bba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 shutdown_connections 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.704+0000 7faa23ddf740 1 -- 172.21.15.38:0/409301593 wait complete. 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa23ddf740 20 client.18798 trim_cache size 0 max 16384 2024-04-22T00:11:59.711 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2024-04-22T00:11:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "b0ef10ea-1e8d-4bb6-afba-b7887a1c349c", 2024-04-22T00:11:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2024-04-22T00:11:59.712 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:59.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 msgr2=0x7faa080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 0x7faa080a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faa1409bd50 tx=0x7faa1408f040 comp rx=0 tx=0).stop 2024-04-22T00:11:59.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 msgr2=0x7faa1c19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7faa1400eb10 tx=0x7faa1400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 shutdown_connections 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7faa080a2d00 0x7faa080a51c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa1c107730 0x7faa1c19c860 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c06a780 0x7faa1c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 --2- 172.21.15.38:0/2044817322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa1c069dc0 0x7faa1c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 >> 172.21.15.38:0/2044817322 conn(0x7faa1c0776a0 msgr2=0x7faa1c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 shutdown_connections 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.705+0000 7faa21120640 1 -- 172.21.15.38:0/2044817322 wait complete. 2024-04-22T00:11:59.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:11:59.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:11:59.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- 172.21.15.38:0/2576110394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 msgr2=0x7ff80c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 --2- 172.21.15.38:0/2576110394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c106a10 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009930 tx=0x7ff7fc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:11:59.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- 172.21.15.38:0/2576110394 shutdown_connections 2024-04-22T00:11:59.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 --2- 172.21.15.38:0/2576110394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100fe0 0x7ff80c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 --2- 172.21.15.38:0/2576110394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100620 0x7ff80c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 --2- 172.21.15.38:0/2576110394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c106a10 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- 172.21.15.38:0/2576110394 >> 172.21.15.38:0/2576110394 conn(0x7ff80c0776a0 msgr2=0x7ff80c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- 172.21.15.38:0/2576110394 shutdown_connections 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- 172.21.15.38:0/2576110394 wait complete. 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 Processor -- start 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.822+0000 7ff81164e640 1 -- start start 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100620 0x7ff80c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100fe0 0x7ff80c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c110a40 con 0x7ff80c106630 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1108c0 con 0x7ff80c100fe0 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c110bc0 con 0x7ff80c100620 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59426/0 (socket says 172.21.15.38:59426) 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 -- 172.21.15.38:0/4201950008 learned_addr learned my addr 172.21.15.38:0/4201950008 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100fe0 0x7ff80c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80affd640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100620 0x7ff80c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.824 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 -- 172.21.15.38:0/4201950008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100620 msgr2=0x7ff80c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100620 0x7ff80c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 -- 172.21.15.38:0/4201950008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100fe0 msgr2=0x7ff80c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100fe0 0x7ff80c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 -- 172.21.15.38:0/4201950008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c19d0e0 con 0x7ff80c106630 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff80b7fe640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff80400eb10 tx=0x7ff80400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff7f3fff640 1 -- 172.21.15.38:0/4201950008 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804004b20 con 0x7ff80c106630 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff7f3fff640 1 -- 172.21.15.38:0/4201950008 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff804004cc0 con 0x7ff80c106630 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff7f3fff640 1 -- 172.21.15.38:0/4201950008 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804010760 con 0x7ff80c106630 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.823+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c19d3d0 con 0x7ff80c106630 2024-04-22T00:11:59.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.824+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1a4c60 con 0x7ff80c106630 2024-04-22T00:11:59.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff7f3fff640 1 -- 172.21.15.38:0/4201950008 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff804005020 con 0x7ff80c106630 2024-04-22T00:11:59.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 Processor -- start 2024-04-22T00:11:59.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 -- start start 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 0x55d2ad633ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad636820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad58f560 con 0x55d2ad62af30 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad537730 con 0x55d2ad62b850 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff814239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad58b520 con 0x55d2ad634430 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff80a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff80a7fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59442/0 (socket says 172.21.15.38:59442) 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff80a7fc640 1 -- 172.21.15.38:0/3139628353 learned_addr learned my addr 172.21.15.38:0/3139628353 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff7f3fff640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 0x7ff7f40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.826+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad636820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80affd640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 0x55d2ad633ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80b7fe640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 0x7ff7f40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 -- 172.21.15.38:0/3139628353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 msgr2=0x55d2ad636820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad636820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 -- 172.21.15.38:0/3139628353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 msgr2=0x55d2ad633ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 0x55d2ad633ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 -- 172.21.15.38:0/3139628353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ad36d7d0 con 0x55d2ad62af30 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80affd640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 0x55d2ad633ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff7f3fff640 1 -- 172.21.15.38:0/4201950008 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff8040cfe60 con 0x7ff80c106630 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff8000135a0 tx=0x7ff800009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1486d1776c2909c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff7f17fa640 1 -- 172.21.15.38:0/3139628353 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800004050 con 0x55d2ad62af30 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff7f17fa640 1 -- 172.21.15.38:0/3139628353 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8000041f0 con 0x55d2ad62af30 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff80b7fe640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 0x7ff7f40a5240 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff804098450 tx=0x7ff80406f060 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.827+0000 7ff7f17fa640 1 -- 172.21.15.38:0/3139628353 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80000f7e0 con 0x55d2ad62af30 2024-04-22T00:11:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- 172.21.15.38:0/3139628353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 msgr2=0x55d2ad62b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff8000135a0 tx=0x7ff800009850 comp rx=0 tx=0).stop 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- 172.21.15.38:0/3139628353 shutdown_connections 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad636820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62b850 0x55d2ad633ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- 172.21.15.38:0/3139628353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62af30 0x55d2ad62b310 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- 172.21.15.38:0/3139628353 >> 172.21.15.38:0/3139628353 conn(0x55d2ad625660 msgr2=0x55d2ad625a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- 172.21.15.38:0/3139628353 shutdown_connections 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- 172.21.15.38:0/3139628353 wait complete. 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- start start 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 Processor -- start 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad6314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad631f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad6371c0 con 0x55d2ad62b850 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad54ae00 con 0x55d2ad62af30 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d2ad637340 con 0x55d2ad634430 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff814239740 10 client.0 _ll_register_callbacks cb 0x55d2ad585eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59454/0 (socket says 172.21.15.38:59454) 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.828+0000 7ff80affd640 1 -- 172.21.15.38:0/3618882095 learned_addr learned my addr 172.21.15.38:0/3618882095 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad6314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad631f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 -- 172.21.15.38:0/3618882095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 msgr2=0x55d2ad631f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad631f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 -- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 msgr2=0x55d2ad6314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad6314a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 -- 172.21.15.38:0/3618882095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ad2a33f0 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff80affd640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc036040 tx=0x7ff7fc009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57c896c1055bac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff7eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc004450 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff7fc005030 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8000041f0 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff7fc04d050 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff814239740 10 client.18810 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d2ad2630f0 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.829+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8000041f0 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7ff7fc0035f0 con 0x55d2ad62b850 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 10 client.18810 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 10 client.18810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 20 client.18810 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:59.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad644710 0x55d2ad646b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 4 client.18810 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2ad59caa0 con 0x55d2ad644710 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad644710 0x55d2ad646b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad644710 0x55d2ad646b40 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19685c98bf8d6af1 server_cookie=6f0c15e148b271d6 in_seq=0 out_seq=0 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.830+0000 7ff7eeff5640 10 client.18810 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff7eeff5640 1 -- 172.21.15.38:0/3618882095 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2ad59caa0 con 0x55d2ad644710 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff7eeff5640 10 client.18810 handle_command_reply: tid=0 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 client.18810 shutdown 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 msgr2=0x55d2ad6319e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc036040 tx=0x7ff7fc009600 comp rx=0 tx=0).stop 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 shutdown_connections 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad631f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad644710 0x55d2ad646b40 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6319e0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 --2- 172.21.15.38:0/3618882095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad6314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff7eeff5640 0 client.18810 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 >> 172.21.15.38:0/3618882095 conn(0x55d2ad625660 msgr2=0x55d2ad6273b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 shutdown_connections 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 1 -- 172.21.15.38:0/3618882095 wait complete. 2024-04-22T00:11:59.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.831+0000 7ff814239740 20 client.18810 trim_cache size 0 max 16384 2024-04-22T00:11:59.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 Processor -- start 2024-04-22T00:11:59.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 -- start start 2024-04-22T00:11:59.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad633bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad64c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1aeb90 con 0x55d2ad62b850 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1b0700 con 0x55d2ad62af30 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff814239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1af290 con 0x55d2ad634430 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59462/0 (socket says 172.21.15.38:59462) 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 -- 172.21.15.38:0/2911752798 learned_addr learned my addr 172.21.15.38:0/2911752798 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad633bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 -- 172.21.15.38:0/2911752798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 msgr2=0x55d2ad64c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad64c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 -- 172.21.15.38:0/2911752798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 msgr2=0x55d2ad633bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad633bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 -- 172.21.15.38:0/2911752798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ad620fa0 con 0x55d2ad62b850 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad633bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff80affd640 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0039c0 tx=0x7ff7fc009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b72b2c3a0b48b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.849+0000 7ff7f0ff9640 1 -- 172.21.15.38:0/2911752798 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0035f0 con 0x55d2ad62b850 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff7f0ff9640 1 -- 172.21.15.38:0/2911752798 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc003790 con 0x55d2ad62b850 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff7f0ff9640 1 -- 172.21.15.38:0/2911752798 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc004eb0 con 0x55d2ad62b850 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- 172.21.15.38:0/2911752798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 msgr2=0x55d2ad640080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0039c0 tx=0x7ff7fc009230 comp rx=0 tx=0).stop 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- 172.21.15.38:0/2911752798 shutdown_connections 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad634430 0x55d2ad64c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad640080 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 --2- 172.21.15.38:0/2911752798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad62af30 0x55d2ad633bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- 172.21.15.38:0/2911752798 >> 172.21.15.38:0/2911752798 conn(0x55d2ad625660 msgr2=0x55d2ad6246e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- 172.21.15.38:0/2911752798 shutdown_connections 2024-04-22T00:11:59.853 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- 172.21.15.38:0/2911752798 wait complete. 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 -- start start 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.850+0000 7ff814239740 1 Processor -- start 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 0x55d2ad6a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad634430 0x55d2ad638710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1b0030 con 0x55d2ad62b850 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1af960 con 0x55d2ad634430 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1acb90 con 0x55d2ad62af30 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff814239740 10 client.0 _ll_register_callbacks cb 0x7ff800005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59468/0 (socket says 172.21.15.38:59468) 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 -- 172.21.15.38:0/134557499 learned_addr learned my addr 172.21.15.38:0/134557499 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad634430 0x55d2ad638710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 0x55d2ad6a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 -- 172.21.15.38:0/134557499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 msgr2=0x55d2ad6a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 0x55d2ad6a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 -- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad634430 msgr2=0x55d2ad638710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad634430 0x55d2ad638710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 -- 172.21.15.38:0/134557499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ad638dd0 con 0x55d2ad62b850 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 0x55d2ad6a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff80affd640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc003a40 tx=0x7ff7fc005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb567e43c9a0cd8b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff7eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.851+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0046c0 con 0x55d2ad62b850 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff7fc004860 con 0x55d2ad62b850 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7fc003790 con 0x55d2ad62b850 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff7fc060050 con 0x55d2ad62b850 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 10 client.18816 fetch_fsmap learned FSMap version 41 2024-04-22T00:11:59.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55d2ad6ab6d0 con 0x55d2ad62b850 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc003790 con 0x55d2ad62b850 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7ff7fc047660 con 0x55d2ad62b850 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 10 client.18816 fetch_fsmap finished waiting for FSMap version 41 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 10 client.18816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 20 client.18816 populate_metadata read hostname 'smithi038' 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad6abc80 0x55d2ad6ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 4 client.18816 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d2ad596df0 con 0x55d2ad6abc80 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.852+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad6abc80 0x55d2ad6ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff80a7fc640 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad6abc80 0x55d2ad6ae0a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=738c3f7c2b088c9c server_cookie=b9d827e5d4e9064e in_seq=0 out_seq=0 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff7eeff5640 10 client.18816 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff7eeff5640 1 -- 172.21.15.38:0/134557499 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d2ad596df0 con 0x55d2ad6abc80 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff7eeff5640 10 client.18816 handle_command_reply: tid=0 2024-04-22T00:11:59.856 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 client.18816 shutdown 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 msgr2=0x55d2ad6a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc003a40 tx=0x7ff7fc005c90 comp rx=0 tx=0).stop 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 shutdown_connections 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d2ad634430 0x55d2ad638710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x55d2ad6abc80 0x55d2ad6ae0a0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2ad62b850 0x55d2ad6a44d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 --2- 172.21.15.38:0/134557499 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2ad62af30 0x55d2ad6a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff7eeff5640 0 client.18816 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 >> 172.21.15.38:0/134557499 conn(0x55d2ad625660 msgr2=0x55d2ad631440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 shutdown_connections 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 1 -- 172.21.15.38:0/134557499 wait complete. 2024-04-22T00:11:59.857 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.853+0000 7ff814239740 20 client.18816 trim_cache size 0 max 16384 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "b0ef10ea-1e8d-4bb6-afba-b7887a1c349c": { 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "b0ef10ea-1e8d-4bb6-afba-b7887a1c349c", 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:11:59.858 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 msgr2=0x7ff7f40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 0x7ff7f40a5240 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff804098450 tx=0x7ff80406f060 comp rx=0 tx=0).stop 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 msgr2=0x7ff80c19c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff80400eb10 tx=0x7ff80400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 shutdown_connections 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff7f40a2d80 0x7ff7f40a5240 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff80c106630 0x7ff80c19c9b0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80c100fe0 0x7ff80c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 --2- 172.21.15.38:0/4201950008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c100620 0x7ff80c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:11:59.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 >> 172.21.15.38:0/4201950008 conn(0x7ff80c0776a0 msgr2=0x7ff80c06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:11:59.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 shutdown_connections 2024-04-22T00:11:59.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:11:59.854+0000 7ff81164e640 1 -- 172.21.15.38:0/4201950008 wait complete. 2024-04-22T00:11:59.862 INFO:tasks.cephfs.filesystem:scrub status for tag:b0ef10ea-1e8d-4bb6-afba-b7887a1c349c - {'path': '/', 'tag': 'b0ef10ea-1e8d-4bb6-afba-b7887a1c349c', 'options': 'recursive,force'} 2024-04-22T00:12:00.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:00 smithi111 ceph-mon[31543]: pgmap v306: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:00.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:00 smithi111 ceph-mon[31543]: scrub queued for path: / 2024-04-22T00:12:00.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:00 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:00.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:00 smithi111 ceph-mon[31543]: scrub summary: active paths [/] 2024-04-22T00:12:00.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:00 smithi111 ceph-mon[31543]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:00.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:00 smithi038 ceph-mon[27972]: pgmap v306: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:00.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:00 smithi038 ceph-mon[27972]: scrub queued for path: / 2024-04-22T00:12:00.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:00 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:00.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:00 smithi038 ceph-mon[27972]: scrub summary: active paths [/] 2024-04-22T00:12:00.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:00 smithi038 ceph-mon[27972]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:01.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:00 smithi203 ceph-mon[31431]: pgmap v306: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:01.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:00 smithi203 ceph-mon[31431]: scrub queued for path: / 2024-04-22T00:12:01.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:00 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:01.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:00 smithi203 ceph-mon[31431]: scrub summary: active paths [/] 2024-04-22T00:12:01.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:00 smithi203 ceph-mon[31431]: scrub summary: idle+waiting paths [/] 2024-04-22T00:12:01.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:01 smithi111 ceph-mon[31543]: scrub summary: idle 2024-04-22T00:12:01.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:01 smithi038 ceph-mon[27972]: scrub summary: idle 2024-04-22T00:12:02.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:01 smithi203 ceph-mon[31431]: scrub summary: idle 2024-04-22T00:12:02.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:02 smithi111 ceph-mon[31543]: pgmap v307: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:12:02.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:02 smithi038 ceph-mon[27972]: pgmap v307: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:12:03.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:02 smithi203 ceph-mon[31431]: pgmap v307: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T00:12:04.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:04 smithi111 ceph-mon[31543]: pgmap v308: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T00:12:04.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:04 smithi038 ceph-mon[27972]: pgmap v308: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T00:12:05.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:04 smithi203 ceph-mon[31431]: pgmap v308: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-22T00:12:05.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:05 smithi111 ceph-mon[31543]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:12:05.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:05 smithi038 ceph-mon[27972]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:12:06.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:05 smithi203 ceph-mon[31431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T00:12:06.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:06 smithi111 ceph-mon[31543]: pgmap v309: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:12:06.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:06 smithi038 ceph-mon[27972]: pgmap v309: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:12:07.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:06 smithi203 ceph-mon[31431]: pgmap v309: 129 pgs: 129 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-04-22T00:12:07.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:07 smithi038 ceph-mon[27972]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T00:12:08.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:07 smithi203 ceph-mon[31431]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T00:12:08.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:07 smithi111 ceph-mon[31543]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T00:12:08.935 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:08 smithi203 ceph-mon[31431]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:12:08.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:08 smithi038 ceph-mon[27972]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:12:09.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:08 smithi111 ceph-mon[31543]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:12:10.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:10 smithi038 ceph-mon[27972]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 60 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:12:11.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:10 smithi203 ceph-mon[31431]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 60 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:12:11.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:10 smithi111 ceph-mon[31543]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 60 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-04-22T00:12:11.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:11 smithi038 ceph-mon[27972]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 64 active+clean+snaptrim_wait, 46 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-22T00:12:12.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:11 smithi203 ceph-mon[31431]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 64 active+clean+snaptrim_wait, 46 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-22T00:12:12.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:11 smithi111 ceph-mon[31543]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 64 active+clean+snaptrim_wait, 46 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-22T00:12:14.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:14 smithi038 ceph-mon[27972]: pgmap v314: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:12:14.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:14 smithi203 ceph-mon[31431]: pgmap v314: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:12:14.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:14 smithi111 ceph-mon[31543]: pgmap v314: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 14 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:12:14.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.916+0000 7f99cd268640 1 -- 172.21.15.38:0/3134378582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c00a6a30 msgr2=0x7f99c00a8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.916+0000 7f99c5ffb640 1 -- 172.21.15.38:0/3134378582 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c80a0020 con 0x7f99c00a6a30 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.916+0000 7f99cd268640 1 --2- 172.21.15.38:0/3134378582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c00a6a30 0x7f99c00a8e20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99c8066eb0 tx=0x7f99c8097950 comp rx=0 tx=0).stop 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.919+0000 7f99cd268640 1 -- 172.21.15.38:0/3134378582 shutdown_connections 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.919+0000 7f99cd268640 1 --2- 172.21.15.38:0/3134378582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c00a6a30 0x7f99c00a8e20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:14.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.919+0000 7f99cd268640 1 --2- 172.21.15.38:0/3134378582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 0x7f99c0094bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:14.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.919+0000 7f99cd268640 1 --2- 172.21.15.38:0/3134378582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c009bd50 0x7f99c00941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:14.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.919+0000 7f99cd268640 1 -- 172.21.15.38:0/3134378582 >> 172.21.15.38:0/3134378582 conn(0x7f99c0009ab0 msgr2=0x7f99c0009ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:14.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.920+0000 7f99cd268640 1 -- 172.21.15.38:0/3134378582 shutdown_connections 2024-04-22T00:12:14.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.921+0000 7f99cd268640 1 -- 172.21.15.38:0/3134378582 wait complete. 2024-04-22T00:12:14.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.922+0000 7f99cd268640 1 Processor -- start 2024-04-22T00:12:14.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 -- start start 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 0x7f99c0138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c009bd50 0x7f99c0138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00ac740 con 0x7f99c00a6a30 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00ac5c0 con 0x7f99c0094790 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99cd268640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00ac8c0 con 0x7f99c009bd50 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47100/0 (socket says 172.21.15.38:47100) 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99c77fe640 1 -- 172.21.15.38:0/2298833655 learned_addr learned my addr 172.21.15.38:0/2298833655 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.924+0000 7f99c6ffd640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 0x7f99c0138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.925+0000 7f99c77fe640 1 -- 172.21.15.38:0/2298833655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c009bd50 msgr2=0x7f99c0138dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.925+0000 7f99c77fe640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c009bd50 0x7f99c0138dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.925+0000 7f99c77fe640 1 -- 172.21.15.38:0/2298833655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 msgr2=0x7f99c0138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.925+0000 7f99c77fe640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 0x7f99c0138890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.925+0000 7f99c77fe640 1 -- 172.21.15.38:0/2298833655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c0133190 con 0x7f99c00a6a30 2024-04-22T00:12:14.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.926+0000 7f99c77fe640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f99c8064f90 tx=0x7f99c809b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:14.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.929+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c80b3070 con 0x7f99c00a6a30 2024-04-22T00:12:14.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.929+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c8062040 con 0x7f99c00a6a30 2024-04-22T00:12:14.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.929+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c80a0070 con 0x7f99c00a6a30 2024-04-22T00:12:14.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.932+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c0133420 con 0x7f99c00a6a30 2024-04-22T00:12:14.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.932+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c009c180 con 0x7f99c00a6a30 2024-04-22T00:12:14.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.933+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c00953e0 con 0x7f99c00a6a30 2024-04-22T00:12:14.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.939+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f99c80a3020 con 0x7f99c00a6a30 2024-04-22T00:12:14.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.940+0000 7f99bffff640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 0x7f99b40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:14.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.940+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9526+0+0 (secure 0 0 0) 0x7f99c815c7f0 con 0x7f99c00a6a30 2024-04-22T00:12:14.942 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.945+0000 7f99c6ffd640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 0x7f99b40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:14.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.947+0000 7f99c6ffd640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 0x7f99b40a53c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005fd0 tx=0x7f99b0005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:14.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:14.953+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f99c80e96f0 con 0x7f99c00a6a30 2024-04-22T00:12:15.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.324+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f99c00a5f90 con 0x7f99b40a2f00 2024-04-22T00:12:15.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.333+0000 7f99bffff640 1 -- 172.21.15.38:0/2298833655 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f99c00a5f90 con 0x7f99b40a2f00 2024-04-22T00:12:15.330 INFO:teuthology.orchestra.run.smithi038.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T00:02:13", "first": "2024-04-22T00:03:00", "last": "2024-04-22T00:12:00", "last_pruned": "2024-04-22T00:12:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-22T00:12:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 msgr2=0x7f99b40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 0x7f99b40a53c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005fd0 tx=0x7f99b0005850 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 msgr2=0x7f99c0132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f99c8064f90 tx=0x7f99c809b610 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 shutdown_connections 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f99b40a2f00 0x7f99b40a53c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99c00a6a30 0x7f99c0132a40 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99c009bd50 0x7f99c0138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 --2- 172.21.15.38:0/2298833655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c0094790 0x7f99c0138890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.336+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 >> 172.21.15.38:0/2298833655 conn(0x7f99c0009ab0 msgr2=0x7f99c00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.337+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 shutdown_connections 2024-04-22T00:12:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.337+0000 7f99cd268640 1 -- 172.21.15.38:0/2298833655 wait complete. 2024-04-22T00:12:15.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T00:12:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.486+0000 7f8bf33cb640 1 -- 172.21.15.38:0/829784623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec104320 msgr2=0x7f8bec110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.486+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/829784623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec104320 0x7f8bec110830 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009cd0 tx=0x7f8be402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 -- 172.21.15.38:0/829784623 shutdown_connections 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/829784623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec104320 0x7f8bec110830 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/829784623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec103960 0x7f8bec103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/829784623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 0x7f8bec109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 -- 172.21.15.38:0/829784623 >> 172.21.15.38:0/829784623 conn(0x7f8bec0ff450 msgr2=0x7f8bec101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 -- 172.21.15.38:0/829784623 shutdown_connections 2024-04-22T00:12:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.487+0000 7f8bf33cb640 1 -- 172.21.15.38:0/829784623 wait complete. 2024-04-22T00:12:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 Processor -- start 2024-04-22T00:12:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 -- start start 2024-04-22T00:12:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec104320 0x7f8bec19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 0x7f8bec19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113d80 con 0x7f8bec103960 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113c00 con 0x7f8bec109970 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf33cb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec113f00 con 0x7f8bec104320 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf1167640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf1167640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47126/0 (socket says 172.21.15.38:47126) 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf1167640 1 -- 172.21.15.38:0/961864799 learned_addr learned my addr 172.21.15.38:0/961864799 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf0966640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec104320 0x7f8bec19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.488+0000 7f8bf1968640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 0x7f8bec19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 -- 172.21.15.38:0/961864799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec104320 msgr2=0x7f8bec19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec104320 0x7f8bec19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 -- 172.21.15.38:0/961864799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 msgr2=0x7f8bec19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 0x7f8bec19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 -- 172.21.15.38:0/961864799 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bec19f170 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf1167640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8be800e9f0 tx=0x7f8be800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be800cd80 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be8010910 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8018990 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bec19f460 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.489+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bec1a6cf0 con 0x7f8bec103960 2024-04-22T00:12:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.490+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bec04f500 con 0x7f8bec103960 2024-04-22T00:12:15.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.492+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f8be8010ac0 con 0x7f8bec103960 2024-04-22T00:12:15.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.492+0000 7f8be27fc640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 0x7f8bd80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:15.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.492+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9526+0+0 (secure 0 0 0) 0x7f8be80cfec0 con 0x7f8bec103960 2024-04-22T00:12:15.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.494+0000 7f8bf0966640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 0x7f8bd80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:15.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.495+0000 7f8bf0966640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 0x7f8bd80a5360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8bec108eb0 tx=0x7f8bdc005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:15.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.498+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8be809c040 con 0x7f8bec103960 2024-04-22T00:12:15.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.783+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 --> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8bec100080 con 0x7f8bd80a2ea0 2024-04-22T00:12:15.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.792+0000 7f8be27fc640 1 -- 172.21.15.38:0/961864799 <== mgr.15528 v2:172.21.15.38:6800/2750091046 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8bec100080 con 0x7f8bd80a2ea0 2024-04-22T00:12:15.789 INFO:teuthology.orchestra.run.smithi038.stdout:/ 1m 6m3h 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 msgr2=0x7f8bd80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 0x7f8bd80a5360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8bec108eb0 tx=0x7f8bdc005f90 comp rx=0 tx=0).stop 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 msgr2=0x7f8bec19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f8be800e9f0 tx=0x7f8be800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 shutdown_connections 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f8bd80a2ea0 0x7f8bd80a5360 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8bec109970 0x7f8bec19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bec104320 0x7f8bec19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 --2- 172.21.15.38:0/961864799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8bec103960 0x7f8bec19a180 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 >> 172.21.15.38:0/961864799 conn(0x7f8bec0ff450 msgr2=0x7f8bec0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 shutdown_connections 2024-04-22T00:12:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:15.794+0000 7f8bf33cb640 1 -- 172.21.15.38:0/961864799 wait complete. 2024-04-22T00:12:15.804 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T00:12:15.861 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T00:12:16.383 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:16 smithi111 ceph-mon[31543]: pgmap v315: 129 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:12:16.383 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:16 smithi111 ceph-mon[31543]: from='client.18822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:16 smithi038 ceph-mon[27972]: pgmap v315: 129 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:12:16.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:16 smithi038 ceph-mon[27972]: from='client.18822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:16 smithi203 ceph-mon[31431]: pgmap v315: 129 pgs: 12 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 89 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:12:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:16 smithi203 ceph-mon[31431]: from='client.18822 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:17.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:17 smithi038 ceph-mon[27972]: from='client.18828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:17.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:17 smithi203 ceph-mon[31431]: from='client.18828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:17.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:17 smithi111 ceph-mon[31543]: from='client.18828 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T00:12:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:18 smithi038 ceph-mon[27972]: pgmap v316: 129 pgs: 7 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 108 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 195 B/s wr, 10 op/s 2024-04-22T00:12:18.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:18 smithi203 ceph-mon[31431]: pgmap v316: 129 pgs: 7 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 108 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 195 B/s wr, 10 op/s 2024-04-22T00:12:18.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:18 smithi111 ceph-mon[31543]: pgmap v316: 129 pgs: 7 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 108 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 195 B/s wr, 10 op/s 2024-04-22T00:12:20.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:20 smithi038 ceph-mon[27972]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-22T00:12:20.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:20 smithi203 ceph-mon[31431]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-22T00:12:20.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:20 smithi111 ceph-mon[31543]: pgmap v317: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-22T00:12:22.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:22 smithi038 ceph-mon[27972]: pgmap v318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-22T00:12:22.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:22 smithi038 ceph-mon[27972]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T00:12:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:22 smithi203 ceph-mon[31431]: pgmap v318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-22T00:12:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:22 smithi203 ceph-mon[31431]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T00:12:22.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:22 smithi111 ceph-mon[31543]: pgmap v318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-22T00:12:22.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:22 smithi111 ceph-mon[31543]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T00:12:24.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:24 smithi038 ceph-mon[27972]: pgmap v320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:24.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:24 smithi203 ceph-mon[31431]: pgmap v320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:24.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:24 smithi111 ceph-mon[31543]: pgmap v320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:26.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:26 smithi038 ceph-mon[27972]: pgmap v321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:26.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:26 smithi203 ceph-mon[31431]: pgmap v321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:26.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:26 smithi111 ceph-mon[31543]: pgmap v321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:28.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:28 smithi038 ceph-mon[27972]: pgmap v322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:28.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:28 smithi203 ceph-mon[31431]: pgmap v322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:28.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:28 smithi111 ceph-mon[31543]: pgmap v322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:29.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T00:12:29.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T00:12:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 -- 172.21.15.38:0/2708266964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c100990 msgr2=0x7fba6c100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 --2- 172.21.15.38:0/2708266964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c100990 0x7fba6c100df0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fba60009900 tx=0x7fba6002bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 -- 172.21.15.38:0/2708266964 shutdown_connections 2024-04-22T00:12:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 --2- 172.21.15.38:0/2708266964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c101330 0x7fba6c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 --2- 172.21.15.38:0/2708266964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c100990 0x7fba6c100df0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 --2- 172.21.15.38:0/2708266964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c106690 0x7fba6c106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 -- 172.21.15.38:0/2708266964 >> 172.21.15.38:0/2708266964 conn(0x7fba6c076440 msgr2=0x7fba6c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 -- 172.21.15.38:0/2708266964 shutdown_connections 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.984+0000 7fba71f1e640 1 -- 172.21.15.38:0/2708266964 wait complete. 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 Processor -- start 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 -- start start 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 0x7fba6c196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 0x7fba6c19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c110a50 con 0x7fba6c101330 2024-04-22T00:12:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1108d0 con 0x7fba6c106690 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba71f1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c110bd0 con 0x7fba6c100990 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38654/0 (socket says 172.21.15.38:38654) 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba6affd640 1 -- 172.21.15.38:0/3937839266 learned_addr learned my addr 172.21.15.38:0/3937839266 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 0x7fba6c196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.985+0000 7fba6bfff640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 0x7fba6c19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 -- 172.21.15.38:0/3937839266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 msgr2=0x7fba6c196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 0x7fba6c196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 -- 172.21.15.38:0/3937839266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 msgr2=0x7fba6c19b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 0x7fba6c19b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 -- 172.21.15.38:0/3937839266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c19beb0 con 0x7fba6c101330 2024-04-22T00:12:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 0x7fba6c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6bfff640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 0x7fba6c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba6affd640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fba6002c240 tx=0x7fba60004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba68ff9640 1 -- 172.21.15.38:0/3937839266 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60043020 con 0x7fba6c101330 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba68ff9640 1 -- 172.21.15.38:0/3937839266 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba600054e0 con 0x7fba6c101330 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6c19c140 con 0x7fba6c101330 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba68ff9640 1 -- 172.21.15.38:0/3937839266 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60003cc0 con 0x7fba6c101330 2024-04-22T00:12:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.986+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6c1a3a30 con 0x7fba6c101330 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.988+0000 7fba68ff9640 1 -- 172.21.15.38:0/3937839266 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fba60046020 con 0x7fba6c101330 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 Processor -- start 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 -- start start 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30c140 0x56285f30c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f315640 0x56285f317a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56285f26e560 con 0x56285f30ca60 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56285f216730 con 0x56285f30c140 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba74be4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56285f26a520 con 0x56285f315640 2024-04-22T00:12:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba68ff9640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 0x7fba540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38662/0 (socket says 172.21.15.38:38662) 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6b7fe640 1 -- 172.21.15.38:0/2665665471 learned_addr learned my addr 172.21.15.38:0/2665665471 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6affd640 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f315640 0x56285f317a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6bfff640 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30c140 0x56285f30c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba6bfff640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 0x7fba540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.989+0000 7fba68ff9640 1 -- 172.21.15.38:0/3937839266 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fba600f19e0 con 0x7fba6c101330 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 -- 172.21.15.38:0/2665665471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f315640 msgr2=0x56285f317a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f315640 0x56285f317a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 -- 172.21.15.38:0/2665665471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30c140 msgr2=0x56285f30c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30c140 0x56285f30c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 -- 172.21.15.38:0/2665665471 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56285f04c7d0 con 0x56285f30ca60 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007850 tx=0x7fba5c007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527152995207c809 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:29.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba6bfff640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 0x7fba540a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fba64009270 tx=0x7fba64005bd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba51ffb640 1 -- 172.21.15.38:0/2665665471 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00d950 con 0x56285f30ca60 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba51ffb640 1 -- 172.21.15.38:0/2665665471 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5c018070 con 0x56285f30ca60 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.990+0000 7fba51ffb640 1 -- 172.21.15.38:0/2665665471 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c013e60 con 0x56285f30ca60 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- 172.21.15.38:0/2665665471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 msgr2=0x56285f315100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fba5c007850 tx=0x7fba5c007d10 comp rx=0 tx=0).stop 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- 172.21.15.38:0/2665665471 shutdown_connections 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f315640 0x56285f317a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f315100 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 --2- 172.21.15.38:0/2665665471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30c140 0x56285f30c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- 172.21.15.38:0/2665665471 >> 172.21.15.38:0/2665665471 conn(0x56285f306870 msgr2=0x56285f306c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- 172.21.15.38:0/2665665471 shutdown_connections 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- 172.21.15.38:0/2665665471 wait complete. 2024-04-22T00:12:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 -- start start 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.991+0000 7fba74be4740 1 Processor -- start 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f312600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f313080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56285f318430 con 0x56285f30ca60 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x56285f229e00 con 0x56285f315640 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56285f3185b0 con 0x56285f30c140 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba74be4740 10 client.0 _ll_register_callbacks cb 0x56285f264eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38670/0 (socket says 172.21.15.38:38670) 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 -- 172.21.15.38:0/839532037 learned_addr learned my addr 172.21.15.38:0/839532037 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f312600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6bfff640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f313080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 -- 172.21.15.38:0/839532037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 msgr2=0x56285f312600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f312600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 -- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 msgr2=0x56285f313080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f313080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 -- 172.21.15.38:0/839532037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56285ef823f0 con 0x56285f30ca60 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba6affd640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fba600bdbb0 tx=0x7fba6008cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6a29103db3c188 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba4affd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60088cb0 con 0x56285f30ca60 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.992+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fba600eefc0 con 0x56285f30ca60 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba5c018070 con 0x56285f30ca60 2024-04-22T00:12:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fba600ef1c0 con 0x56285f30ca60 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 10 client.18840 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56285ef420f0 con 0x56285f30ca60 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5c018070 con 0x56285f30ca60 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fba600c6400 con 0x56285f30ca60 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 10 client.18840 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 10 client.18840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 20 client.18840 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f3258c0 0x56285f327d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 4 client.18840 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56285f27baa0 con 0x56285f3258c0 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.993+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f3258c0 0x56285f327d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.994+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f3258c0 0x56285f327d20 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec9328c0b0680df2 server_cookie=17bdd94883114276 in_seq=0 out_seq=0 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.994+0000 7fba4affd640 10 client.18840 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba4affd640 1 -- 172.21.15.38:0/839532037 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56285f27baa0 con 0x56285f3258c0 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba4affd640 10 client.18840 handle_command_reply: tid=0 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 client.18840 shutdown 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 msgr2=0x56285f312b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fba600bdbb0 tx=0x7fba6008cb50 comp rx=0 tx=0).stop 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 shutdown_connections 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f313080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f3258c0 0x56285f327d20 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30ca60 0x56285f312b40 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 --2- 172.21.15.38:0/839532037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f312600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba4affd640 0 client.18840 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 >> 172.21.15.38:0/839532037 conn(0x56285f306870 msgr2=0x56285f3085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 shutdown_connections 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 1 -- 172.21.15.38:0/839532037 wait complete. 2024-04-22T00:12:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:29.995+0000 7fba74be4740 20 client.18840 trim_cache size 0 max 16384 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 Processor -- start 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 -- start start 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30ca60 0x56285f321320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f303560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1ac8f0 con 0x56285f30c140 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1ae430 con 0x56285f315640 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba74be4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1acfc0 con 0x56285f30ca60 2024-04-22T00:12:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38682/0 (socket says 172.21.15.38:38682) 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 -- 172.21.15.38:0/3719362326 learned_addr learned my addr 172.21.15.38:0/3719362326 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 -- 172.21.15.38:0/3719362326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30ca60 msgr2=0x56285f321320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30ca60 0x56285f321320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 -- 172.21.15.38:0/3719362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 msgr2=0x56285f303560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f303560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.013+0000 7fba6b7fe640 1 -- 172.21.15.38:0/3719362326 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56285f3066a0 con 0x56285f30c140 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fba5c004810 tx=0x7fba5c00f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ade4ff5472e7d26 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba517fa640 1 -- 172.21.15.38:0/3719362326 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c013680 con 0x56285f30c140 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba517fa640 1 -- 172.21.15.38:0/3719362326 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5c018040 con 0x56285f30c140 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba517fa640 1 -- 172.21.15.38:0/3719362326 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c026590 con 0x56285f30c140 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 -- 172.21.15.38:0/3719362326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 msgr2=0x56285f314d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fba5c004810 tx=0x7fba5c00f040 comp rx=0 tx=0).stop 2024-04-22T00:12:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 -- 172.21.15.38:0/3719362326 shutdown_connections 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f315640 0x56285f303560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30ca60 0x56285f321320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 --2- 172.21.15.38:0/3719362326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f30c140 0x56285f314d30 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 -- 172.21.15.38:0/3719362326 >> 172.21.15.38:0/3719362326 conn(0x56285f306870 msgr2=0x56285f302e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 -- 172.21.15.38:0/3719362326 shutdown_connections 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.014+0000 7fba74be4740 1 -- 172.21.15.38:0/3719362326 wait complete. 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 -- start start 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 Processor -- start 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f319440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30ca60 0x56285f319980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1add60 con 0x56285f315640 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1ad690 con 0x56285f30ca60 2024-04-22T00:12:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c1aa8f0 con 0x56285f30c140 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba74be4740 10 client.0 _ll_register_callbacks cb 0x7fba600035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38690/0 (socket says 172.21.15.38:38690) 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba6bfff640 1 -- 172.21.15.38:0/1401137805 learned_addr learned my addr 172.21.15.38:0/1401137805 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.015+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f319440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6affd640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30ca60 0x56285f319980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 -- 172.21.15.38:0/1401137805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 msgr2=0x56285f319440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f319440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 -- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30ca60 msgr2=0x56285f319980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30ca60 0x56285f319980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 -- 172.21.15.38:0/1401137805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56285f38b060 con 0x56285f315640 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba6bfff640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fba64009bf0 tx=0x7fba64009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f997337e85127038 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba4affd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64026030 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fba64004b20 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba5c018040 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fba64032050 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba74be4740 10 client.18846 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56285f38c350 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.016+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5c018040 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fba6400f040 con 0x56285f315640 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 10 client.18846 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 10 client.18846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 20 client.18846 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f38cb30 0x56285f38ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 4 client.18846 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56285f275df0 con 0x56285f38cb30 2024-04-22T00:12:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f38cb30 0x56285f38ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba6b7fe640 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f38cb30 0x56285f38ef50 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77da351cce42f491 server_cookie=7309af5035849d04 in_seq=0 out_seq=0 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba4affd640 10 client.18846 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba4affd640 1 -- 172.21.15.38:0/1401137805 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56285f275df0 con 0x56285f38cb30 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba4affd640 10 client.18846 handle_command_reply: tid=0 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.017+0000 7fba74be4740 1 client.18846 shutdown 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 msgr2=0x56285f319ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fba64009bf0 tx=0x7fba64009640 comp rx=0 tx=0).stop 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 shutdown_connections 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x56285f38cb30 0x56285f38ef50 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56285f315640 0x56285f319ec0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56285f30ca60 0x56285f319980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba4affd640 0 client.18846 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 --2- 172.21.15.38:0/1401137805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56285f30c140 0x56285f319440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 >> 172.21.15.38:0/1401137805 conn(0x56285f306870 msgr2=0x56285f312e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 shutdown_connections 2024-04-22T00:12:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 1 -- 172.21.15.38:0/1401137805 wait complete. 2024-04-22T00:12:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba74be4740 20 client.18846 trim_cache size 0 max 16384 2024-04-22T00:12:30.023 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:30.023 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2024-04-22T00:12:30.024 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2024-04-22T00:12:30.024 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 msgr2=0x7fba540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 0x7fba540a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fba64009270 tx=0x7fba64005bd0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 msgr2=0x7fba6c197450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fba6002c240 tx=0x7fba60004430 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 shutdown_connections 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fba540a2d00 0x7fba540a51c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba6c106690 0x7fba6c19b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba6c101330 0x7fba6c197450 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 --2- 172.21.15.38:0/3937839266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba6c100990 0x7fba6c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.018+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 >> 172.21.15.38:0/3937839266 conn(0x7fba6c076440 msgr2=0x7fba6c0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.019+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 shutdown_connections 2024-04-22T00:12:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.019+0000 7fba71f1e640 1 -- 172.21.15.38:0/3937839266 wait complete. 2024-04-22T00:12:30.027 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T00:12:30.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:12:30.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 -- 172.21.15.38:0/1202697423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 msgr2=0x7fd4a0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1202697423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0102e00 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fd490009900 tx=0x7fd49002bcd0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 -- 172.21.15.38:0/1202697423 shutdown_connections 2024-04-22T00:12:30.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1202697423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a010bd60 0x7fd4a010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1202697423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a0103340 0x7fd4a010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1202697423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0102e00 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 -- 172.21.15.38:0/1202697423 >> 172.21.15.38:0/1202697423 conn(0x7fd4a00777a0 msgr2=0x7fd4a0077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 -- 172.21.15.38:0/1202697423 shutdown_connections 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.132+0000 7fd4a586d640 1 -- 172.21.15.38:0/1202697423 wait complete. 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 Processor -- start 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 -- start start 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a0103340 0x7fd4a0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a010bd60 0x7fd4a019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a0111b30 con 0x7fd4a0102a20 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01119b0 con 0x7fd4a010bd60 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd4a586d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a0111cb0 con 0x7fd4a0103340 2024-04-22T00:12:30.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd49effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd49effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38716/0 (socket says 172.21.15.38:38716) 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd49effd640 1 -- 172.21.15.38:0/1116291878 learned_addr learned my addr 172.21.15.38:0/1116291878 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd497fff640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a0103340 0x7fd4a0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.133+0000 7fd49f7fe640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a010bd60 0x7fd4a019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 -- 172.21.15.38:0/1116291878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a0103340 msgr2=0x7fd4a0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a0103340 0x7fd4a0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 -- 172.21.15.38:0/1116291878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a010bd60 msgr2=0x7fd4a019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a010bd60 0x7fd4a019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 -- 172.21.15.38:0/1116291878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a019cfc0 con 0x7fd4a0102a20 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd49effd640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd490005940 tx=0x7fd49002cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490041070 con 0x7fd4a0102a20 2024-04-22T00:12:30.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd490004450 con 0x7fd4a0102a20 2024-04-22T00:12:30.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a019d250 con 0x7fd4a0102a20 2024-04-22T00:12:30.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490034440 con 0x7fd4a0102a20 2024-04-22T00:12:30.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.134+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a01a4b40 con 0x7fd4a0102a20 2024-04-22T00:12:30.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.135+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a00689b0 con 0x7fd4a0102a20 2024-04-22T00:12:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.136+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd490044070 con 0x7fd4a0102a20 2024-04-22T00:12:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.137+0000 7fd4a486b640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 0x7fd48c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.137+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd4900f1490 con 0x7fd4a0102a20 2024-04-22T00:12:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.137+0000 7fd497fff640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 0x7fd48c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.137+0000 7fd497fff640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 0x7fd48c0a51c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd4a010da10 tx=0x7fd48800b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.137 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.140+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd4900bded0 con 0x7fd4a0102a20 2024-04-22T00:12:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.365+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4a01a4d10 con 0x7fd4a0102a20 2024-04-22T00:12:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.366+0000 7fd4a486b640 1 -- 172.21.15.38:0/1116291878 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11935 (secure 0 0 0) 0x7fd4900bd7d0 con 0x7fd4a0102a20 2024-04-22T00:12:30.364 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:12:30.364 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:12:03.690688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:12:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 42 2024-04-22T00:12:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 msgr2=0x7fd48c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 0x7fd48c0a51c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd4a010da10 tx=0x7fd48800b390 comp rx=0 tx=0).stop 2024-04-22T00:12:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 msgr2=0x7fd4a0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fd490005940 tx=0x7fd49002cd10 comp rx=0 tx=0).stop 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 shutdown_connections 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd48c0a2d00 0x7fd48c0a51c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4a010bd60 0x7fd4a019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4a0103340 0x7fd4a0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 --2- 172.21.15.38:0/1116291878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4a0102a20 0x7fd4a0197fd0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 >> 172.21.15.38:0/1116291878 conn(0x7fd4a00777a0 msgr2=0x7fd4a00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 shutdown_connections 2024-04-22T00:12:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.368+0000 7fd4a586d640 1 -- 172.21.15.38:0/1116291878 wait complete. 2024-04-22T00:12:30.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 -- 172.21.15.38:0/3843438010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069d40 msgr2=0x7f4a8406a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3843438010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069d40 0x7f4a8406a1c0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009930 tx=0x7f4a7802bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 -- 172.21.15.38:0/3843438010 shutdown_connections 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3843438010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 0x7f4a8410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3843438010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84069d40 0x7f4a8406a1c0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3843438010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84107720 0x7f4a84107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 -- 172.21.15.38:0/3843438010 >> 172.21.15.38:0/3843438010 conn(0x7f4a84076440 msgr2=0x7f4a84076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 -- 172.21.15.38:0/3843438010 shutdown_connections 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 -- 172.21.15.38:0/3843438010 wait complete. 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.484+0000 7f4a89e66640 1 Processor -- start 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 -- start start 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84069d40 0x7f4a84198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 0x7f4a841985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84111b30 con 0x7f4a84107720 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841119b0 con 0x7f4a84069d40 2024-04-22T00:12:30.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a89e66640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a84111cb0 con 0x7f4a8406a700 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38732/0 (socket says 172.21.15.38:38732) 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 -- 172.21.15.38:0/3470057350 learned_addr learned my addr 172.21.15.38:0/3470057350 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a837fe640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84069d40 0x7f4a84198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 0x7f4a841985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 -- 172.21.15.38:0/3470057350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 msgr2=0x7f4a841985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 0x7f4a841985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 -- 172.21.15.38:0/3470057350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84069d40 msgr2=0x7f4a84198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84069d40 0x7f4a84198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 -- 172.21.15.38:0/3470057350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8419d030 con 0x7f4a84107720 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.485+0000 7f4a83fff640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00eaa0 tx=0x7f4a7c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.486+0000 7f4a80ff9640 1 -- 172.21.15.38:0/3470057350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00cdb0 con 0x7f4a84107720 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.486+0000 7f4a80ff9640 1 -- 172.21.15.38:0/3470057350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a7c005020 con 0x7f4a84107720 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.486+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8419d320 con 0x7f4a84107720 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.486+0000 7f4a80ff9640 1 -- 172.21.15.38:0/3470057350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c0116b0 con 0x7f4a84107720 2024-04-22T00:12:30.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.486+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a841a4bb0 con 0x7f4a84107720 2024-04-22T00:12:30.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a80ff9640 1 -- 172.21.15.38:0/3470057350 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4a7c005560 con 0x7f4a84107720 2024-04-22T00:12:30.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 Processor -- start 2024-04-22T00:12:30.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 -- start start 2024-04-22T00:12:30.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a80ff9640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 0x7f4a6c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x56031501b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560314f74530 con 0x560315010940 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560314f1c830 con 0x560315019520 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.488+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560314f704f0 con 0x560315010020 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 0x7f4a6c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a80ff9640 1 -- 172.21.15.38:0/3470057350 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f4a7c0cfae0 con 0x7f4a84107720 2024-04-22T00:12:30.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a82ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:36576/0 (socket says 172.21.15.38:36576) 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 -- 172.21.15.38:0/1257077326 learned_addr learned my addr 172.21.15.38:0/1257077326 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a82ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38748/0 (socket says 172.21.15.38:38748) 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a83fff640 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x56031501b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 -- 172.21.15.38:0/1257077326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 msgr2=0x56031501b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x56031501b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 -- 172.21.15.38:0/1257077326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 msgr2=0x560315018fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 -- 172.21.15.38:0/1257077326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560314f402c0 con 0x560315010020 2024-04-22T00:12:30.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 0x7f4a6c0a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006010 tx=0x7f4a78009680 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a837fe640 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400bc60 tx=0x7f4a7400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34a8639fe3a86590 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.489+0000 7f4a69ffb640 1 -- 172.21.15.38:0/1257077326 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74015070 con 0x560315010020 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a69ffb640 1 -- 172.21.15.38:0/1257077326 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a74011920 con 0x560315010020 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a69ffb640 1 -- 172.21.15.38:0/1257077326 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74010890 con 0x560315010020 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- 172.21.15.38:0/1257077326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 msgr2=0x560315010400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400bc60 tx=0x7f4a7400f730 comp rx=0 tx=0).stop 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- 172.21.15.38:0/1257077326 shutdown_connections 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x56031501b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010940 0x560315018fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/1257077326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010020 0x560315010400 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- 172.21.15.38:0/1257077326 >> 172.21.15.38:0/1257077326 conn(0x56031500a750 msgr2=0x56031500ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- 172.21.15.38:0/1257077326 shutdown_connections 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- 172.21.15.38:0/1257077326 wait complete. 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.490+0000 7f4a8ca49740 1 -- start start 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 Processor -- start 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x560315016270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315016880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56031501c310 con 0x560315010020 2024-04-22T00:12:30.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x560314f2ff10 con 0x560315010940 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56031501c490 con 0x560315019520 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a8ca49740 10 client.0 _ll_register_callbacks cb 0x560314c87be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38760/0 (socket says 172.21.15.38:38760) 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 -- 172.21.15.38:0/388040639 learned_addr learned my addr 172.21.15.38:0/388040639 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a83fff640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315016880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x560315016270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 -- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 msgr2=0x560315016880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315016880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 -- 172.21.15.38:0/388040639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 msgr2=0x560315016270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x560315016270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.491+0000 7f4a837fe640 1 -- 172.21.15.38:0/388040639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560314f01c80 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a837fe640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4a74005d20 tx=0x7f4a7400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b297768922b9d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74015070 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f4a74011710 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a74011920 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4a7402e050 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 10 client.18864 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560314c883f0 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a74011920 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f4a740048b0 con 0x560315010020 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 10 client.18864 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 10 client.18864 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 20 client.18864 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315029570 0x56031502b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 4 client.18864 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.492+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560314f81a70 con 0x560315029570 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.493+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315029570 0x56031502b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.493+0000 7f4a62ffd640 10 client.18864 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.493+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315029570 0x56031502b960 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b2a746e6bda57838 server_cookie=dac9cad20121879d in_seq=0 out_seq=0 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a62ffd640 1 -- 172.21.15.38:0/388040639 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560314f81a70 con 0x560315029570 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a62ffd640 10 client.18864 handle_command_reply: tid=0 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 client.18864 shutdown 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 msgr2=0x560315018ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f4a74005d20 tx=0x7f4a7400b830 comp rx=0 tx=0).stop 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 shutdown_connections 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315029570 0x56031502b960 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315016880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a62ffd640 0 client.18864 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x560315016270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/388040639 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x560315018ee0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 >> 172.21.15.38:0/388040639 conn(0x56031500a750 msgr2=0x56031500c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 shutdown_connections 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 1 -- 172.21.15.38:0/388040639 wait complete. 2024-04-22T00:12:30.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.494+0000 7f4a8ca49740 20 client.18864 trim_cache size 0 max 16384 2024-04-22T00:12:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 Processor -- start 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 -- start start 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010940 0x560315021ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x5603150189a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841aed00 con 0x560315010020 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841b0840 con 0x560315019520 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841af3d0 con 0x560315010940 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38770/0 (socket says 172.21.15.38:38770) 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 -- 172.21.15.38:0/2648904737 learned_addr learned my addr 172.21.15.38:0/2648904737 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010940 0x560315021ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 -- 172.21.15.38:0/2648904737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010940 msgr2=0x560315021ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010940 0x560315021ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a83fff640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x5603150189a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 -- 172.21.15.38:0/2648904737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 msgr2=0x5603150189a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x5603150189a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 -- 172.21.15.38:0/2648904737 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560314d527d0 con 0x560315010020 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a83fff640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x5603150189a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a837fe640 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400b9f0 tx=0x7f4a74005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a0bac1940a6e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a697fa640 1 -- 172.21.15.38:0/2648904737 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74037070 con 0x560315010020 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a697fa640 1 -- 172.21.15.38:0/2648904737 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a74010820 con 0x560315010020 2024-04-22T00:12:30.512 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.508+0000 7f4a697fa640 1 -- 172.21.15.38:0/2648904737 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74015070 con 0x560315010020 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- 172.21.15.38:0/2648904737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 msgr2=0x5603150251d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400b9f0 tx=0x7f4a74005870 comp rx=0 tx=0).stop 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- 172.21.15.38:0/2648904737 shutdown_connections 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315019520 0x5603150189a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315010940 0x560315021ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/2648904737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x5603150251d0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- 172.21.15.38:0/2648904737 >> 172.21.15.38:0/2648904737 conn(0x56031500a750 msgr2=0x5603150152a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- 172.21.15.38:0/2648904737 shutdown_connections 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- 172.21.15.38:0/2648904737 wait complete. 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.509+0000 7f4a8ca49740 1 -- start start 2024-04-22T00:12:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 Processor -- start 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x56031508c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315017ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841b0170 con 0x560315010020 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841afaa0 con 0x560315010940 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a841acd00 con 0x560315019520 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a8ca49740 10 client.0 _ll_register_callbacks cb 0x7f4a78009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38780/0 (socket says 172.21.15.38:38780) 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 -- 172.21.15.38:0/3717729754 learned_addr learned my addr 172.21.15.38:0/3717729754 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x56031508c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a83fff640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315017ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 -- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 msgr2=0x560315017ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315017ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 -- 172.21.15.38:0/3717729754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 msgr2=0x56031508c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x56031508c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 -- 172.21.15.38:0/3717729754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560315018600 con 0x560315010020 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a837fe640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f4a74004c50 tx=0x7f4a7400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7ca90472878c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.510+0000 7f4a62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74040060 con 0x560315010020 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f4a74004680 con 0x560315010020 2024-04-22T00:12:30.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a74010820 con 0x560315010020 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4a74047050 con 0x560315010020 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 10 client.18870 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5603150935f0 con 0x560315010020 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a74010820 con 0x560315010020 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f4a74011710 con 0x560315010020 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 10 client.18870 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 10 client.18870 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 20 client.18870 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315093bf0 0x560315095fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 4 client.18870 mds_command: new command op to 24448 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:12:30.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 --> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560314f7bdc0 con 0x560315093bf0 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.511+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315093bf0 0x560315095fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a62ffd640 10 client.18870 ms_handle_connect on v2:172.21.15.203:6834/521403621 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a82ffd640 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315093bf0 0x560315095fe0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9b3ae48c022d9d7 server_cookie=e57b0530493c15b in_seq=0 out_seq=0 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a62ffd640 1 -- 172.21.15.38:0/3717729754 <== mds.4 v2:172.21.15.203:6834/521403621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560314f7bdc0 con 0x560315093bf0 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a62ffd640 10 client.18870 handle_command_reply: tid=0 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 client.18870 shutdown 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 msgr2=0x56031508be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f4a74004c50 tx=0x7f4a7400ff70 comp rx=0 tx=0).stop 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 shutdown_connections 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:6834/521403621,v1:172.21.15.203:6835/521403621] conn(0x560315093bf0 0x560315095fe0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560315019520 0x560315017ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560315010940 0x56031508c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a62ffd640 0 client.18870 ms_handle_reset on v2:172.21.15.203:6834/521403621 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 --2- 172.21.15.38:0/3717729754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560315010020 0x56031508be60 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 >> 172.21.15.38:0/3717729754 conn(0x56031500a750 msgr2=0x560315008fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 shutdown_connections 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 1 -- 172.21.15.38:0/3717729754 wait complete. 2024-04-22T00:12:30.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.512+0000 7f4a8ca49740 20 client.18870 trim_cache size 0 max 16384 2024-04-22T00:12:30.519 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 msgr2=0x7f4a6c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 0x7f4a6c0a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4a78006010 tx=0x7f4a78009680 comp rx=0 tx=0).stop 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 msgr2=0x7f4a8419c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00eaa0 tx=0x7f4a7c00ef60 comp rx=0 tx=0).stop 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 shutdown_connections 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f4a6c0a2d00 0x7f4a6c0a51c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4a84107720 0x7f4a8419c900 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a8406a700 0x7f4a841985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 --2- 172.21.15.38:0/3470057350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a84069d40 0x7f4a84198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 >> 172.21.15.38:0/3470057350 conn(0x7f4a84076440 msgr2=0x7f4a840fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 shutdown_connections 2024-04-22T00:12:30.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.513+0000 7f4a89e66640 1 -- 172.21.15.38:0/3470057350 wait complete. 2024-04-22T00:12:30.521 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:30 smithi038 ceph-mon[27972]: pgmap v323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:30.522 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:30 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1116291878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:12:30.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 -- 172.21.15.38:0/592290811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100610 msgr2=0x7ffa1c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 --2- 172.21.15.38:0/592290811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100610 0x7ffa1c100a90 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009930 tx=0x7ffa0802bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 -- 172.21.15.38:0/592290811 shutdown_connections 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 --2- 172.21.15.38:0/592290811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 --2- 172.21.15.38:0/592290811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100610 0x7ffa1c100a90 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 --2- 172.21.15.38:0/592290811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c1066b0 0x7ffa1c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 -- 172.21.15.38:0/592290811 >> 172.21.15.38:0/592290811 conn(0x7ffa1c077740 msgr2=0x7ffa1c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 -- 172.21.15.38:0/592290811 shutdown_connections 2024-04-22T00:12:30.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.629+0000 7ffa23193640 1 -- 172.21.15.38:0/592290811 wait complete. 2024-04-22T00:12:30.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 Processor -- start 2024-04-22T00:12:30.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 -- start start 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c100610 0x7ffa1c198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c1066b0 0x7ffa1c1a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1109f0 con 0x7ffa1c100fd0 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c110870 con 0x7ffa1c100610 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa23193640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c110b70 con 0x7ffa1c1066b0 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38792/0 (socket says 172.21.15.38:38792) 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa1bfff640 1 -- 172.21.15.38:0/3843083649 learned_addr learned my addr 172.21.15.38:0/3843083649 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c100610 0x7ffa1c198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.630+0000 7ffa21730640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c1066b0 0x7ffa1c1a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 -- 172.21.15.38:0/3843083649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c1066b0 msgr2=0x7ffa1c1a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c1066b0 0x7ffa1c1a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 -- 172.21.15.38:0/3843083649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c100610 msgr2=0x7ffa1c198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c100610 0x7ffa1c198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 -- 172.21.15.38:0/3843083649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c1a4450 con 0x7ffa1c100fd0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009230 tx=0x7ffa0802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa19ffb640 1 -- 172.21.15.38:0/3843083649 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08041070 con 0x7ffa1c100fd0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c1a4620 con 0x7ffa1c100fd0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa19ffb640 1 -- 172.21.15.38:0/3843083649 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa08004450 con 0x7ffa1c100fd0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa19ffb640 1 -- 172.21.15.38:0/3843083649 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08034440 con 0x7ffa1c100fd0 2024-04-22T00:12:30.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.631+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c1a4ae0 con 0x7ffa1c100fd0 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.633+0000 7ffa19ffb640 1 -- 172.21.15.38:0/3843083649 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ffa08044070 con 0x7ffa1c100fd0 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.633+0000 7ffa25d76740 1 Processor -- start 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 -- start start 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c55b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55705c4b2530 con 0x55705c54dcd0 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55705c45a830 con 0x55705c54e5f0 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa25d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55705c4ae4f0 con 0x55705c55b930 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38804/0 (socket says 172.21.15.38:38804) 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 -- 172.21.15.38:0/1753715090 learned_addr learned my addr 172.21.15.38:0/1753715090 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c55b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa19ffb640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 0x7ffa0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 -- 172.21.15.38:0/1753715090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 msgr2=0x55705c55dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 -- 172.21.15.38:0/1753715090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 msgr2=0x55705c55b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c55b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 -- 172.21.15.38:0/1753715090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55705c47e2c0 con 0x55705c54dcd0 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c55b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa21730640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 0x7ffa0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.634+0000 7ffa19ffb640 1 -- 172.21.15.38:0/3843083649 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ffa080f1120 con 0x7ffa1c100fd0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa21730640 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffa140103f0 tx=0x7ffa140108b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a61026d244ad3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa06ffd640 1 -- 172.21.15.38:0/1753715090 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1401c900 con 0x55705c54dcd0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa06ffd640 1 -- 172.21.15.38:0/1753715090 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1401b3d0 con 0x55705c54dcd0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa06ffd640 1 -- 172.21.15.38:0/1753715090 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa140255e0 con 0x55705c54dcd0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 -- 172.21.15.38:0/1753715090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 msgr2=0x55705c54e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa21730640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 0x7ffa0c0a51c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffa14020000 tx=0x7ffa1401d000 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffa140103f0 tx=0x7ffa140108b0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 -- 172.21.15.38:0/1753715090 shutdown_connections 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c55b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1753715090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c54e0b0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 -- 172.21.15.38:0/1753715090 >> 172.21.15.38:0/1753715090 conn(0x55705c548750 msgr2=0x55705c548b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.635+0000 7ffa25d76740 1 -- 172.21.15.38:0/1753715090 shutdown_connections 2024-04-22T00:12:30.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 -- 172.21.15.38:0/1753715090 wait complete. 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 -- start start 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 Processor -- start 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c5585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c558b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55705c55e770 con 0x55705c54dcd0 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55705c46df10 con 0x55705c54e5f0 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55705c55e8f0 con 0x55705c55b930 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa25d76740 10 client.0 _ll_register_callbacks cb 0x55705c1c5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa20f2f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa20f2f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38820/0 (socket says 172.21.15.38:38820) 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa20f2f640 1 -- 172.21.15.38:0/2376257336 learned_addr learned my addr 172.21.15.38:0/2376257336 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.636+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c5585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa21730640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c558b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 -- 172.21.15.38:0/2376257336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 msgr2=0x55705c558b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c558b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 -- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 msgr2=0x55705c5585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c5585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 -- 172.21.15.38:0/2376257336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55705c43fc80 con 0x55705c54dcd0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ffa100126e0 tx=0x7ffa10012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fd45291d9b9d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ff9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10012190 con 0x55705c54dcd0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ffa10004030 con 0x55705c54dcd0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1401b3d0 con 0x55705c54dcd0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffa10025050 con 0x55705c54dcd0 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa25d76740 10 client.18882 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55705c1c63f0 con 0x55705c54dcd0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.637+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1401b3d0 con 0x55705c54dcd0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7ffa10003730 con 0x55705c54dcd0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 10 client.18882 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 10 client.18882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 20 client.18882 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c56ba50 0x55705c56dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 4 client.18882 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55705c4bfa70 con 0x55705c56ba50 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c56ba50 0x55705c56dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ff9fbfff640 10 client.18882 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.638+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c56ba50 0x55705c56dea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a68ee6af455cf39f server_cookie=6be7ac82be405c26 in_seq=0 out_seq=0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2376257336 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55705c4bfa70 con 0x55705c56ba50 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ff9fbfff640 10 client.18882 handle_command_reply: tid=0 2024-04-22T00:12:30.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 client.18882 shutdown 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 msgr2=0x55705c55b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ffa100126e0 tx=0x7ffa10012ba0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 shutdown_connections 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c558b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c5585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c56ba50 0x55705c56dea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2376257336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c55b260 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ff9fbfff640 0 client.18882 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 >> 172.21.15.38:0/2376257336 conn(0x55705c548750 msgr2=0x55705c54a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 shutdown_connections 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.639+0000 7ffa25d76740 1 -- 172.21.15.38:0/2376257336 wait complete. 2024-04-22T00:12:30.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.640+0000 7ffa25d76740 20 client.18882 trim_cache size 0 max 16384 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 Processor -- start 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 -- start start 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c567670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1aca90 con 0x55705c54dcd0 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1ae5d0 con 0x55705c54e5f0 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa25d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1ad160 con 0x55705c55b930 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38836/0 (socket says 172.21.15.38:38836) 2024-04-22T00:12:30.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 -- 172.21.15.38:0/1545705831 learned_addr learned my addr 172.21.15.38:0/1545705831 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa21730640 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 -- 172.21.15.38:0/1545705831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 msgr2=0x55705c55ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 -- 172.21.15.38:0/1545705831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 msgr2=0x55705c567670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c567670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa20f2f640 1 -- 172.21.15.38:0/1545705831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55705c2907d0 con 0x55705c54dcd0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.653+0000 7ffa21730640 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ffa100096e0 tx=0x7ffa1001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab9d94373e80dbb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa067fc640 1 -- 172.21.15.38:0/1545705831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1002e070 con 0x55705c54dcd0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa067fc640 1 -- 172.21.15.38:0/1545705831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1000fe00 con 0x55705c54dcd0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa067fc640 1 -- 172.21.15.38:0/1545705831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10005070 con 0x55705c54dcd0 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 -- 172.21.15.38:0/1545705831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 msgr2=0x55705c558b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ffa100096e0 tx=0x7ffa1001c4e0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 -- 172.21.15.38:0/1545705831 shutdown_connections 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54e5f0 0x55705c567670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 --2- 172.21.15.38:0/1545705831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54dcd0 0x55705c558b10 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 -- 172.21.15.38:0/1545705831 >> 172.21.15.38:0/1545705831 conn(0x55705c548750 msgr2=0x55705c553c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 -- 172.21.15.38:0/1545705831 shutdown_connections 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.654+0000 7ffa25d76740 1 -- 172.21.15.38:0/1545705831 wait complete. 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 -- start start 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 Processor -- start 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54dcd0 0x55705c563ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1adf00 con 0x55705c54e5f0 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1ad830 con 0x55705c54dcd0 2024-04-22T00:12:30.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c1aaa90 con 0x55705c55b930 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa25d76740 10 client.0 _ll_register_callbacks cb 0x7ffa10009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa1bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38848/0 (socket says 172.21.15.38:38848) 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa1bfff640 1 -- 172.21.15.38:0/2447783553 learned_addr learned my addr 172.21.15.38:0/2447783553 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.655+0000 7ffa21730640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54dcd0 0x55705c563ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 -- 172.21.15.38:0/2447783553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 msgr2=0x55705c55f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 -- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54dcd0 msgr2=0x55705c563ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54dcd0 0x55705c563ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 -- 172.21.15.38:0/2447783553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55705c55ff90 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa1bfff640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009430 tx=0x7ffa08000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c2d7b04cb25fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0804b040 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ffa080ee760 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1000fe00 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffa080bcd20 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 10 client.18888 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55705c5d1f80 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1000fe00 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7ffa080d6230 con 0x55705c54e5f0 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 10 client.18888 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 10 client.18888 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-22T00:12:30.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 20 client.18888 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c5d2750 0x55705c5d4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 4 client.18888 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.656+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 --> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55705c4b9dc0 con 0x55705c5d2750 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c5d2750 0x55705c5d4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ff9fbfff640 10 client.18888 ms_handle_connect on v2:172.21.15.111:6836/782972998 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa20f2f640 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c5d2750 0x55705c5d4b70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b732bd18eb713d55 server_cookie=e3e631092ba58381 in_seq=0 out_seq=0 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ff9fbfff640 1 -- 172.21.15.38:0/2447783553 <== mds.3 v2:172.21.15.111:6836/782972998 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55705c4b9dc0 con 0x55705c5d2750 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ff9fbfff640 10 client.18888 handle_command_reply: tid=0 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 client.18888 shutdown 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 msgr2=0x55705c55f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009430 tx=0x7ffa08000bc0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 shutdown_connections 2024-04-22T00:12:30.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55705c55b930 0x55705c55f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:6836/782972998,v1:172.21.15.111:6837/782972998] conn(0x55705c5d2750 0x55705c5d4b70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55705c54e5f0 0x55705c55f280 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 --2- 172.21.15.38:0/2447783553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55705c54dcd0 0x55705c563ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ff9fbfff640 0 client.18888 ms_handle_reset on v2:172.21.15.111:6836/782972998 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 >> 172.21.15.38:0/2447783553 conn(0x55705c548750 msgr2=0x55705c553c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 shutdown_connections 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.657+0000 7ffa25d76740 1 -- 172.21.15.38:0/2447783553 wait complete. 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa25d76740 20 client.18888 trim_cache size 0 max 16384 2024-04-22T00:12:30.662 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 msgr2=0x7ffa0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 0x7ffa0c0a51c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffa14020000 tx=0x7ffa1401d000 comp rx=0 tx=0).stop 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 msgr2=0x7ffa1c1993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009230 tx=0x7ffa0802cba0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 shutdown_connections 2024-04-22T00:12:30.663 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ffa0c0a2d00 0x7ffa0c0a51c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa1c1066b0 0x7ffa1c1a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa1c100fd0 0x7ffa1c1993d0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 --2- 172.21.15.38:0/3843083649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa1c100610 0x7ffa1c198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 >> 172.21.15.38:0/3843083649 conn(0x7ffa1c077740 msgr2=0x7ffa1c0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 shutdown_connections 2024-04-22T00:12:30.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.658+0000 7ffa23193640 1 -- 172.21.15.38:0/3843083649 wait complete. 2024-04-22T00:12:30.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 -- 172.21.15.38:0/162335369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0324100610 msgr2=0x7f0324100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 --2- 172.21.15.38:0/162335369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0324100610 0x7f0324100a90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0310009930 tx=0x7f031002bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 -- 172.21.15.38:0/162335369 shutdown_connections 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 --2- 172.21.15.38:0/162335369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f032410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 --2- 172.21.15.38:0/162335369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0324100610 0x7f0324100a90 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 --2- 172.21.15.38:0/162335369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03241066b0 0x7f0324106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 -- 172.21.15.38:0/162335369 >> 172.21.15.38:0/162335369 conn(0x7f03240769c0 msgr2=0x7f0324076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 -- 172.21.15.38:0/162335369 shutdown_connections 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.773+0000 7f032b137640 1 -- 172.21.15.38:0/162335369 wait complete. 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 Processor -- start 2024-04-22T00:12:30.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 -- start start 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0324100610 0x7f0324197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f0324198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0324111b30 con 0x7f03241066b0 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03241119b0 con 0x7f0324100fd0 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f032b137640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0324111cb0 con 0x7f0324100610 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38872/0 (socket says 172.21.15.38:38872) 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f03296d4640 1 -- 172.21.15.38:0/88668075 learned_addr learned my addr 172.21.15.38:0/88668075 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f0328ed3640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0324100610 0x7f0324197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.774+0000 7f0323fff640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f0324198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 -- 172.21.15.38:0/88668075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0324100610 msgr2=0x7f0324197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0324100610 0x7f0324197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 -- 172.21.15.38:0/88668075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 msgr2=0x7f0324198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f0324198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 -- 172.21.15.38:0/88668075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032419cfc0 con 0x7f03241066b0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f0323fff640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f0324198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f03296d4640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f031c00eaa0 tx=0x7f031c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f0321ffb640 1 -- 172.21.15.38:0/88668075 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c00cdb0 con 0x7f03241066b0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032419d2b0 con 0x7f03241066b0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f0321ffb640 1 -- 172.21.15.38:0/88668075 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031c005020 con 0x7f03241066b0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f0321ffb640 1 -- 172.21.15.38:0/88668075 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c0116b0 con 0x7f03241066b0 2024-04-22T00:12:30.775 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.775+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03241a4b40 con 0x7f03241066b0 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.777+0000 7f0321ffb640 1 -- 172.21.15.38:0/88668075 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f031c005560 con 0x7f03241066b0 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 Processor -- start 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 -- start start 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f0321ffb640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 0x7f03140a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f0323fff640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 0x7f03140a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f0321ffb640 1 -- 172.21.15.38:0/88668075 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f031c0cfa30 con 0x7f03241066b0 2024-04-22T00:12:30.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e67400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e6ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bc53dcb530 con 0x55bc53e70520 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bc53d73830 con 0x55bc53e67020 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f032ddfb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bc53dc74f0 con 0x55bc53e67940 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38884/0 (socket says 172.21.15.38:38884) 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.778+0000 7f03296d4640 1 -- 172.21.15.38:0/1962280667 learned_addr learned my addr 172.21.15.38:0/1962280667 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f0328ed3640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e67400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f0323fff640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e6ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 -- 172.21.15.38:0/1962280667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 msgr2=0x55bc53e6ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e6ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 -- 172.21.15.38:0/1962280667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 msgr2=0x55bc53e67400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.783 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e67400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 -- 172.21.15.38:0/1962280667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc53d972c0 con 0x55bc53e70520 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f0323fff640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 0x7f03140a95d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0310036040 tx=0x7f031002c6d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f0328ed3640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e67400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f03296d4640 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f031c098020 tx=0x7f031c09b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1129f2736d5b918 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f030effd640 1 -- 172.21.15.38:0/1962280667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c01f070 con 0x55bc53e70520 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f030effd640 1 -- 172.21.15.38:0/1962280667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031c0cd070 con 0x55bc53e70520 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.779+0000 7f030effd640 1 -- 172.21.15.38:0/1962280667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c09aa80 con 0x55bc53e70520 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- 172.21.15.38:0/1962280667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 msgr2=0x55bc53e72910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f031c098020 tx=0x7f031c09b630 comp rx=0 tx=0).stop 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- 172.21.15.38:0/1962280667 shutdown_connections 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e72910 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e6ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- 172.21.15.38:0/1962280667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e67400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- 172.21.15.38:0/1962280667 >> 172.21.15.38:0/1962280667 conn(0x55bc53e61750 msgr2=0x55bc53e61b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- 172.21.15.38:0/1962280667 shutdown_connections 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- 172.21.15.38:0/1962280667 wait complete. 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 -- start start 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 Processor -- start 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e6ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.780+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bc53e73310 con 0x55bc53e67940 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55bc53d86f10 con 0x55bc53e67020 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bc53e73490 con 0x55bc53e70520 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 10 client.0 _ll_register_callbacks cb 0x55bc53adebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38896/0 (socket says 172.21.15.38:38896) 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 -- 172.21.15.38:0/674226473 learned_addr learned my addr 172.21.15.38:0/674226473 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0328ed3640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e6ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f03296d4640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 -- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 msgr2=0x55bc53e6da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 -- 172.21.15.38:0/674226473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 msgr2=0x55bc53e6ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e6ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 -- 172.21.15.38:0/674226473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc53d58c80 con 0x55bc53e67940 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f03296d4640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0323fff640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f031002c820 tx=0x7f031002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8366d9999d16c533 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0303fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031004d070 con 0x55bc53e67940 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f03100048c0 con 0x55bc53e67940 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f031c0cd070 con 0x55bc53e67940 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f0310059050 con 0x55bc53e67940 2024-04-22T00:12:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 10 client.18900 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.781+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bc53adf3f0 con 0x55bc53e67940 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031c0cd070 con 0x55bc53e67940 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f0310048070 con 0x55bc53e67940 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 10 client.18900 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 10 client.18900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 20 client.18900 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53e805f0 0x55bc53e82a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 4 client.18900 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc53dd8a70 con 0x55bc53e805f0 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.782+0000 7f0328ed3640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53e805f0 0x55bc53e82a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f0328ed3640 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53e805f0 0x55bc53e82a30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8bef5135e9f40f4 server_cookie=5564af62b4dbb9cf in_seq=0 out_seq=0 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f0303fff640 10 client.18900 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f0303fff640 1 -- 172.21.15.38:0/674226473 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc53dd8a70 con 0x55bc53e805f0 2024-04-22T00:12:30.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f0303fff640 10 client.18900 handle_command_reply: tid=0 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 client.18900 shutdown 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 msgr2=0x55bc53e6d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f031002c820 tx=0x7f031002bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 shutdown_connections 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53e805f0 0x55bc53e82a30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67940 0x55bc53e6d300 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 --2- 172.21.15.38:0/674226473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e6ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f0303fff640 0 client.18900 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.783+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 >> 172.21.15.38:0/674226473 conn(0x55bc53e61750 msgr2=0x55bc53e634a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.784+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 shutdown_connections 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.784+0000 7f032ddfb740 1 -- 172.21.15.38:0/674226473 wait complete. 2024-04-22T00:12:30.788 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.784+0000 7f032ddfb740 20 client.18900 trim_cache size 0 max 16384 2024-04-22T00:12:30.789 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:30 smithi203 ceph-mon[31431]: pgmap v323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:30.789 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:30 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1116291878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 Processor -- start 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 -- start start 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e7c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e88300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03241aca70 con 0x55bc53e70520 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ae5b0 con 0x55bc53e67020 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f032ddfb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ad140 con 0x55bc53e67940 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.797+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38906/0 (socket says 172.21.15.38:38906) 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 -- 172.21.15.38:0/3833497465 learned_addr learned my addr 172.21.15.38:0/3833497465 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f0323fff640 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e88300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 -- 172.21.15.38:0/3833497465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 msgr2=0x55bc53e88300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e88300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 -- 172.21.15.38:0/3833497465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 msgr2=0x55bc53e7c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e7c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 -- 172.21.15.38:0/3833497465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc53ba97d0 con 0x55bc53e70520 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f03296d4640 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f031c09b8c0 tx=0x7f031c09a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3becbb9f769c6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f030e7fc640 1 -- 172.21.15.38:0/3833497465 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c0b8020 con 0x55bc53e70520 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f030e7fc640 1 -- 172.21.15.38:0/3833497465 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031c098230 con 0x55bc53e70520 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.798+0000 7f030e7fc640 1 -- 172.21.15.38:0/3833497465 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c01f040 con 0x55bc53e70520 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- 172.21.15.38:0/3833497465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 msgr2=0x55bc53e88840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f031c09b8c0 tx=0x7f031c09a0d0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- 172.21.15.38:0/3833497465 shutdown_connections 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e70520 0x55bc53e88840 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e67940 0x55bc53e88300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- 172.21.15.38:0/3833497465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67020 0x55bc53e7c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- 172.21.15.38:0/3833497465 >> 172.21.15.38:0/3833497465 conn(0x55bc53e61750 msgr2=0x55bc53e60850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- 172.21.15.38:0/3833497465 shutdown_connections 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- 172.21.15.38:0/3833497465 wait complete. 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- start start 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 Processor -- start 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67940 0x55bc53e6a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f03241adee0 con 0x55bc53e67020 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f03241ad810 con 0x55bc53e67940 2024-04-22T00:12:30.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.799+0000 7f032ddfb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03241aaa70 con 0x55bc53e70520 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f032ddfb740 10 client.0 _ll_register_callbacks cb 0x7f0310009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38908/0 (socket says 172.21.15.38:38908) 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 -- 172.21.15.38:0/416964565 learned_addr learned my addr 172.21.15.38:0/416964565 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f03296d4640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 -- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 msgr2=0x55bc53e6ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 -- 172.21.15.38:0/416964565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67940 msgr2=0x55bc53e6a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0323fff640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67940 0x55bc53e6a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67940 0x55bc53e6a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 -- 172.21.15.38:0/416964565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc53ee5d70 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0328ed3640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f0318004700 tx=0x7f031800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdeeb926afc39c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0303fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031801c3b0 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f031801c550 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f031c098230 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f031802b050 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f032ddfb740 10 client.18906 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.800+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bc53ee7090 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f031c098230 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f0318010430 con 0x55bc53e67020 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 10 client.18906 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 10 client.18906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-22T00:12:30.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 20 client.18906 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53ee7810 0x55bc53ee9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 4 client.18906 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 --> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc53dd2dc0 con 0x55bc53ee7810 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.801+0000 7f0323fff640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53ee7810 0x55bc53ee9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f0303fff640 10 client.18906 ms_handle_connect on v2:172.21.15.203:6836/1809489419 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f0323fff640 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53ee7810 0x55bc53ee9c30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee8a3c6db0e7d78 server_cookie=c9e9956564fb7f71 in_seq=0 out_seq=0 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f0303fff640 1 -- 172.21.15.38:0/416964565 <== mds.2 v2:172.21.15.203:6836/1809489419 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc53dd2dc0 con 0x55bc53ee7810 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f0303fff640 10 client.18906 handle_command_reply: tid=0 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 client.18906 shutdown 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 msgr2=0x55bc53e6a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f0318004700 tx=0x7f031800bf00 comp rx=0 tx=0).stop 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 shutdown_connections 2024-04-22T00:12:30.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:6836/1809489419,v1:172.21.15.203:6837/1809489419] conn(0x55bc53ee7810 0x55bc53ee9c30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc53e70520 0x55bc53e6ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc53e67940 0x55bc53e6a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f0303fff640 0 client.18906 ms_handle_reset on v2:172.21.15.203:6836/1809489419 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 --2- 172.21.15.38:0/416964565 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bc53e67020 0x55bc53e6a190 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 >> 172.21.15.38:0/416964565 conn(0x55bc53e61750 msgr2=0x55bc53e67e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 shutdown_connections 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 1 -- 172.21.15.38:0/416964565 wait complete. 2024-04-22T00:12:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.802+0000 7f032ddfb740 20 client.18906 trim_cache size 0 max 16384 2024-04-22T00:12:30.810 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:12:30.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 msgr2=0x7f03140a95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 0x7f03140a95d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0310036040 tx=0x7f031002c6d0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 msgr2=0x7f032419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f031c00eaa0 tx=0x7f031c00ef60 comp rx=0 tx=0).stop 2024-04-22T00:12:30.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 shutdown_connections 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f03140a7110 0x7f03140a95d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03241066b0 0x7f032419c890 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0324100fd0 0x7f0324198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 --2- 172.21.15.38:0/88668075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0324100610 0x7f0324197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 >> 172.21.15.38:0/88668075 conn(0x7f03240769c0 msgr2=0x7f03240fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 shutdown_connections 2024-04-22T00:12:30.812 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.803+0000 7f032b137640 1 -- 172.21.15.38:0/88668075 wait complete. 2024-04-22T00:12:30.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T00:12:30.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:30 smithi111 ceph-mon[31543]: pgmap v323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T00:12:30.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:30 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1116291878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.919+0000 7f38b5572640 1 -- 172.21.15.38:0/3186936718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0069dc0 msgr2=0x7f38b006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.919+0000 7f38b5572640 1 --2- 172.21.15.38:0/3186936718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0069dc0 0x7f38b006a240 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f3898009930 tx=0x7f389802bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- 172.21.15.38:0/3186936718 shutdown_connections 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- 172.21.15.38:0/3186936718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 0x7f38b010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- 172.21.15.38:0/3186936718 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0069dc0 0x7f38b006a240 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- 172.21.15.38:0/3186936718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0107730 0x7f38b0107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- 172.21.15.38:0/3186936718 >> 172.21.15.38:0/3186936718 conn(0x7f38b00776a0 msgr2=0x7f38b0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- 172.21.15.38:0/3186936718 shutdown_connections 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- 172.21.15.38:0/3186936718 wait complete. 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 Processor -- start 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- start start 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0069dc0 0x7f38b0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 0x7f38b01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0111b40 con 0x7f38b0107730 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01119c0 con 0x7f38b0069dc0 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.920+0000 7f38b5572640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0111cc0 con 0x7f38b006a780 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38928/0 (socket says 172.21.15.38:38928) 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 -- 172.21.15.38:0/2380187361 learned_addr learned my addr 172.21.15.38:0/2380187361 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0069dc0 0x7f38b0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38a7fff640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 0x7f38b01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.921 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 -- 172.21.15.38:0/2380187361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 msgr2=0x7f38b01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 0x7f38b01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 -- 172.21.15.38:0/2380187361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0069dc0 msgr2=0x7f38b0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0069dc0 0x7f38b0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 -- 172.21.15.38:0/2380187361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b019d000 con 0x7f38b0107730 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38af7fe640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f38a800eaa0 tx=0x7f38a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38acff9640 1 -- 172.21.15.38:0/2380187361 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cdb0 con 0x7f38b0107730 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38acff9640 1 -- 172.21.15.38:0/2380187361 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a8005020 con 0x7f38b0107730 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b019d2f0 con 0x7f38b0107730 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38acff9640 1 -- 172.21.15.38:0/2380187361 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a80116b0 con 0x7f38b0107730 2024-04-22T00:12:30.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.921+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b01a4b80 con 0x7f38b0107730 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.923+0000 7f38acff9640 1 -- 172.21.15.38:0/2380187361 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f38a8005560 con 0x7f38b0107730 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 Processor -- start 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 -- start start 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c91e30 0x55d248c92210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248ca1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d248bf6530 con 0x55d248c927e0 2024-04-22T00:12:30.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d248b9e830 con 0x55d248c9f930 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38b8231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d248bf24f0 con 0x55d248c91e30 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38acff9640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 0x7f389c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38944/0 (socket says 172.21.15.38:38944) 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38aeffd640 1 -- 172.21.15.38:0/2812444783 learned_addr learned my addr 172.21.15.38:0/2812444783 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38a7fff640 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c91e30 0x55d248c92210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.924+0000 7f38af7fe640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 0x7f389c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38acff9640 1 -- 172.21.15.38:0/2380187361 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f38a80cf9e0 con 0x7f38b0107730 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38a7fff640 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248ca1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 -- 172.21.15.38:0/2812444783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c91e30 msgr2=0x55d248c92210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c91e30 0x55d248c92210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 -- 172.21.15.38:0/2812444783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 msgr2=0x55d248ca1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248ca1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 -- 172.21.15.38:0/2812444783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d248bc22c0 con 0x55d248c927e0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f38a000aba0 tx=0x7f38a000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c7c687c45a181b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38af7fe640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 0x7f389c0a51c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a809bb50 tx=0x7f38a80054d0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38a57fa640 1 -- 172.21.15.38:0/2812444783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a001d070 con 0x55d248c927e0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38a57fa640 1 -- 172.21.15.38:0/2812444783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a0018b20 con 0x55d248c927e0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.925+0000 7f38a57fa640 1 -- 172.21.15.38:0/2812444783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0010b20 con 0x55d248c927e0 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- 172.21.15.38:0/2812444783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 msgr2=0x55d248c9f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f38a000aba0 tx=0x7f38a000f790 comp rx=0 tx=0).stop 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- 172.21.15.38:0/2812444783 shutdown_connections 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248ca1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c9f3f0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 --2- 172.21.15.38:0/2812444783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c91e30 0x55d248c92210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- 172.21.15.38:0/2812444783 >> 172.21.15.38:0/2812444783 conn(0x55d248c8c750 msgr2=0x55d248c8cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- 172.21.15.38:0/2812444783 shutdown_connections 2024-04-22T00:12:30.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- 172.21.15.38:0/2812444783 wait complete. 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 -- start start 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.926+0000 7f38b8231740 1 Processor -- start 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248c9f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c9c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d248ca26f0 con 0x55d248c9f930 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x55d248bb1f10 con 0x55d248c91e30 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d248ca2870 con 0x55d248c927e0 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38b8231740 10 client.0 _ll_register_callbacks cb 0x55d248909be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38946/0 (socket says 172.21.15.38:38946) 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 -- 172.21.15.38:0/3942738742 learned_addr learned my addr 172.21.15.38:0/3942738742 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38aeffd640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c9c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248c9f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 -- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 msgr2=0x55d248c9c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c9c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 -- 172.21.15.38:0/3942738742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 msgr2=0x55d248c9f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248c9f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 -- 172.21.15.38:0/3942738742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d248b83c80 con 0x55d248c9f930 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38a7fff640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f389802c820 tx=0x7f3898009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616c53c4182f2ed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38967fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.927+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898004f50 con 0x55d248c9f930 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f38980050f0 con 0x55d248c9f930 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38a0018b20 con 0x55d248c9f930 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f38980052f0 con 0x55d248c9f930 2024-04-22T00:12:30.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 10 client.18918 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d24890a3f0 con 0x55d248c9f930 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a0018b20 con 0x55d248c9f930 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f389804c020 con 0x55d248c9f930 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 10 client.18918 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 10 client.18918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 20 client.18918 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248caf940 0x55d248cb1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 4 client.18918 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d248c03a70 con 0x55d248caf940 2024-04-22T00:12:30.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.928+0000 7f38aeffd640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248caf940 0x55d248cb1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.929+0000 7f38967fc640 10 client.18918 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.929+0000 7f38aeffd640 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248caf940 0x55d248cb1d30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c6836bd2a233a7b server_cookie=f5ba81ce7859ff62 in_seq=0 out_seq=0 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.929+0000 7f38967fc640 1 -- 172.21.15.38:0/3942738742 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d248c03a70 con 0x55d248caf940 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.929+0000 7f38967fc640 10 client.18918 handle_command_reply: tid=0 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.929+0000 7f38b8231740 1 client.18918 shutdown 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 msgr2=0x55d248c9cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f389802c820 tx=0x7f3898009260 comp rx=0 tx=0).stop 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 shutdown_connections 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248caf940 0x55d248cb1d30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c9f930 0x55d248c9cb20 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c9c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38967fc640 0 client.18918 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 --2- 172.21.15.38:0/3942738742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248c9f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 >> 172.21.15.38:0/3942738742 conn(0x55d248c8c750 msgr2=0x55d248c8e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 shutdown_connections 2024-04-22T00:12:30.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 1 -- 172.21.15.38:0/3942738742 wait complete. 2024-04-22T00:12:30.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.930+0000 7f38b8231740 20 client.18918 trim_cache size 0 max 16384 2024-04-22T00:12:30.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 Processor -- start 2024-04-22T00:12:30.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 -- start start 2024-04-22T00:12:30.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248cab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 0x55d248c9ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01acaa0 con 0x55d248c927e0 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01ae5e0 con 0x55d248c91e30 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.943+0000 7f38b8231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01ad170 con 0x55d248c9f930 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38956/0 (socket says 172.21.15.38:38956) 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 -- 172.21.15.38:0/2604755971 learned_addr learned my addr 172.21.15.38:0/2604755971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 -- 172.21.15.38:0/2604755971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 msgr2=0x55d248c9ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38a7fff640 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 0x55d248c9ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 0x55d248c9ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 -- 172.21.15.38:0/2604755971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 msgr2=0x55d248cab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248cab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 -- 172.21.15.38:0/2604755971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d248cb20a0 con 0x55d248c927e0 2024-04-22T00:12:30.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38a7fff640 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 0x55d248c9ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38aeffd640 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f38a000aba0 tx=0x7f38a000aad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a35bd307359d43d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38a4ff9640 1 -- 172.21.15.38:0/2604755971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a001d040 con 0x55d248c927e0 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38a4ff9640 1 -- 172.21.15.38:0/2604755971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a001e040 con 0x55d248c927e0 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.944+0000 7f38a4ff9640 1 -- 172.21.15.38:0/2604755971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00291d0 con 0x55d248c927e0 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- 172.21.15.38:0/2604755971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 msgr2=0x55d248c97aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f38a000aba0 tx=0x7f38a000aad0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- 172.21.15.38:0/2604755971 shutdown_connections 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c9f930 0x55d248c9ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c927e0 0x55d248c97aa0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 --2- 172.21.15.38:0/2604755971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c91e30 0x55d248cab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- 172.21.15.38:0/2604755971 >> 172.21.15.38:0/2604755971 conn(0x55d248c8c750 msgr2=0x55d248c97390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- 172.21.15.38:0/2604755971 shutdown_connections 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- 172.21.15.38:0/2604755971 wait complete. 2024-04-22T00:12:30.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 -- start start 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 Processor -- start 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.945+0000 7f38b8231740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c99770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248c99cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01adf10 con 0x55d248c91e30 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01ad840 con 0x55d248c9f930 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01aaaa0 con 0x55d248c927e0 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 10 client.0 _ll_register_callbacks cb 0x7f3898005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38966/0 (socket says 172.21.15.38:38966) 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 -- 172.21.15.38:0/420221712 learned_addr learned my addr 172.21.15.38:0/420221712 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38aeffd640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c99770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 -- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 msgr2=0x55d248c99770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c99770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 -- 172.21.15.38:0/420221712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 msgr2=0x55d248c99cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248c99cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 -- 172.21.15.38:0/420221712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2489d47d0 con 0x55d248c91e30 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38a7fff640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x55d248c91a00 tx=0x7f389802ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a20b522ca26a82a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38967fc640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389805d060 con 0x55d248c91e30 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3898004c60 con 0x55d248c91e30 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38a001e040 con 0x55d248c91e30 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f3898064050 con 0x55d248c91e30 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 10 client.18924 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.946+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55d248d15d60 con 0x55d248c91e30 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a001e040 con 0x55d248c91e30 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f389804c040 con 0x55d248c91e30 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 10 client.18924 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 10 client.18924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 20 client.18924 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248d164e0 0x55d248d18900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 4 client.18924 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 --> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d248c9ceb0 con 0x55d248d164e0 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.947+0000 7f38aeffd640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248d164e0 0x55d248d18900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38aeffd640 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248d164e0 0x55d248d18900 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82878b2f17699a05 server_cookie=4803004f5a07d44c in_seq=0 out_seq=0 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38967fc640 10 client.18924 ms_handle_connect on v2:172.21.15.203:6838/3602809941 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38967fc640 1 -- 172.21.15.38:0/420221712 <== mds.1 v2:172.21.15.203:6838/3602809941 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d248c9ceb0 con 0x55d248d164e0 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38967fc640 10 client.18924 handle_command_reply: tid=0 2024-04-22T00:12:30.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 client.18924 shutdown 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 msgr2=0x55d248c99230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x55d248c91a00 tx=0x7f389802ce00 comp rx=0 tx=0).stop 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 shutdown_connections 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:6838/3602809941,v1:172.21.15.203:6839/3602809941] conn(0x55d248d164e0 0x55d248d18900 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d248c9f930 0x55d248c99cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38967fc640 0 client.18924 ms_handle_reset on v2:172.21.15.203:6838/3602809941 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d248c927e0 0x55d248c99770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 --2- 172.21.15.38:0/420221712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d248c91e30 0x55d248c99230 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 >> 172.21.15.38:0/420221712 conn(0x55d248c8c750 msgr2=0x55d248c8b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 shutdown_connections 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 1 -- 172.21.15.38:0/420221712 wait complete. 2024-04-22T00:12:30.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b8231740 20 client.18924 trim_cache size 0 max 16384 2024-04-22T00:12:30.953 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:12:30.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 msgr2=0x7f389c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 0x7f389c0a51c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a809bb50 tx=0x7f38a80054d0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 msgr2=0x7f38b019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.948+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f38a800eaa0 tx=0x7f38a800ef60 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 shutdown_connections 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f389c0a2d00 0x7f389c0a51c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38b0107730 0x7f38b019c8d0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38b006a780 0x7f38b01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 --2- 172.21.15.38:0/2380187361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b0069dc0 0x7f38b0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 >> 172.21.15.38:0/2380187361 conn(0x7f38b00776a0 msgr2=0x7f38b00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 shutdown_connections 2024-04-22T00:12:30.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:30.949+0000 7f38b5572640 1 -- 172.21.15.38:0/2380187361 wait complete. 2024-04-22T00:12:30.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T00:12:31.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 -- 172.21.15.38:0/207982074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 msgr2=0x7f7aac107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 --2- 172.21.15.38:0/207982074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac107b00 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009930 tx=0x7f7aa802bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:31.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 -- 172.21.15.38:0/207982074 shutdown_connections 2024-04-22T00:12:31.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 --2- 172.21.15.38:0/207982074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac1020d0 0x7f7aac10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 --2- 172.21.15.38:0/207982074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac101710 0x7f7aac101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 --2- 172.21.15.38:0/207982074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac107b00 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 -- 172.21.15.38:0/207982074 >> 172.21.15.38:0/207982074 conn(0x7f7aac0fd180 msgr2=0x7f7aac0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 -- 172.21.15.38:0/207982074 shutdown_connections 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.064+0000 7f7ab442e640 1 -- 172.21.15.38:0/207982074 wait complete. 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 Processor -- start 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 -- start start 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac101710 0x7f7aac198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac1020d0 0x7f7aac1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac111b30 con 0x7f7aac107720 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1119b0 con 0x7f7aac101710 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab442e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac111cb0 con 0x7f7aac1020d0 2024-04-22T00:12:31.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38982/0 (socket says 172.21.15.38:38982) 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 -- 172.21.15.38:0/553051416 learned_addr learned my addr 172.21.15.38:0/553051416 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac1020d0 0x7f7aac1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac101710 0x7f7aac198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 -- 172.21.15.38:0/553051416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac1020d0 msgr2=0x7f7aac1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac1020d0 0x7f7aac1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 -- 172.21.15.38:0/553051416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac101710 msgr2=0x7f7aac198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.065+0000 7f7ab29cb640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac101710 0x7f7aac198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7ab29cb640 1 -- 172.21.15.38:0/553051416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aac19d060 con 0x7f7aac107720 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7ab29cb640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400eaa0 tx=0x7f7aa400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7aa37fe640 1 -- 172.21.15.38:0/553051416 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa400cdb0 con 0x7f7aac107720 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7aa37fe640 1 -- 172.21.15.38:0/553051416 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4005020 con 0x7f7aac107720 2024-04-22T00:12:31.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7aa37fe640 1 -- 172.21.15.38:0/553051416 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40116b0 con 0x7f7aac107720 2024-04-22T00:12:31.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aac19d350 con 0x7f7aac107720 2024-04-22T00:12:31.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.066+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aac1a4be0 con 0x7f7aac107720 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7aa37fe640 1 -- 172.21.15.38:0/553051416 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7aa4005560 con 0x7f7aac107720 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 Processor -- start 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 -- start start 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae549130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae558d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558eae4ad530 con 0x558eae549670 2024-04-22T00:12:31.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558eae455830 con 0x558eae548d50 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.069+0000 7f7ab7014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558eae4a94f0 con 0x558eae556930 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38990/0 (socket says 172.21.15.38:38990) 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 -- 172.21.15.38:0/3080065614 learned_addr learned my addr 172.21.15.38:0/3080065614 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7aa37fe640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 0x7f7a980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae549130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae558d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 -- 172.21.15.38:0/3080065614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 msgr2=0x558eae558d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae558d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 -- 172.21.15.38:0/3080065614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 msgr2=0x558eae549130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae549130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 -- 172.21.15.38:0/3080065614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eae4792c0 con 0x558eae549670 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae558d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:31.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab29cb640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 0x7f7a980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7aa37fe640 1 -- 172.21.15.38:0/553051416 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f7aa40cfaa0 con 0x7f7aac107720 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009930 tx=0x7f7aa8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168f1e5bc1e83363 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7aa0bfe640 1 -- 172.21.15.38:0/3080065614 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa802ca80 con 0x558eae549670 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7aa0bfe640 1 -- 172.21.15.38:0/3080065614 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa802cc20 con 0x558eae549670 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.070+0000 7f7aa0bfe640 1 -- 172.21.15.38:0/3080065614 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8004920 con 0x558eae549670 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 -- 172.21.15.38:0/3080065614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 msgr2=0x558eae5563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab29cb640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 0x7f7a980a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7aa409b930 tx=0x7f7aa401eb40 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009930 tx=0x7f7aa8005d50 comp rx=0 tx=0).stop 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 -- 172.21.15.38:0/3080065614 shutdown_connections 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae558d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae5563f0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 --2- 172.21.15.38:0/3080065614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae549130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 -- 172.21.15.38:0/3080065614 >> 172.21.15.38:0/3080065614 conn(0x558eae543750 msgr2=0x558eae543b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 -- 172.21.15.38:0/3080065614 shutdown_connections 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.071+0000 7f7ab7014740 1 -- 172.21.15.38:0/3080065614 wait complete. 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 -- start start 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 Processor -- start 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae548d50 0x558eae556290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 0x558eae553b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558eae559690 con 0x558eae549670 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x558eae468f10 con 0x558eae556930 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558eae559810 con 0x558eae548d50 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab7014740 10 client.0 _ll_register_callbacks cb 0x558eae1c0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38996/0 (socket says 172.21.15.38:38996) 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 -- 172.21.15.38:0/819258318 learned_addr learned my addr 172.21.15.38:0/819258318 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae548d50 0x558eae556290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 0x558eae553b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 -- 172.21.15.38:0/819258318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae548d50 msgr2=0x558eae556290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae548d50 0x558eae556290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 -- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 msgr2=0x558eae553b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 0x558eae553b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 -- 172.21.15.38:0/819258318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eae43ac80 con 0x558eae549670 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 0x558eae553b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005ec0 tx=0x7f7aa802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c60125fbd635d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.072+0000 7f7a95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa80037f0 con 0x558eae549670 2024-04-22T00:12:31.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f7aa8003990 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa802cc20 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7aa8003b90 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 10 client.18936 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558eae1c13f0 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa802cc20 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7aa803d420 con 0x558eae549670 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 10 client.18936 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 10 client.18936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 20 client.18936 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae566730 0x558eae568b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 4 client.18936 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.073+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eae4baa70 con 0x558eae566730 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.074+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae566730 0x558eae568b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.074+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae566730 0x558eae568b20 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15a76a34edf5cff server_cookie=1042b809d776a339 in_seq=0 out_seq=0 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.074+0000 7f7a95ffb640 10 client.18936 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7a95ffb640 1 -- 172.21.15.38:0/819258318 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eae4baa70 con 0x558eae566730 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7a95ffb640 10 client.18936 handle_command_reply: tid=0 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 client.18936 shutdown 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 msgr2=0x558eae553620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005ec0 tx=0x7f7aa802bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 shutdown_connections 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae556930 0x558eae553b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae566730 0x558eae568b20 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae549670 0x558eae553620 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 --2- 172.21.15.38:0/819258318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae548d50 0x558eae556290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7a95ffb640 0 client.18936 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 >> 172.21.15.38:0/819258318 conn(0x558eae543750 msgr2=0x558eae5454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 shutdown_connections 2024-04-22T00:12:31.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 1 -- 172.21.15.38:0/819258318 wait complete. 2024-04-22T00:12:31.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.075+0000 7f7ab7014740 20 client.18936 trim_cache size 0 max 16384 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 Processor -- start 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 -- start start 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae562590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae549670 0x558eae555d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ac9a0 con 0x558eae556930 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ae4e0 con 0x558eae548d50 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.088+0000 7f7ab7014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ad070 con 0x558eae549670 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39002/0 (socket says 172.21.15.38:39002) 2024-04-22T00:12:31.092 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 -- 172.21.15.38:0/46584177 learned_addr learned my addr 172.21.15.38:0/46584177 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae549670 0x558eae555d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 -- 172.21.15.38:0/46584177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae549670 msgr2=0x558eae555d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae549670 0x558eae555d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 -- 172.21.15.38:0/46584177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 msgr2=0x558eae562590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae562590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 -- 172.21.15.38:0/46584177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eae28b7d0 con 0x558eae556930 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0ff810 tx=0x7f7a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8230a6528a1128 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7a97fff640 1 -- 172.21.15.38:0/46584177 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c026070 con 0x558eae556930 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7a97fff640 1 -- 172.21.15.38:0/46584177 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c0200f0 con 0x558eae556930 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.089+0000 7f7a97fff640 1 -- 172.21.15.38:0/46584177 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c003850 con 0x558eae556930 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- 172.21.15.38:0/46584177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 msgr2=0x558eae556290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0ff810 tx=0x7f7a9c0046c0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- 172.21.15.38:0/46584177 shutdown_connections 2024-04-22T00:12:31.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae556930 0x558eae556290 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae549670 0x558eae555d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- 172.21.15.38:0/46584177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae548d50 0x558eae562590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- 172.21.15.38:0/46584177 >> 172.21.15.38:0/46584177 conn(0x558eae543750 msgr2=0x558eae54ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- 172.21.15.38:0/46584177 shutdown_connections 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- 172.21.15.38:0/46584177 wait complete. 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- start start 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 Processor -- start 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae549670 0x558eae55a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae55ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ade10 con 0x558eae548d50 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1ad740 con 0x558eae549670 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.090+0000 7f7ab7014740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac1aa9a0 con 0x558eae556930 2024-04-22T00:12:31.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab7014740 10 client.0 _ll_register_callbacks cb 0x7f7aa8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39008/0 (socket says 172.21.15.38:39008) 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 -- 172.21.15.38:0/2777230754 learned_addr learned my addr 172.21.15.38:0/2777230754 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae549670 0x558eae55a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 -- 172.21.15.38:0/2777230754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 msgr2=0x558eae55ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae55ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 -- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae549670 msgr2=0x558eae55a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae549670 0x558eae55a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 -- 172.21.15.38:0/2777230754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eae5cb9c0 con 0x558eae548d50 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab19c9640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c006010 tx=0x7f7a9c020330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce89b9d42e1b50e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7a95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c030060 con 0x558eae548d50 2024-04-22T00:12:31.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f7a9c004050 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a9c0200f0 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7a9c037050 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab7014740 10 client.18942 fetch_fsmap learned FSMap version 42 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.091+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558eae5ccb80 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c0200f0 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f7a9c004a30 con 0x558eae548d50 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 10 client.18942 fetch_fsmap finished waiting for FSMap version 42 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 10 client.18942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T00:12:31.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 20 client.18942 populate_metadata read hostname 'smithi038' 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae5cd170 0x558eae5cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 4 client.18942 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 --> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558eae4b4dc0 con 0x558eae5cd170 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae5cd170 0x558eae5cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7ab21ca640 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae5cd170 0x558eae5cf590 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=781ab805e3e6072d server_cookie=e7ffee3f86e91cf7 in_seq=0 out_seq=0 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.092+0000 7f7a95ffb640 10 client.18942 ms_handle_connect on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:31.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7a95ffb640 1 -- 172.21.15.38:0/2777230754 <== mds.0 v2:172.21.15.111:6838/3890542559 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558eae4b4dc0 con 0x558eae5cd170 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7a95ffb640 10 client.18942 handle_command_reply: tid=0 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 client.18942 shutdown 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 msgr2=0x558eae55f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c006010 tx=0x7f7a9c020330 comp rx=0 tx=0).stop 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 shutdown_connections 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eae556930 0x558eae55ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558eae549670 0x558eae55a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.111:6838/3890542559,v1:172.21.15.111:6839/3890542559] conn(0x558eae5cd170 0x558eae5cf590 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 --2- 172.21.15.38:0/2777230754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558eae548d50 0x558eae55f030 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 >> 172.21.15.38:0/2777230754 conn(0x558eae543750 msgr2=0x558eae544af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7a95ffb640 0 client.18942 ms_handle_reset on v2:172.21.15.111:6838/3890542559 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 shutdown_connections 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 1 -- 172.21.15.38:0/2777230754 wait complete. 2024-04-22T00:12:31.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab7014740 20 client.18942 trim_cache size 0 max 16384 2024-04-22T00:12:31.100 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2024-04-22T00:12:31.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 msgr2=0x7f7a980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 0x7f7a980a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7aa409b930 tx=0x7f7aa401eb40 comp rx=0 tx=0).stop 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 msgr2=0x7f7aac19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400eaa0 tx=0x7f7aa400ef60 comp rx=0 tx=0).stop 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 shutdown_connections 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7a980a2d00 0x7f7a980a51c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7aac107720 0x7f7aac19c930 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aac1020d0 0x7f7aac1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 --2- 172.21.15.38:0/553051416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7aac101710 0x7f7aac198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:31.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 >> 172.21.15.38:0/553051416 conn(0x7f7aac0fd180 msgr2=0x7f7aac0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:31.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 shutdown_connections 2024-04-22T00:12:31.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:31.093+0000 7f7ab442e640 1 -- 172.21.15.38:0/553051416 wait complete. 2024-04-22T00:12:31.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:31 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:12:31.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:31 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:12:31.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:31 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T00:12:32.105 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-22T00:12:32.106 INFO:tasks.fwd_scrub:done joining 2024-04-22T00:12:32.106 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T00:12:32.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:12:32.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.221+0000 7fd5c4882640 1 -- 172.21.15.38:0/2087617787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 msgr2=0x7fd5c010e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.221 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.221+0000 7fd5c4882640 1 --2- 172.21.15.38:0/2087617787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c010e6b0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009ca0 tx=0x7fd5b802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 -- 172.21.15.38:0/2087617787 shutdown_connections 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 --2- 172.21.15.38:0/2087617787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c010e6b0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 --2- 172.21.15.38:0/2087617787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100cb0 0x7fd5c01063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 --2- 172.21.15.38:0/2087617787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c0100300 0x7fd5c01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 -- 172.21.15.38:0/2087617787 >> 172.21.15.38:0/2087617787 conn(0x7fd5c0075920 msgr2=0x7fd5c0075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 -- 172.21.15.38:0/2087617787 shutdown_connections 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 -- 172.21.15.38:0/2087617787 wait complete. 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 Processor -- start 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.222+0000 7fd5c4882640 1 -- start start 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100300 0x7fd5c0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c019c8f0 0x7fd5c019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0111b90 con 0x7fd5c0106ac0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0111a10 con 0x7fd5c019c8f0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0111d10 con 0x7fd5c0100300 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39022/0 (socket says 172.21.15.38:39022) 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 -- 172.21.15.38:0/627186997 learned_addr learned my addr 172.21.15.38:0/627186997 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5be59c640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100300 0x7fd5c0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bed9d640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c019c8f0 0x7fd5c019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 -- 172.21.15.38:0/627186997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100300 msgr2=0x7fd5c0198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100300 0x7fd5c0198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 -- 172.21.15.38:0/627186997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c019c8f0 msgr2=0x7fd5c019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c019c8f0 0x7fd5c019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 -- 172.21.15.38:0/627186997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c019d3e0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5bdd9b640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400d770 tx=0x7fd5b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4017070 con 0x7fd5c0106ac0 2024-04-22T00:12:32.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b4009c30 con 0x7fd5c0106ac0 2024-04-22T00:12:32.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c019d6d0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b400f620 con 0x7fd5c0106ac0 2024-04-22T00:12:32.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.223+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c01a4ec0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.224+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c00689b0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.225+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd5b401a070 con 0x7fd5c0106ac0 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.226+0000 7fd5af7fe640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 0x7fd5a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.226+0000 7fd5be59c640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 0x7fd5a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.226+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd5b40cf2c0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.227+0000 7fd5be59c640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 0x7fd5a80a51c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009bd0 tx=0x7fd5b0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.231+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd5b4009de0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.419+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5c0109c70 con 0x7fd5c0106ac0 2024-04-22T00:12:32.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.421+0000 7fd5af7fe640 1 -- 172.21.15.38:0/627186997 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25197 (secure 0 0 0) 0x7fd5b409baf0 con 0x7fd5c0106ac0 2024-04-22T00:12:32.419 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:12:32.419 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":100,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:12:21.417761+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:12:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 msgr2=0x7fd5a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 0x7fd5a80a51c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009bd0 tx=0x7fd5b0005f90 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 msgr2=0x7fd5c0198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400d770 tx=0x7fd5b400dc30 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 shutdown_connections 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd5a80a2d00 0x7fd5a80a51c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5c019c8f0 0x7fd5c019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5c0106ac0 0x7fd5c0198570 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 --2- 172.21.15.38:0/627186997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5c0100300 0x7fd5c0198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 >> 172.21.15.38:0/627186997 conn(0x7fd5c0075920 msgr2=0x7fd5c00fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 shutdown_connections 2024-04-22T00:12:32.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.423+0000 7fd5c4882640 1 -- 172.21.15.38:0/627186997 wait complete. 2024-04-22T00:12:32.432 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:12:32.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:12:32.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 -- 172.21.15.38:0/2814530834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df01066b0 msgr2=0x7f5df0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2814530834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df01066b0 0x7f5df0106a90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009930 tx=0x7f5de002bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 -- 172.21.15.38:0/2814530834 shutdown_connections 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2814530834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df0100fd0 0x7f5df010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2814530834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.542+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2814530834 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df01066b0 0x7f5df0106a90 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.543+0000 7f5df5e08640 1 -- 172.21.15.38:0/2814530834 >> 172.21.15.38:0/2814530834 conn(0x7f5df0077740 msgr2=0x7f5df0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.543+0000 7f5df5e08640 1 -- 172.21.15.38:0/2814530834 shutdown_connections 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.543+0000 7f5df5e08640 1 -- 172.21.15.38:0/2814530834 wait complete. 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.543+0000 7f5df5e08640 1 Processor -- start 2024-04-22T00:12:32.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.543+0000 7f5df5e08640 1 -- start start 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 0x7f5df01a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0110a40 con 0x7f5df0100fd0 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01108c0 con 0x7f5df0100610 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0110bc0 con 0x7f5df01066b0 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39038/0 (socket says 172.21.15.38:39038) 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 -- 172.21.15.38:0/2450740419 learned_addr learned my addr 172.21.15.38:0/2450740419 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5def7fe640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deffff640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 0x7f5df01a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 -- 172.21.15.38:0/2450740419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 msgr2=0x7f5df01a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 0x7f5df01a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 -- 172.21.15.38:0/2450740419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 msgr2=0x7f5df0198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 -- 172.21.15.38:0/2450740419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df01a4400 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5def7fe640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deffff640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 0x7f5df01a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5deeffd640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e9f0 tx=0x7f5dd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd800cd80 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd8010910 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df01a45d0 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.544+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8018990 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.545+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df01a4af0 con 0x7f5df0100fd0 2024-04-22T00:12:32.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.546+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df0068560 con 0x7f5df0100fd0 2024-04-22T00:12:32.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.547+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5dd8010ac0 con 0x7f5df0100fd0 2024-04-22T00:12:32.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.548+0000 7f5decff9640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 0x7f5ddc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.548+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f5dd80cff70 con 0x7f5df0100fd0 2024-04-22T00:12:32.546 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.548+0000 7f5def7fe640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 0x7f5ddc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.548+0000 7f5def7fe640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 0x7f5ddc0a51c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5de002c170 tx=0x7f5de002df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.548 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.551+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5dd809c800 con 0x7f5df0100fd0 2024-04-22T00:12:32.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:32 smithi111 ceph-mon[31543]: pgmap v324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:32.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:32 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/627186997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:12:32.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:32 smithi038 ceph-mon[27972]: pgmap v324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:32.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:32 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/627186997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:12:32.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.736+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5df002d170 con 0x7f5df0100fd0 2024-04-22T00:12:32.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.737+0000 7f5decff9640 1 -- 172.21.15.38:0/2450740419 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f5dd809c100 con 0x7f5df0100fd0 2024-04-22T00:12:32.734 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:12:32.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 msgr2=0x7f5ddc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 0x7f5ddc0a51c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5de002c170 tx=0x7f5de002df90 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 msgr2=0x7f5df0199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800e9f0 tx=0x7f5dd800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 shutdown_connections 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f5ddc0a2d00 0x7f5ddc0a51c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5df01066b0 0x7f5df01a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5df0100fd0 0x7f5df0199380 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 --2- 172.21.15.38:0/2450740419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5df0100610 0x7f5df0198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 >> 172.21.15.38:0/2450740419 conn(0x7f5df0077740 msgr2=0x7f5df00ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 shutdown_connections 2024-04-22T00:12:32.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.739+0000 7f5df5e08640 1 -- 172.21.15.38:0/2450740419 wait complete. 2024-04-22T00:12:32.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:12:32.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:32 smithi203 ceph-mon[31431]: pgmap v324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:32.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:32 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/627186997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 -- 172.21.15.38:0/194502015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc107720 msgr2=0x7fd0cc107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/194502015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc107720 0x7fd0cc107b00 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009930 tx=0x7fd0bc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 -- 172.21.15.38:0/194502015 shutdown_connections 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/194502015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 0x7fd0cc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/194502015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc101710 0x7fd0cc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/194502015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc107720 0x7fd0cc107b00 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 -- 172.21.15.38:0/194502015 >> 172.21.15.38:0/194502015 conn(0x7fd0cc0fd180 msgr2=0x7fd0cc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 -- 172.21.15.38:0/194502015 shutdown_connections 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.862+0000 7fd0d0d77640 1 -- 172.21.15.38:0/194502015 wait complete. 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 Processor -- start 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 -- start start 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 0x7fd0cc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc107720 0x7fd0cc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc111b30 con 0x7fd0cc101710 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1119b0 con 0x7fd0cc1020d0 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0d0d77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc111cb0 con 0x7fd0cc107720 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39058/0 (socket says 172.21.15.38:39058) 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 -- 172.21.15.38:0/1956809201 learned_addr learned my addr 172.21.15.38:0/1956809201 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0cad9d640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc107720 0x7fd0cc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0c3fff640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 0x7fd0cc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 -- 172.21.15.38:0/1956809201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc107720 msgr2=0x7fd0cc19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc107720 0x7fd0cc19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 -- 172.21.15.38:0/1956809201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 msgr2=0x7fd0cc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 0x7fd0cc198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.863+0000 7fd0ca59c640 1 -- 172.21.15.38:0/1956809201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc19d040 con 0x7fd0cc101710 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0ca59c640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009230 tx=0x7fd0bc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc041070 con 0x7fd0cc101710 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0bc004450 con 0x7fd0cc101710 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc003e50 con 0x7fd0cc101710 2024-04-22T00:12:32.864 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc19d2d0 con 0x7fd0cc101710 2024-04-22T00:12:32.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.864+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc1a4bc0 con 0x7fd0cc101710 2024-04-22T00:12:32.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.865+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc0689b0 con 0x7fd0cc101710 2024-04-22T00:12:32.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.866+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fd0bc044070 con 0x7fd0cc101710 2024-04-22T00:12:32.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.867+0000 7fd0c37fe640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 0x7fd0b80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:32.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.867+0000 7fd0c3fff640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 0x7fd0b80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:32.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.867+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd0bc0f1360 con 0x7fd0cc101710 2024-04-22T00:12:32.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.867+0000 7fd0c3fff640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 0x7fd0b80a53e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0ff870 tx=0x7fd0b4005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:32.867 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:32.870+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd0bc0bde20 con 0x7fd0cc101710 2024-04-22T00:12:33.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.057+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd0cc109cb0 con 0x7fd0cc101710 2024-04-22T00:12:33.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.058+0000 7fd0c37fe640 1 -- 172.21.15.38:0/1956809201 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) ==== 104+0+11 (secure 0 0 0) 0x7fd0bc0bd720 con 0x7fd0cc101710 2024-04-22T00:12:33.056 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:12:33.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 msgr2=0x7fd0b80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 0x7fd0b80a53e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0ff870 tx=0x7fd0b4005f90 comp rx=0 tx=0).stop 2024-04-22T00:12:33.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 msgr2=0x7fd0cc198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.058 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009230 tx=0x7fd0bc02cb30 comp rx=0 tx=0).stop 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 shutdown_connections 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.060+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fd0b80a2f20 0x7fd0b80a53e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0cc107720 0x7fd0cc19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0cc1020d0 0x7fd0cc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 --2- 172.21.15.38:0/1956809201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101710 0x7fd0cc198050 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 >> 172.21.15.38:0/1956809201 conn(0x7fd0cc0fd180 msgr2=0x7fd0cc0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 shutdown_connections 2024-04-22T00:12:33.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.061+0000 7fd0d0d77640 1 -- 172.21.15.38:0/1956809201 wait complete. 2024-04-22T00:12:33.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:12:33.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 -- 172.21.15.38:0/640433095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 msgr2=0x7f9f8410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 --2- 172.21.15.38:0/640433095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f8410d4f0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009cd0 tx=0x7f9f7c02c0f0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 -- 172.21.15.38:0/640433095 shutdown_connections 2024-04-22T00:12:33.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 --2- 172.21.15.38:0/640433095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f8410d4f0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 --2- 172.21.15.38:0/640433095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84100620 0x7f9f84100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 --2- 172.21.15.38:0/640433095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84106630 0x7f9f84106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 -- 172.21.15.38:0/640433095 >> 172.21.15.38:0/640433095 conn(0x7f9f840776a0 msgr2=0x7f9f84077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 -- 172.21.15.38:0/640433095 shutdown_connections 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.180+0000 7f9f89777640 1 -- 172.21.15.38:0/640433095 wait complete. 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 Processor -- start 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 -- start start 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84100620 0x7f9f84197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84106630 0x7f9f8419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9f84110a40 con 0x7f9f84100fe0 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841108c0 con 0x7f9f84106630 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f89777640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f84110bc0 con 0x7f9f84100620 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.181+0000 7f9f827fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39074/0 (socket says 172.21.15.38:39074) 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 -- 172.21.15.38:0/1246886301 learned_addr learned my addr 172.21.15.38:0/1246886301 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f82ffd640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84100620 0x7f9f84197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f837fe640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84106630 0x7f9f8419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 -- 172.21.15.38:0/1246886301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84100620 msgr2=0x7f9f84197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84100620 0x7f9f84197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 -- 172.21.15.38:0/1246886301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84106630 msgr2=0x7f9f8419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84106630 0x7f9f8419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 -- 172.21.15.38:0/1246886301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8419cf80 con 0x7f9f84100fe0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f827fc640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e9f0 tx=0x7f9f7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7800cd80 con 0x7f9f84100fe0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8419d270 con 0x7f9f84100fe0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f78010910 con 0x7f9f84100fe0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f780188f0 con 0x7f9f84100fe0 2024-04-22T00:12:33.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.182+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f841a4b00 con 0x7f9f84100fe0 2024-04-22T00:12:33.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.183+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f840689b0 con 0x7f9f84100fe0 2024-04-22T00:12:33.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.185+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f9f78010ac0 con 0x7f9f84100fe0 2024-04-22T00:12:33.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.185+0000 7f9f6bfff640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 0x7f9f6c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.185+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f9f780cfef0 con 0x7f9f84100fe0 2024-04-22T00:12:33.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.185+0000 7f9f82ffd640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 0x7f9f6c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.184 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.186+0000 7f9f82ffd640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 0x7f9f6c0a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9f74005fd0 tx=0x7f9f7400b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.189+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9f7809c130 con 0x7f9f84100fe0 2024-04-22T00:12:33.386 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.389+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9f84109ba0 con 0x7f9f84100fe0 2024-04-22T00:12:33.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.389+0000 7f9f6bfff640 1 -- 172.21.15.38:0/1246886301 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) ==== 100+0+11 (secure 0 0 0) 0x7f9f780a0430 con 0x7f9f84100fe0 2024-04-22T00:12:33.387 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:12:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.391+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 msgr2=0x7f9f6c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.391+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 0x7f9f6c0a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9f74005fd0 tx=0x7f9f7400b390 comp rx=0 tx=0).stop 2024-04-22T00:12:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.391+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 msgr2=0x7f9f841984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.391+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e9f0 tx=0x7f9f7800eeb0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 shutdown_connections 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f9f6c0a2d00 0x7f9f6c0a51c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f84106630 0x7f9f8419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9f84100fe0 0x7f9f841984d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 --2- 172.21.15.38:0/1246886301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f84100620 0x7f9f84197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 >> 172.21.15.38:0/1246886301 conn(0x7f9f840776a0 msgr2=0x7f9f8406a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 shutdown_connections 2024-04-22T00:12:33.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.392+0000 7f9f89777640 1 -- 172.21.15.38:0/1246886301 wait complete. 2024-04-22T00:12:33.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:12:33.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- 172.21.15.38:0/1059218837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458101710 msgr2=0x7ff458101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 --2- 172.21.15.38:0/1059218837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458101710 0x7ff458101b90 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009930 tx=0x7ff44c02bc30 comp rx=0 tx=0).stop 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- 172.21.15.38:0/1059218837 shutdown_connections 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 --2- 172.21.15.38:0/1059218837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff45810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 --2- 172.21.15.38:0/1059218837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458101710 0x7ff458101b90 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 --2- 172.21.15.38:0/1059218837 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458107720 0x7ff458107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- 172.21.15.38:0/1059218837 >> 172.21.15.38:0/1059218837 conn(0x7ff458078b90 msgr2=0x7ff458078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- 172.21.15.38:0/1059218837 shutdown_connections 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- 172.21.15.38:0/1059218837 wait complete. 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 Processor -- start 2024-04-22T00:12:33.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.524+0000 7ff45e525640 1 -- start start 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff4581985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff458111b80 con 0x7ff458107720 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7ff458111a00 con 0x7ff458101710 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45e525640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff458111d00 con 0x7ff4581020d0 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff457fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45cac2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff457fff640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.38:33500/0 (socket says 172.21.15.38:33500) 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff45cac2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39098/0 (socket says 172.21.15.38:39098) 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff457fff640 1 -- 172.21.15.38:0/1682691299 learned_addr learned my addr 172.21.15.38:0/1682691299 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:33.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.525+0000 7ff4577fe640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff4581985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 -- 172.21.15.38:0/1682691299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 msgr2=0x7ff4581985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff4581985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 -- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 msgr2=0x7ff45819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 -- 172.21.15.38:0/1682691299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45819d050 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff4577fe640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff4581985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff45cac2640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff457fff640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff44800d950 tx=0x7ff44800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448017030 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff448004020 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45819d340 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4480047a0 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.526+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4581a4bd0 con 0x7ff458101710 2024-04-22T00:12:33.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.527+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff458067cd0 con 0x7ff458101710 2024-04-22T00:12:33.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.529+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7ff4480041d0 con 0x7ff458101710 2024-04-22T00:12:33.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.529+0000 7ff4557fa640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 0x7ff4400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.529+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ff4480cf290 con 0x7ff458101710 2024-04-22T00:12:33.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.529+0000 7ff4577fe640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 0x7ff4400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.530+0000 7ff4577fe640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 0x7ff4400a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009900 tx=0x7ff44c02df90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.533+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff44809bb20 con 0x7ff458101710 2024-04-22T00:12:33.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:33 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2450740419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:33 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1956809201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:33 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1246886301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.741+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff45802d170 con 0x7ff458101710 2024-04-22T00:12:33.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.742+0000 7ff4557fa640 1 -- 172.21.15.38:0/1682691299 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v100) ==== 103+0+11 (secure 0 0 0) 0x7ff44809b420 con 0x7ff458101710 2024-04-22T00:12:33.739 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:12:33.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 msgr2=0x7ff4400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 0x7ff4400a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009900 tx=0x7ff44c02df90 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 msgr2=0x7ff458198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff44800d950 tx=0x7ff44800de10 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 shutdown_connections 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7ff4400a2d00 0x7ff4400a51c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff458107720 0x7ff45819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff4581020d0 0x7ff4581985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 --2- 172.21.15.38:0/1682691299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff458101710 0x7ff458198060 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 >> 172.21.15.38:0/1682691299 conn(0x7ff458078b90 msgr2=0x7ff4580ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 shutdown_connections 2024-04-22T00:12:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.744+0000 7ff45e525640 1 -- 172.21.15.38:0/1682691299 wait complete. 2024-04-22T00:12:33.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T00:12:33.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:33 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2450740419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:33 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1956809201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:33 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1246886301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:33 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2450740419' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:33 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1956809201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:33 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1246886301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 -- 172.21.15.38:0/1068929944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 msgr2=0x7f5368102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 --2- 172.21.15.38:0/1068929944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368102db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5358009860 tx=0x7f535802bca0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 -- 172.21.15.38:0/1068929944 shutdown_connections 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 --2- 172.21.15.38:0/1068929944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 0x7f536810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 --2- 172.21.15.38:0/1068929944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f536810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 --2- 172.21.15.38:0/1068929944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368102db0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 -- 172.21.15.38:0/1068929944 >> 172.21.15.38:0/1068929944 conn(0x7f5368077740 msgr2=0x7f5368077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 -- 172.21.15.38:0/1068929944 shutdown_connections 2024-04-22T00:12:33.859 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.859+0000 7f536dc88640 1 -- 172.21.15.38:0/1068929944 wait complete. 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 Processor -- start 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 -- start start 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 0x7f536819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5368111b10 con 0x7f53681032f0 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f5368111990 con 0x7f536810bd10 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f536dc88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5368111c90 con 0x7f53681029d0 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f535effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f53677fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f535effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53102/0 (socket says 172.21.15.38:53102) 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f535effd640 1 -- 172.21.15.38:0/1263125330 learned_addr learned my addr 172.21.15.38:0/1263125330 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:12:33.860 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.860+0000 7f5367fff640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 0x7f536819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 -- 172.21.15.38:0/1263125330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 msgr2=0x7f536819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 0x7f536819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 -- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 msgr2=0x7f5368198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 -- 172.21.15.38:0/1263125330 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536819cfd0 con 0x7f53681029d0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f535effd640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53677fe640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368198030 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5358037040 tx=0x7f535802c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358041020 con 0x7f53681029d0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53580044b0 con 0x7f53681029d0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536819d260 con 0x7f53681029d0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358039040 con 0x7f53681029d0 2024-04-22T00:12:33.861 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.861+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53681a4b50 con 0x7f53681029d0 2024-04-22T00:12:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.862+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5368068560 con 0x7f53681029d0 2024-04-22T00:12:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.863+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f5358044020 con 0x7f53681029d0 2024-04-22T00:12:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.864+0000 7f53657fa640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 0x7f53540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:12:33.862 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.864+0000 7f535effd640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 0x7f53540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:12:33.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.864+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f53580f0d30 con 0x7f53681029d0 2024-04-22T00:12:33.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.864+0000 7f535effd640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 0x7f53540a5240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f53500096c0 tx=0x7f5350009260 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:12:33.866 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:33.868+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f535804b020 con 0x7f53681029d0 2024-04-22T00:12:34.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.103+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f5368106ec0 con 0x7f53681029d0 2024-04-22T00:12:34.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.112+0000 7f53657fa640 1 -- 172.21.15.38:0/1263125330 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740903 (secure 0 0 0) 0x7f53580bd670 con 0x7f53681029d0 2024-04-22T00:12:34.121 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:12:34.121 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:34.121 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 26, 2024-04-22T00:12:34.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "active_gid": 15528, 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "active_name": "x", 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addrs": { 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2750091046 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-04-22T00:12:34.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2750091046 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addr": "172.21.15.38:6801/2750091046", 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "active_change": "2024-04-22T00:02:04.818404+0000", 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "available": true, 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "standbys": [ 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 25066, 2024-04-22T00:12:34.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "z", 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.126 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.129 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.130 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T00:12:34.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.135 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T00:12:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.137 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T00:12:34.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.139 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.140 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.142 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:34.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.145 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.146 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.147 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.148 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T00:12:34.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.151 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.152 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.153 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.154 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.155 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T00:12:34.156 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T00:12:34.158 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.160 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T00:12:34.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T00:12:34.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.167 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.168 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.169 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T00:12:34.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.171 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.172 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.173 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T00:12:34.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.177 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.178 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T00:12:34.179 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T00:12:34.181 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.182 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T00:12:34.183 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.186 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.187 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T00:12:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.191 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.192 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T00:12:34.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.194 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T00:12:34.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.199 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.200 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T00:12:34.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.203 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.205 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T00:12:34.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.209 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.211 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.212 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.213 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.214 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T00:12:34.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.220 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.221 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.222 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.224 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.225 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T00:12:34.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.228 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T00:12:34.230 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.232 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.233 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.234 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T00:12:34.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.236 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T00:12:34.238 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.239 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.242 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.243 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T00:12:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T00:12:34.249 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.251 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.253 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.254 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T00:12:34.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.257 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.259 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T00:12:34.261 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.266 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.268 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T00:12:34.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.271 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.272 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.273 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T00:12:34.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.276 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.278 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.279 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.280 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T00:12:34.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T00:12:34.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.287 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.288 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T00:12:34.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T00:12:34.292 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.295 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T00:12:34.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.305 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.306 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.307 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.310 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.311 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.312 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T00:12:34.313 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.314 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:34.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:34.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.318 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.320 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.323 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T00:12:34.324 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.327 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.329 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T00:12:34.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.335 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.337 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T00:12:34.338 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T00:12:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T00:12:34.341 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.342 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.344 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.345 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.347 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.348 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.350 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.351 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.352 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.353 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T00:12:34.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.355 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.357 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T00:12:34.358 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.359 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.360 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.361 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.362 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.364 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T00:12:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.367 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.369 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.371 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.374 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.375 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.376 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.377 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.378 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.380 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.382 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.384 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.385 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.386 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T00:12:34.388 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T00:12:34.390 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.392 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.393 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.394 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T00:12:34.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.399 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.401 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.403 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.404 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.406 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.407 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.409 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.410 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.411 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.413 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.414 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.417 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T00:12:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T00:12:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T00:12:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T00:12:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T00:12:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T00:12:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T00:12:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T00:12:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.546 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.547 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T00:12:34.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.550 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T00:12:34.551 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.552 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.553 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.555 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.556 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.559 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.561 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.562 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.564 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T00:12:34.565 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.566 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.567 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.568 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.569 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.570 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.572 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.573 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.574 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.575 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.576 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.577 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.578 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.579 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.580 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T00:12:34.581 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.584 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.585 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.587 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T00:12:34.588 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.590 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.591 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T00:12:34.592 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.593 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.594 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T00:12:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.597 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.598 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.599 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.600 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.601 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.602 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T00:12:34.603 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.605 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.606 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.608 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.611 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.612 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.613 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.614 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.615 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.616 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.617 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.618 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.619 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.620 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.621 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T00:12:34.622 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.623 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.625 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.626 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.628 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.629 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.630 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.633 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T00:12:34.635 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.636 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.637 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:34.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 25175, 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "y", 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.639 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.640 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.641 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.642 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.643 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.644 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.647 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.649 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T00:12:34.650 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.652 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T00:12:34.653 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.655 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.658 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.659 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.660 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.661 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T00:12:34.662 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T00:12:34.663 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.664 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.665 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.666 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.667 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.668 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.669 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.670 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.671 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.672 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.673 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T00:12:34.674 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.675 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.676 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.677 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:34.678 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.680 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.681 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.682 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.683 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.684 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.685 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.686 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.687 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T00:12:34.688 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T00:12:34.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.693 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.694 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T00:12:34.695 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.696 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T00:12:34.697 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T00:12:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.700 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.701 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T00:12:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.703 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T00:12:34.704 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T00:12:34.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.707 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T00:12:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T00:12:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.708 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T00:12:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.709 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.710 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.711 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T00:12:34.712 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T00:12:34.715 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T00:12:34.716 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.717 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.718 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.719 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T00:12:34.720 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.721 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.723 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.725 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.726 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.727 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.730 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.731 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.732 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.733 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.734 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.735 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.738 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.739 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.740 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.741 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.742 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.743 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.744 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.745 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T00:12:34.746 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T00:12:34.747 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.749 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.750 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T00:12:34.752 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.753 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.754 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.755 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.756 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.757 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.758 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T00:12:34.759 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.762 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.763 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.764 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.766 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T00:12:34.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.768 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T00:12:34.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T00:12:34.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T00:12:34.771 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T00:12:34.772 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.774 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.775 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.776 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.777 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.778 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.779 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.782 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.783 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.785 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.786 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.787 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.788 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.789 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.790 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.791 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.792 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.793 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.794 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.795 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.796 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T00:12:34.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T00:12:34.798 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.799 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.800 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.801 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.802 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.803 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.804 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.805 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.808 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.810 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.811 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.812 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.813 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.814 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.815 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.816 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.818 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.820 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.821 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.822 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.824 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.826 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T00:12:34.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.828 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.829 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.830 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.832 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.833 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.834 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.835 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.836 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.837 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.838 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.839 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T00:12:34.840 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.841 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.842 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.843 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.845 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.846 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.847 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.848 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.849 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.849 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T00:12:34.849 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T00:12:34.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.850 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.852 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.853 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.854 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T00:12:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.856 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.857 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.859 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.861 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.862 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.863 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.864 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.865 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.866 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.867 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.868 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.869 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.870 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.871 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.872 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.873 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.874 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.875 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.876 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.878 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.881 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.882 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.883 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.884 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.887 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.888 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.889 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.890 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.891 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.892 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.893 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T00:12:34.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T00:12:34.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.896 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T00:12:34.898 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.899 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.900 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.901 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.902 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.903 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.905 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.907 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.908 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.911 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.912 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.913 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.914 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.915 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.917 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.918 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.919 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.920 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.921 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.923 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.925 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.927 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.928 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.931 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.932 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.933 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.934 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.935 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.936 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.937 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.938 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.939 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.940 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.941 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.942 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.943 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.944 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.945 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.947 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.948 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.949 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.950 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T00:12:34.951 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:34.952 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.954 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.955 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.956 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.957 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.958 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.959 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.960 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.961 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.962 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.963 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.964 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.965 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T00:12:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T00:12:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.968 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:34.969 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.971 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.972 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.974 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.977 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T00:12:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T00:12:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T00:12:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T00:12:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T00:12:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T00:12:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T00:12:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T00:12:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "modules": [ 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "iostat", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "nfs", 2024-04-22T00:12:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "restful", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_schedule" 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.099 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-04-22T00:12:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-04-22T00:12:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.107 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.108 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-04-22T00:12:35.109 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.113 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.115 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.116 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.116 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.116 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:35.116 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.117 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.119 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T00:12:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T00:12:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-04-22T00:12:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-04-22T00:12:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T00:12:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-04-22T00:12:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-04-22T00:12:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-04-22T00:12:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-04-22T00:12:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-04-22T00:12:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-04-22T00:12:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-04-22T00:12:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-04-22T00:12:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_samba": { 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_samba", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Samba/SMB container image", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "grafana_dashboards_path": { 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "grafana_dashboards_path", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T00:12:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T00:12:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T00:12:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T00:12:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T00:12:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T00:12:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T00:12:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T00:12:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.227 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T00:12:35.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T00:12:35.229 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.230 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T00:12:35.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.232 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.233 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.238 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.240 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.243 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.245 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.248 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.251 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.254 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.255 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.256 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.258 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.259 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.262 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-04-22T00:12:35.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T00:12:35.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:35.266 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:35.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-04-22T00:12:35.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:35.270 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.275 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.278 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.279 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.280 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-04-22T00:12:35.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-04-22T00:12:35.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:35.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.286 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.288 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.289 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.291 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.292 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.294 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.295 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-04-22T00:12:35.297 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.300 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.302 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.303 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.306 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.307 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.308 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.309 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.312 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.315 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.316 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.320 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.322 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.323 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.324 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.326 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.328 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.330 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.331 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.333 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.335 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.336 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.338 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.339 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.340 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-04-22T00:12:35.342 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-04-22T00:12:35.344 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.345 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.346 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.347 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.348 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.349 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.351 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.352 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.353 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.354 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.357 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.360 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.362 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.363 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.364 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.365 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.366 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.367 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.368 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.369 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.370 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.371 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.373 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.375 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.377 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.378 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.379 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.380 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.381 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.383 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.384 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.385 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.386 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.389 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-04-22T00:12:35.391 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-04-22T00:12:35.392 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:35.393 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.394 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.395 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.397 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.398 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.399 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.400 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.401 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.402 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.404 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.405 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.406 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.408 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.409 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.411 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.412 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:35.413 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.414 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-04-22T00:12:35.417 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-04-22T00:12:35.418 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.419 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.420 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.421 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.428 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.429 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.430 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.432 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.433 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.434 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.435 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.436 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.438 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.441 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.442 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-04-22T00:12:35.443 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.444 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.446 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.447 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.448 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-04-22T00:12:35.449 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.450 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T00:12:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-04-22T00:12:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-04-22T00:12:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-04-22T00:12:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-04-22T00:12:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "services": { 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard": "https://172.21.15.38:8443/" 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "always_on_modules": { 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus": [ 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T00:12:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific": [ 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T00:12:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy": [ 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T00:12:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "reef": [ 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T00:12:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "squid": [ 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "last_failure_osd_epoch": 92, 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "active_clients": [ 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2074786597 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "libcephsqlite", 2024-04-22T00:12:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3169555064 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2328543159 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3431985928 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1342284463 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-04-22T00:12:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3873349339 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-22T00:12:35.553 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:35.554 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: pgmap v325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-04-22T00:12:35.554 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1682691299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1263125330' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.555 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:34 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: pgmap v325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.556 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1682691299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1263125330' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.557 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:12:35.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:12:35.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:34 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 msgr2=0x7f53540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 0x7f53540a5240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f53500096c0 tx=0x7f5350009260 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 msgr2=0x7f5368198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368198030 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5358037040 tx=0x7f535802c8f0 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 shutdown_connections 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f53540a2d80 0x7f53540a5240 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f536810bd10 0x7f536819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f53681032f0 0x7f5368198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 --2- 172.21.15.38:0/1263125330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53681029d0 0x7f5368198030 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 >> 172.21.15.38:0/1263125330 conn(0x7f5368077740 msgr2=0x7f53680fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 shutdown_connections 2024-04-22T00:12:35.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:12:34.118+0000 7f536dc88640 1 -- 172.21.15.38:0/1263125330 wait complete. 2024-04-22T00:12:35.565 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T00:12:35.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: pgmap v325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1682691299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1263125330' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T00:12:35.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:34 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' 2024-04-22T00:12:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:12:35.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:35 smithi111 ceph-mon[31543]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:12:35.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:35 smithi038 ceph-mon[27972]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi111 to 176.2M 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi111 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi203 to 184758067: error parsing value: Value '184758067' is below minimum 939524096 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-22T00:12:36.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:35 smithi203 ceph-mon[31431]: Unable to set osd_memory_target on smithi038 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-22T00:12:36.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:36 smithi038 ceph-mon[27972]: pgmap v326: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:37.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:36 smithi203 ceph-mon[31431]: pgmap v326: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:37.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:36 smithi111 ceph-mon[31543]: pgmap v326: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:37.777 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 1468, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1470, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 468269, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 466825, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.147283577, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.043173866, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.035595720, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.056618116, 2024-04-22T00:12:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1468, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.069315484, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000047217 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 656, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 815, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 163944, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 650653, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 7, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.076294309, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.024795146, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.016889172, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.025956288, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 813, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.049248547, 2024-04-22T00:12:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000060576 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 443808 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 256, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 230, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1708492, 2024-04-22T00:12:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 133710, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 3, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.048029551, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013516974, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.039646331, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005034083, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 229, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.012761481, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000055726 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1708224, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 133392 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T00:12:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3480, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.104423555, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000030006 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T00:12:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-22T00:12:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 2, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 2, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 2, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 1, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-22T00:12:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:12:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 17, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-22T00:12:37.791 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4224242, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 4221637, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 17 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.792 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 4, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 19, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 15, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 2, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 15, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 13, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 2, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 17, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 15, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 50484, 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215420 2024-04-22T00:12:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-22T00:12:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-22T00:12:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-22T00:12:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-22T00:12:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-22T00:12:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-22T00:12:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 3, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 1, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 1832, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 189393, 2024-04-22T00:12:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 51, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 178, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 15072, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 40, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-22T00:12:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 2119, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 2119, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2119, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 25.496558617, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.012032354 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2119, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 2120 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 2119, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 2119, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 1375, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 742, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-22T00:12:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-22T00:12:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 100, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 11, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 5, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 5, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-22T00:12:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 2376, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 2110784, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 2376, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2376, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2110784, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 2119, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 29943, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2119, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 29943, 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 2119, 2024-04-22T00:12:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 2119, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2119, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2119, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x5603f71f66e0": { 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:37.826 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:38.666 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-22T00:12:38.666 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-22T00:12:38.666 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-04-22T00:12:38.666 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-04-22T00:12:38.666 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-22T00:12:38.667 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:12:38.667 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:12:38.667 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-04-22T00:12:38.667 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T00:12:38.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T00:12:38.836 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:38 smithi203 ceph-mon[31431]: pgmap v327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:38.921 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:12:38.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:38 smithi038 ceph-mon[27972]: pgmap v327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:39.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:38 smithi111 ceph-mon[31543]: pgmap v327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:40.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:40 smithi038 ceph-mon[27972]: pgmap v328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-22T00:12:40.700 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001363073, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000363915, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:40.701 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 6, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 168, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3392, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 185168, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-22T00:12:40.702 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.014318666, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.009070906, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002999395, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 166, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.015863216, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000095561 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:40.703 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 185168 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 230, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 219, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1583216, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 134464, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-22T00:12:40.704 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.043414646, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013504659, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.035185803, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004776928, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 218, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013204327, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000060570 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:40.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1583216, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 134464 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:40.706 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:40.707 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:40.708 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148994, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 31, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T00:12:40.709 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:40.710 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 232, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1550803, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 232, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:40.711 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 232, 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1550803, 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:40.712 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:41.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:40 smithi203 ceph-mon[31431]: pgmap v328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:41.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:40 smithi111 ceph-mon[31543]: pgmap v328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:41.574 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-22T00:12:41.575 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-22T00:12:41.575 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T00:12:41.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T00:12:41.827 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:12:42.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:42 smithi038 ceph-mon[27972]: pgmap v329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:43.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:42 smithi203 ceph-mon[31431]: pgmap v329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:43.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:42 smithi111 ceph-mon[31543]: pgmap v329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:43.668 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 676, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 678, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 227081, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 214969, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 2, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.084488593, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.031050288, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.020066762, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.025696542, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 676, 2024-04-22T00:12:43.669 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.045932474, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000067947 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 1250, 2024-04-22T00:12:43.670 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1410, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 426984, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 843041, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 7, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.103599919, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.032684839, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.026397122, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.035976738, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1408, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.074623472, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000052999 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.671 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 447392 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 260, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 234, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1710116, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 135637, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-22T00:12:43.672 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.047441802, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013965432, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.039071177, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004815512, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 233, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011434747, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000049076 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1708768, 2024-04-22T00:12:43.673 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 134672 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3218, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.093660859, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000029105 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:43.674 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.675 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 5, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 5, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-22T00:12:43.676 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 5, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 1, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-22T00:12:43.677 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:12:43.678 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-22T00:12:43.679 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-22T00:12:43.680 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 48, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-22T00:12:43.681 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 2, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 4194304, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 4301008, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 4298403, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 48 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 4, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 31, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 27, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 2, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 18, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 16, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 5, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 35, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 30, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-22T00:12:43.683 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 51716, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215420 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-22T00:12:43.684 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-22T00:12:43.685 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-22T00:12:43.686 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-22T00:12:43.688 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.689 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.690 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.692 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:43.693 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:43.695 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 4, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-22T00:12:43.696 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 4248, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:43.697 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:43.698 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:43.699 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 197490, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 56, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:43.700 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 178, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 17184, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 52, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-22T00:12:43.701 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 1923, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 1923, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1923, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 83.845548164, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.043601429 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-22T00:12:43.702 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1923, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 1924 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 1923, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 1923, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 1230, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 691, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-22T00:12:43.703 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:43.704 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-22T00:12:43.705 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 100, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 11, 2024-04-22T00:12:43.706 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 6, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 6, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-22T00:12:43.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:43.708 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 2182, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 2151616, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 2182, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 2182, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 2151616, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:43.709 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 1923, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 106709, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 1923, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 106709, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.710 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 1923, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 1923, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 1923, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1923, 2024-04-22T00:12:43.711 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:43.712 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x55b70868a6e0": { 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-22T00:12:43.713 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:43.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:43.715 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:12:44.472 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:12:44.473 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-04-22T00:12:44.473 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T00:12:44.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T00:12:44.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:44 smithi038 ceph-mon[27972]: pgmap v330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:44.726 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T00:12:45.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:44 smithi203 ceph-mon[31431]: pgmap v330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:45.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:44 smithi111 ceph-mon[31543]: pgmap v330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:46.522 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-22T00:12:46.522 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:46.522 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001364901, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000394733, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:46.523 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 169, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 320, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 185248, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-22T00:12:46.524 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.013985949, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.009628659, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002035912, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 166, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.013922750, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000083871 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T00:12:46.525 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 185248 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 236, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 223, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1586832, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 136192, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.045343053, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013864049, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.036829672, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004715707, 2024-04-22T00:12:46.526 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 223, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.022633704, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000101496 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1586832, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 136192 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-22T00:12:46.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:46.528 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148794, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 32, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-22T00:12:46.530 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:46.531 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 234, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1551063, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 234, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 234, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1551063, 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-22T00:12:46.532 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-22T00:12:46.533 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-22T00:12:46.533 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-22T00:12:46.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:46.533 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-22T00:12:46.533 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-22T00:12:46.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:46 smithi038 ceph-mon[27972]: pgmap v331: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:46.884 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:46 smithi111 ceph-mon[31543]: pgmap v331: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:47.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:46 smithi203 ceph-mon[31431]: pgmap v331: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-22T00:12:47.440 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-22T00:12:47.440 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T00:12:47.440 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T00:12:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T00:12:48.949 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:48 smithi111 ceph-mon[31543]: pgmap v332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:48.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:48 smithi038 ceph-mon[27972]: pgmap v332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:49.037 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:48 smithi203 ceph-mon[31431]: pgmap v332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 9178, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 12703, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 2966433, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 22590634, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 25, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.808208996, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.260210586, 2024-04-22T00:12:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.249175071, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.242716009, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 12704, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.808584028, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000142363 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 11243, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001855186, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000165 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.786 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 323120 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 7955, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 9471, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 4557608, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 19586579, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 14, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.701275568, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.215262861, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.237297485, 2024-04-22T00:12:49.787 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.222478477, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9474, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.378701843, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000145524 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5918, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001127917, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000190 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1802112, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 172992 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.788 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 20941, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 26220, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 7160323, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 25887097, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 215, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 12, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 1.681825009, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.578696540, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 1.047423221, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.262770225, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 26211, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 3.168222451, 2024-04-22T00:12:49.789 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000120873 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 34297, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.006345745, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000185 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 447584, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 256432 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T00:12:49.790 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9265, 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 7.508416247, 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000810406 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:49.791 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4979, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.281293184, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000056495 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 2700, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 2691, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2691, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 13.355408624, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004962990 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.792 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 9, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 3, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 1, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 571, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 3, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 1397, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 367, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 1019, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 11, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 873, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 23, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 2, 2024-04-22T00:12:49.793 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 2, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 6, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 3614, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 3386, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 1, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 5, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 39, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 8, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 12, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 8, 2024-04-22T00:12:49.794 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 27, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 4111, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 4111, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 926, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 19, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 907, 2024-04-22T00:12:49.795 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 3185, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 16, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 763, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 4283, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 6, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 10204, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 8323, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 118, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 1344 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 368, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 13, 2024-04-22T00:12:49.796 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 9, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 8, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 692, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 79, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 921, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 781, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 61, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T00:12:49.797 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 19, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 8, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 3, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 13166, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 12257, 2024-04-22T00:12:49.798 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 12257, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 909, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 12, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 45, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 43, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 43, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 1, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 1, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 45258735, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 48430189, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T00:12:49.799 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2730, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 49.432874811, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.018107280 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 912, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 1727, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 815, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 122, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 148, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 26, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 1397, 2024-04-22T00:12:49.800 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 2497, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 1100, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 2, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 884, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 882, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 80776, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 231804 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 3574, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 21, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 2700, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 47, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 122, 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T00:12:49.801 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 763, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.837328941, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001097416 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 163, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.048919369, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000300118 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.802 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 61, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.066930669, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001097224 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 311, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.107740999, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000346434 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.803 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.805 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002320868, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000232086 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 103, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.139570825, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001355056 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.806 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 10, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.217146100, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.021714610 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 48, 2024-04-22T00:12:49.807 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.013010709, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000271056 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 185, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.923932339, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004994228 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 111, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 7.563967878, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.068143854 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:49.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 95, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.984668615, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.020891248 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.809 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.059145442, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.014786360 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 8, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.036669481, 2024-04-22T00:12:49.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.004583685 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.001613269, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000403317 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 9, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.022710346, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002523371 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 13, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.008128148, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000625242 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 793, 2024-04-22T00:12:49.812 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1.321604626, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001666588 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 7, 2024-04-22T00:12:49.813 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 5, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 157, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 78, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 614, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:49.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:49.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:49.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 964050, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 708, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 1056, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 12, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 178, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:49.817 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 3361572, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 46737, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T00:12:49.818 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 14408, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 47425582, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 33, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 14408, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 14408, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 1023.857036481, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.071061704 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 14408, 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 17321 2024-04-22T00:12:49.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 14408, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 4173, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 10235, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 451, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 909, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 39, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 2750, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 132, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 1, 2024-04-22T00:12:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 5528, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 4134, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 1420, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T00:12:49.821 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 6, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T00:12:49.822 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 100, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 16, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 12, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 12, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 1655, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 8, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 288 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.823 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 1 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 52, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 9, 2024-04-22T00:12:49.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 4995, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 781, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 38070, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 11186149, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 38070, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 38070, 2024-04-22T00:12:49.825 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 11186149, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:49.826 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 14408, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 47525403, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 14408, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 47525403, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:49.827 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 14408, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 14408, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 14408, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 14408, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 781, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 99873, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 781, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:49.829 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 20, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 99873, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x559210f28500": { 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 13166, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 44235885, 2024-04-22T00:12:49.830 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 13166, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2730, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 44235885, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:49.831 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T00:12:49.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:49 smithi038 ceph-mon[27972]: pgmap v333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:50.034 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:49 smithi203 ceph-mon[31431]: pgmap v333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:50.081 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:49 smithi111 ceph-mon[31543]: pgmap v333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds.exported=8 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds.imported=8 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=368 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=13 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-22T00:12:50.604 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=3 2024-04-22T00:12:50.604 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T00:12:50.605 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T00:12:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T00:12:52.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:52 smithi111 ceph-mon[31543]: pgmap v334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:52.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:52 smithi038 ceph-mon[27972]: pgmap v334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:52.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:52 smithi203 ceph-mon[31431]: pgmap v334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 1847, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 1849, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 19608412, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 587356, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 7, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.162806705, 2024-04-22T00:12:52.576 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.052638845, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.046031465, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.051841867, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1847, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.089634909, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000048529 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 306, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 471, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 16858012, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 546970, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 7, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.044913533, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.010892824, 2024-04-22T00:12:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.021261352, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.009532100, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 469, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.029174731, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000062206 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.579 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 451504 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 701, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 675, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 10208069, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 277639, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.086062457, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.022182965, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.052370110, 2024-04-22T00:12:52.580 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.019134587, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 674, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.040117706, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000059521 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1710560, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 138672 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.581 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 4368, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.123425099, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000028256 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:52.582 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 2354, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 822, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 1532, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T00:12:52.584 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 1180, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 2, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T00:12:52.585 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:52.586 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-22T00:12:52.587 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T00:12:52.588 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T00:12:52.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 13166, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 32, 2024-04-22T00:12:52.590 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 45258735, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 48430189, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 48430053, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 13166 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 903, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 1693, 2024-04-22T00:12:52.591 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 790, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 113, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 134, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 21, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 2354, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 2417, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 63, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 61268, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 215420 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T00:12:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.593 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.596 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.597 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T00:12:52.597 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.597 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.597 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.599 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T00:12:52.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.603 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.604 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 7, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 5, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 5, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 3562, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T00:12:52.605 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:52.606 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:52.607 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 243862, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 1056, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T00:12:52.608 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 178, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 4044030, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 52363, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:52.609 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 2584, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 26, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 4, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 2584, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2584, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 8.444843185, 2024-04-22T00:12:52.610 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003268128 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2584, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2585 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 2584, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 2584, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 0, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 1520, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 0, 2024-04-22T00:12:52.611 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 1061, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 2, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:52.612 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T00:12:52.613 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 100, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 11, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 7, 2024-04-22T00:12:52.614 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 7, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 2, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 0 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T00:12:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:52.616 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 2850, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 46401450, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 2850, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2850, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 46401450, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:52.617 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2584, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 44241933, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2584, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 44241933, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.618 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2584, 2024-04-22T00:12:52.619 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 2584, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2584, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2584, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:52.620 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.621 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55a8bbd5e6e0": { 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:52.622 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:52.623 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T00:12:53.467 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-22T00:12:53.467 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:12:53.468 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-04-22T00:12:53.468 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T00:12:53.468 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T00:12:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T00:12:54.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:54 smithi038 ceph-mon[27972]: pgmap v335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:54.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:54 smithi203 ceph-mon[31431]: pgmap v335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:54.555 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:54 smithi111 ceph-mon[31543]: pgmap v335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-22T00:12:55.556 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T00:12:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 96, 2024-04-22T00:12:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 118, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 34791, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 297015, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 29, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 4, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.020916192, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.003873160, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.011622651, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.001203077, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 116, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.015772893, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000135973 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 97, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000018326, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000188 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 3850, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2171, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1524159, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 1014032, 2024-04-22T00:12:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 25, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 8, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.206665667, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.089452817, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.152477034, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.003825237, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2170, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.092262051, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000042517 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 5771, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000850309, 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000147 2024-04-22T00:12:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 4800, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 135872 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 475, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 525, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1834225, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 653670, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 17, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 6, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.065656223, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.021803156, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.050149984, 2024-04-22T00:12:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.006845550, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 523, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.036932345, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000070616 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 201, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000026475, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000131 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1733312, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 473328 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 92, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.005555930, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000060390 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 15, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000361976, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000024131 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 12, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 2, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002621301, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.001310650 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 10, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 11, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 19, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 3, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 6, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 10, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 13, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 176, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-22T00:12:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 4, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 222, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 114, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 10, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 95, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T00:12:55.565 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 1, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 3, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 1, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 1, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 10, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 4, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 234, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 6, 2024-04-22T00:12:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 10, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 10, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 1, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 2, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 1, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 2, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 2 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 3, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 298, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 196, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 190, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 3, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 1, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 1, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 1, 2024-04-22T00:12:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 17, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 17, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 2, 2024-04-22T00:12:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4224242, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 0, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 15, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.076327747, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.005088516 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 0 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 24, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 114, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 90, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 23, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 139, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 116, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 19, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 196, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 177, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 3, 2024-04-22T00:12:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 3, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 55072, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 223612 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 15, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 11, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 12, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 21, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 24, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000400608, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000400608 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.002220693, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.002220693 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 3, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 1, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 2, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 1, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-22T00:12:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 622, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 378176, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 137, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 178, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 104245, 2024-04-22T00:12:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 394, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 81, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 42420, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 33, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 81, 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T00:12:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 81, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.588989127, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.007271470 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 81, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 107 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 81, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 11, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 70, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T00:12:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 10, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 1, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 16, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 13, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 14, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 10, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 20, 2024-04-22T00:12:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T00:12:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 100, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 11, 2024-04-22T00:12:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 10, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 10, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 20, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 0, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 3 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 1, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T00:12:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 2, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 1, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 2, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 1, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 4417, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 2976290, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 4417, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 4417, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2976290, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 81, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 42577, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 81, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 42577, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 81, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 81, 2024-04-22T00:12:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 81, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 81, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 1, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 157, 2024-04-22T00:12:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 1, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 1, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 157, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x55e41348c500": { 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 17, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 29938, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 17, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 15, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 29938, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:55.603 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds.exported=1 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds.imported=1 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=3 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=298 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:12:56.357 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-22T00:12:56.357 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T00:12:56.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T00:12:56.380 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:56 smithi111 ceph-mon[31543]: pgmap v336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:56.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:56 smithi038 ceph-mon[27972]: pgmap v336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:56.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:56 smithi203 ceph-mon[31431]: pgmap v336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T00:12:56.599 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T00:12:58.294 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:12:58 smithi111 ceph-mon[31543]: pgmap v337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 789, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 791, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 327710, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 250966, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 6, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.068527937, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.026444331, 2024-04-22T00:12:58.295 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.016330920, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.019143606, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 789, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.032906383, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000041706 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:12:58.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 688, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 857, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 169098, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 669561, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 7, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 5, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.070531820, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.022420657, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.015014689, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.023869440, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 854, 2024-04-22T00:12:58.297 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.044933569, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000052615 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 451408 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 1035, 2024-04-22T00:12:58.298 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 1007, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1971328, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 384150, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 3, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.129798678, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.038287015, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.060305669, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.034107540, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 1007, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.059999560, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000059582 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.299 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1714288, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 140880 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 1, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-MDSRank": { 2024-04-22T00:12:58.300 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 3738, 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.110837942, 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000029651 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "finisher-PurgeQueue": { 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "queue_len": 0, 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "complete_latency": { 2024-04-22T00:12:58.301 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "request": 0, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "reply": 0, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "reply_latency": { 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.302 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_reply": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "forward": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_complete": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_fetch_keys": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_commit": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_split": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_merge": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes": 16, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_top": 0, 2024-04-22T00:12:58.303 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_bottom": 16, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pin_tail": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_pinned": 16, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_expired": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "inodes_with_caps": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "subtrees": 1, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_hit": 0, 2024-04-22T00:12:58.304 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_forward": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_discover": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_remote_ino": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "traverse_lock": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "load_cent": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "q": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "exported": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "exported_inodes": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "imported": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "imported_inodes": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_dir_fetch": 0, 2024-04-22T00:12:58.305 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_backtrace_fetch": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "openino_peer_discover": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rfiles": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rbytes": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "root_rsnaps": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_set_tag": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:12:58.306 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_file_inodes": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_grant": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps": 0, 2024-04-22T00:12:58.307 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_cap_release": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "process_request_cap_release": 0 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache": { 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_update_receipt": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_try_discover": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_send_discover": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "dir_handle_discover": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays": 0, 2024-04-22T00:12:58.308 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_delayed": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_created": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_enqueued": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_reintegrated": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "strays_migrated": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_processing": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:12:58.309 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_started": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_completed": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_exportdir": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_flush": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_fragstats": 0, 2024-04-22T00:12:58.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ireq_inodestats": 0 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log": { 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evlrg": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evadd": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evex": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evtrm": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "ev": 53, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evexg": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "evexd": 0, 2024-04-22T00:12:58.311 INFO:teuthology.orchestra.run.smithi111.stdout: "segadd": 0, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "segex": 0, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "segtrm": 0, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "seg": 2, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "segmjr": 2, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "segexg": 0, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "segexd": 0, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "expos": 4194304, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "wrpos": 4314993, 2024-04-22T00:12:58.312 INFO:teuthology.orchestra.run.smithi111.stdout: "rdpos": 4309843, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "jlat": { 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "replayed": 53 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mem": { 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 11, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "ino+": 29, 2024-04-22T00:12:58.313 INFO:teuthology.orchestra.run.smithi111.stdout: "ino-": 18, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dir": 6, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dir+": 20, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dir-": 14, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dn": 16, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dn+": 44, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "dn-": 28, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "cap": 0, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "cap+": 0, 2024-04-22T00:12:58.314 INFO:teuthology.orchestra.run.smithi111.stdout: "cap-": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "rss": 51088, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "heap": 215420 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server": { 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_client_request": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "dispatch_server_request": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_request": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_client_session": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "handle_peer_request": 0, 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "req_create_latency": { 2024-04-22T00:12:58.315 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getattr_latency": { 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getfilelock_latency": { 2024-04-22T00:12:58.316 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "req_link_latency": { 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookup_latency": { 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookuphash_latency": { 2024-04-22T00:12:58.317 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupino_latency": { 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupname_latency": { 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.318 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupparent_latency": { 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lookupsnap_latency": { 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "req_lssnap_latency": { 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mkdir_latency": { 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.320 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mknod_latency": { 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "req_mksnap_latency": { 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "req_open_latency": { 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "req_readdir_latency": { 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rename_latency": { 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "req_renamesnap_latency": { 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.322 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "req_snapdiff_latency": { 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmdir_latency": { 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmsnap_latency": { 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.323 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "req_rmxattr_latency": { 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setattr_latency": { 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setdirlayout_latency": { 2024-04-22T00:12:58.324 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setfilelock_latency": { 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setlayout_latency": { 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.325 INFO:teuthology.orchestra.run.smithi111.stdout: "req_setxattr_latency": { 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "req_symlink_latency": { 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "req_unlink_latency": { 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "req_getvxattr_latency": { 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessions": { 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "session_count": 2, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "session_add": 5, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "session_remove": 3, 2024-04-22T00:12:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_open": 4, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "sessions_stale": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "total_load": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "average_load": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "avg_session_uptime": 2986, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "mdthresh_evicted": 0 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:12:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:12:58.329 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:12:58.330 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 190685, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 66, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2024-04-22T00:12:58.331 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 9320, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 178, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 51630, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 742, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2024-04-22T00:12:58.332 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter": { 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_active": 0, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_laggy": 0, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send": 2240, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_send_bytes": 21, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_resend": 3, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_reply": 2240, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "op_latency": { 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2240, 2024-04-22T00:12:58.333 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 8.675838869, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.003873142 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "op_inflight": 0, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "oplen_avg": { 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 2240, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 2241 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "op": 2240, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "op_r": 2240, 2024-04-22T00:12:58.334 INFO:teuthology.orchestra.run.smithi111.stdout: "op_w": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "op_rmw": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "op_pg": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_stat": 1449, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_create": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_read": 789, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_write": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writefull": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_writesame": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_append": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_zero": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_truncate": 0, 2024-04-22T00:12:58.335 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_delete": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_mapext": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_sparse_read": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_clonerange": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_getxattr": 1, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_setxattr": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_rmxattr": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_call": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_watch": 0, 2024-04-22T00:12:58.336 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_notify": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "osdop_other": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_active": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_send": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_resend": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "linger_ping": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_active": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_send": 0, 2024-04-22T00:12:58.337 INFO:teuthology.orchestra.run.smithi111.stdout: "poolop_resend": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_active": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_send": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "poolstat_resend": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_active": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_send": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "statfs_resend": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "command_active": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "command_send": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "command_resend": 0, 2024-04-22T00:12:58.338 INFO:teuthology.orchestra.run.smithi111.stdout: "map_epoch": 100, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "map_full": 0, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "map_inc": 11, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_sessions": 5, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_open": 5, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_session_close": 0, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_laggy": 0, 2024-04-22T00:12:58.339 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_wr": 0, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_rd": 2, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_del": 0 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "oft": { 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_objs": 0, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_updates": 0, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "omap_total_removes": 0 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.340 INFO:teuthology.orchestra.run.smithi111.stdout: "osd-slow-ops": { 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "slow_ops_count": 0 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "purge_queue": { 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops": 0, 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing": 0, 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executing_high_water": 0, 2024-04-22T00:12:58.341 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed_ops": 0, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_executed": 0, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "pq_item_in_journal": 0 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 2508, 2024-04-22T00:12:58.342 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 2225763, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 2508, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2508, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2225763, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.343 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_bytes": { 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:58.344 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2240, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 120694, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2240, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 120694, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.345 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-objecter_ops": { 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 1024, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 2240, 2024-04-22T00:12:58.346 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 2240, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 2240, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 2240, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:58.347 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.348 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-write_buf_throttle-0x562570bd46e0": { 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 3758096384, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 0, 2024-04-22T00:12:58.349 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 0, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 0, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2024-04-22T00:12:58.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.351 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-22T00:12:58.351 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-22T00:12:58.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:12:58 smithi038 ceph-mon[27972]: pgmap v337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:58.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:12:58 smithi203 ceph-mon[31431]: pgmap v337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T00:12:59.066 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:12:59.067 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-04-22T00:12:59.067 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T00:12:59.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T00:12:59.354 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T00:13:00.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:00 smithi038 ceph-mon[27972]: pgmap v338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:00.533 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:00 smithi203 ceph-mon[31431]: pgmap v338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:00.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:00 smithi111 ceph-mon[31543]: pgmap v338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 4055, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2487, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1621928, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1063419, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 14, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.230799022, 2024-04-22T00:13:01.301 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.082894952, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.174679237, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.012017798, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2485, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.138429967, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000055706 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 6175, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001113962, 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000180 2024-04-22T00:13:01.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1568, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 5712 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 504, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 530, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1834925, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 543712, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 30, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.074535312, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.022103448, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.056034937, 2024-04-22T00:13:01.303 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007134572, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 530, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.035705782, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000067369 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 265, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000046098, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000173 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1743888, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 161264 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.304 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 184, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 469, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 100420, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 791502, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 28, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 8, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.036396006, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.014037239, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.019855278, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000621005, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 466, 2024-04-22T00:13:01.305 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.047821807, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000102621 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 385, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000073376, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000190 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 459056 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T00:13:01.306 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 195, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.035476057, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000181928 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:01.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 17, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000645502, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000037970 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 44, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 32, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 32, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.130335233, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004072976 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.308 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 12, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 27, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 1, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 9, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 17, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 25, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 181, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 1, 2024-04-22T00:13:01.309 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 1, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 268, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 143, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 9, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 92, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 3, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 11, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 3, 2024-04-22T00:13:01.310 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 10, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 60, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 6, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 60, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 40, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:13:01.311 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 40, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 20, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 2, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 12, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 20, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 18, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 2, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 2, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 20 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 18, 2024-04-22T00:13:01.312 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 286, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 187, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 180, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 33, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 4, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 6, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 2, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:13:01.313 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 3, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T00:13:01.314 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 53, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 53, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4314993, 2024-04-22T00:13:01.315 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 38, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.189364871, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004983286 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 25, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 117, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 92, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 21, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 136, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 115, 2024-04-22T00:13:01.316 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 27, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 216, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 189, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 1, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 16, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 15, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 55544, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 75, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 12, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 44, 2024-04-22T00:13:01.317 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 30, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 39, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001724985, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000862492 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000401706, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000200853 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.318 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 11, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.008745012, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000795001 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T00:13:01.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000081360, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000081360 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T00:13:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.030641125, 2024-04-22T00:13:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.015320562 2024-04-22T00:13:01.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.323 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 6, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.004158512, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000693085 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002130558, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001065279 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.324 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.073237438, 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.018309359 2024-04-22T00:13:01.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T00:13:01.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.009214537, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004607268 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 5, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 3, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 9, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 4, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 618, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.330 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:13:01.331 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T00:13:01.332 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 362987, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 147, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 352, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 4, 2024-04-22T00:13:01.333 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9320, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 178, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 113942, 2024-04-22T00:13:01.334 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 1134, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 166, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 135455, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 166, 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T00:13:01.335 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 166, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 3.057927538, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.018421250 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 166, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 203 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 166, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 61, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 105, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T00:13:01.336 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 39, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 15, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 16, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 60, 2024-04-22T00:13:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T00:13:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 100, 2024-04-22T00:13:01.339 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 9, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 9, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 36, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 6 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T00:13:01.340 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 4, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 2, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 4, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 2, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:13:01.341 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4739, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 3110374, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4739, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 4739, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3110374, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.342 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 166, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 135769, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 166, 2024-04-22T00:13:01.343 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 135769, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:01.344 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 166, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 166, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 166, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 166, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:01.345 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 314, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 314, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:01.346 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55d757c54500": { 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 53, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 120689, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 53, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 38, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 120689, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:01.347 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:01.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:01.348 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.348 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:01.348 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T00:13:02.158 INFO:tasks.check_counter:Daemon mds.c mds.exported=3 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds.imported=3 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=18 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=286 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:13:02.159 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-22T00:13:02.159 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T00:13:02.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T00:13:02.425 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T00:13:02.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:02 smithi038 ceph-mon[27972]: pgmap v339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:02.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:02 smithi203 ceph-mon[31431]: pgmap v339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:02.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:02 smithi111 ceph-mon[31543]: pgmap v339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T00:13:04.282 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T00:13:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:13:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 4243, 2024-04-22T00:13:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2551, 2024-04-22T00:13:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 3257246, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1214543, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 27, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 7, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.266442425, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.090988781, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.211737633, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.015102540, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2555, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.122980840, 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000048133 2024-04-22T00:13:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5867, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001164372, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000198 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1733648, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 172640 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 231, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 252, 2024-04-22T00:13:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 141211, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 336467, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 19, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 7, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.027438753, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.009364747, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.017341012, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.001236805, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 249, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.022114352, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000088812 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 325, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000056887, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000175 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 132832 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 64, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 204, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 27067, 2024-04-22T00:13:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 569009, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 28, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 8, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.018335679, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.006403175, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.009416940, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000990024, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 198, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.031919416, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000161209 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 40, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000006801, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000170 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 324816 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T00:13:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 141, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.024251731, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000171998 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 19, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000694418, 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000036548 2024-04-22T00:13:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 18, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 14, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 14, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.028290496, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002020749 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T00:13:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 4, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 16, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 2, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 1, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 13, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 185, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 1, 2024-04-22T00:13:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 1, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 4, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 238, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 125, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 1, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 97, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 4, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 12, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 4, 2024-04-22T00:13:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 4, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 40, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 6, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 40, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:13:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 40, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 8, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 8, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 4, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 6, 2024-04-22T00:13:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 8 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 12, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 291, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 192, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 181, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 12, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 4, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 4, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T00:13:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 4, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T00:13:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 48, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 48, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T00:13:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4301008, 2024-04-22T00:13:04.298 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 32, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.253673554, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.007927298 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 21, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 121, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 100, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 20, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 135, 2024-04-22T00:13:04.299 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 115, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 16, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 205, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 189, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 1, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 15, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 14, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 55396, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 34, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 11, 2024-04-22T00:13:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 18, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 25, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 31, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T00:13:04.301 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002363023, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000590755 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T00:13:04.302 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000527608, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000263804 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.307 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.308 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.005963884, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001490971 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.310 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.311 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T00:13:04.312 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.019435981, 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004858995 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:13:04.314 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-04-22T00:13:04.315 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 4, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 2, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 8, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 4, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 602, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:13:04.316 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:13:04.317 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:13:04.318 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T00:13:04.319 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 386311, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 141, 2024-04-22T00:13:04.320 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9320, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 178, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T00:13:04.321 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 89661, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 363, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T00:13:04.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.323 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T00:13:04.323 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T00:13:04.323 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T00:13:04.323 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 137, 2024-04-22T00:13:04.323 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 120770, 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 137, 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 137, 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 3.489917325, 2024-04-22T00:13:04.324 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.025473849 2024-04-22T00:13:04.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.325 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T00:13:04.325 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T00:13:04.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 137, 2024-04-22T00:13:04.325 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 167 2024-04-22T00:13:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 137, 2024-04-22T00:13:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 41, 2024-04-22T00:13:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 96, 2024-04-22T00:13:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T00:13:04.327 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T00:13:04.327 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-22T00:13:04.327 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-22T00:13:04.327 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1, 2024-04-22T00:13:04.327 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 33, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 12, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 20, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 40, 2024-04-22T00:13:04.328 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T00:13:04.329 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T00:13:04.330 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 100, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 12, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 12, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 26, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 5 2024-04-22T00:13:04.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 8, 2024-04-22T00:13:04.332 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 4, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 8, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 4, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4534, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 2997266, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4534, 2024-04-22T00:13:04.333 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 4534, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2997266, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:04.334 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 137, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 121398, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 137, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 121398, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.335 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 137, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 137, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 137, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 137, 2024-04-22T00:13:04.336 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 628, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:04.337 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 628, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55a8f3fae500": { 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:04.338 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:04.339 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:04.339 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 48, 2024-04-22T00:13:04.339 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 106704, 2024-04-22T00:13:04.339 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:04.339 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 48, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 32, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 106704, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:04.340 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:04.341 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:04.341 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.341 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:04.341 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T00:13:04.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:04 smithi203 ceph-mon[31431]: pgmap v340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-22T00:13:04.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:04 smithi038 ceph-mon[27972]: pgmap v340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-22T00:13:04.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:04 smithi111 ceph-mon[31543]: pgmap v340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-22T00:13:05.103 INFO:tasks.check_counter:Daemon mds.f mds.exported=4 2024-04-22T00:13:05.103 INFO:tasks.check_counter:Daemon mds.f mds.imported=4 2024-04-22T00:13:05.103 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=12 2024-04-22T00:13:05.103 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=291 2024-04-22T00:13:05.104 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-22T00:13:05.104 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:13:05.104 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:13:05.104 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-22T00:13:05.104 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T00:13:05.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T00:13:05.347 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T00:13:06.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:06 smithi038 ceph-mon[27972]: pgmap v341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-04-22T00:13:06.504 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:06 smithi203 ceph-mon[31431]: pgmap v341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-04-22T00:13:06.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:06 smithi111 ceph-mon[31543]: pgmap v341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-04-22T00:13:07.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:07 smithi038 ceph-mon[27972]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T00:13:07.507 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:07 smithi203 ceph-mon[31431]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3889, 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2185, 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1555770, 2024-04-22T00:13:07.508 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 876237, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 28, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.217243203, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.075027753, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.175436471, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007420493, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2183, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.136624056, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000062585 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5825, 2024-04-22T00:13:07.509 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001143358, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000196 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 411, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 412, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1784404, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 384004, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 24, 2024-04-22T00:13:07.510 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.068450233, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.020152878, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.050677598, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007185305, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 412, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.032534081, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000078966 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 147, 2024-04-22T00:13:07.511 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000046509, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000316 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1747152, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 157392 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 201, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 316, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 114024, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 648673, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 18, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.027372706, 2024-04-22T00:13:07.512 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.011268094, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.014746792, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.001312281, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 313, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.031658553, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000101145 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 242, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000040251, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000166 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.513 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 456736 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 87, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.010353638, 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000119007 2024-04-22T00:13:07.514 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 17, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000610155, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000035891 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 22, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 5, 2024-04-22T00:13:07.515 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.013675050, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002735010 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 17, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 17, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 1, 2024-04-22T00:13:07.516 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 6, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 10, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 182, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 1, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 1, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 4, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 236, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 114, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 17, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 98, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T00:13:07.517 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 2, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 7, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 2, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 3, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 20, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 6, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 20, 2024-04-22T00:13:07.518 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 20, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 2, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 4, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 2, 2024-04-22T00:13:07.519 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 2, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 4 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 6, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 300, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 197, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 187, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 6, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 2, 2024-04-22T00:13:07.520 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 2, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 2, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:13:07.521 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 24, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 24, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T00:13:07.522 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4249236, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 17, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.084093989, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004946705 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.523 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 22, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 117, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 95, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 21, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 139, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 118, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 17, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 201, 2024-04-22T00:13:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 184, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 1, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 7, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 6, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 56156, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 32, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 11, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 22, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 9, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 26, 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T00:13:07.525 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000107319, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000107319 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.528 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.529 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.530 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.530 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T00:13:07.530 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.530 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T00:13:07.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002894806, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001447403 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.532 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T00:13:07.534 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.535 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.536 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.010672925, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005336462 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.537 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 3, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 3, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 652, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:13:07.538 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T00:13:07.539 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T00:13:07.540 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 401665, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 138, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T00:13:07.541 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9560, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 184, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 94917, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 388, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T00:13:07.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 96, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 68782, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 96, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 96, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.691983619, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.007208162 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T00:13:07.543 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 96, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 126 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 96, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 21, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 75, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 18, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 11, 2024-04-22T00:13:07.544 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 16, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 20, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:13:07.545 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T00:13:07.546 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 101, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 12, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 11, 2024-04-22T00:13:07.547 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 11, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 26, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 4 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T00:13:07.548 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 4, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 2, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 4, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 2, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:07.549 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4497, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 3030075, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4497, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 4497, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3030075, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.550 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 96, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 69096, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 96, 2024-04-22T00:13:07.551 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 69096, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:07.552 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 96, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 96, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 96, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 96, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:07.553 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 314, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 314, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.554 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x5634f58c6500": { 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 24, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 54932, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 24, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 17, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 54932, 2024-04-22T00:13:07.555 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:07.556 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T00:13:07.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:07 smithi111 ceph-mon[31543]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T00:13:08.376 INFO:tasks.check_counter:Daemon mds.i mds.exported=2 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds.imported=2 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=6 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=300 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:13:08.377 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-22T00:13:08.377 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T00:13:08.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid eab6f840-0039-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T00:13:08.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:08 smithi038 ceph-mon[27972]: pgmap v343: 129 pgs: 25 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 94 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:08.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:08 smithi203 ceph-mon[31431]: pgmap v343: 129 pgs: 25 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 94 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:08.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:08 smithi111 ceph-mon[31543]: pgmap v343: 129 pgs: 25 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 94 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:08.868 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T00:13:10.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:10 smithi038 ceph-mon[27972]: pgmap v344: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:10.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:10 smithi203 ceph-mon[31431]: pgmap v344: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:10.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:10 smithi111 ceph-mon[31543]: pgmap v344: 129 pgs: 55 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 55 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:11.199 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-22T00:13:11.199 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T00:13:11.199 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 2173, 2024-04-22T00:13:11.199 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2175, 2024-04-22T00:13:11.199 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 671097, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 691021, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.186045750, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.056854785, 2024-04-22T00:13:11.200 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.049499874, 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.066222949, 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2173, 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.078869831, 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000036295 2024-04-22T00:13:11.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T00:13:11.202 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 273, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 247, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1714172, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 143866, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 3, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.049224996, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.013658570, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.040775134, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.006058974, 2024-04-22T00:13:11.203 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 246, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.012103979, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000049203 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1713616, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 143168 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T00:13:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 7, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 178, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 3660, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 457726, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.010274494, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.006223496, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.002236724, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000021579, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 176, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.012895908, 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000073272 2024-04-22T00:13:11.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 457408 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.206 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3648, 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.098576189, 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000027021 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-22T00:13:11.207 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.208 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 3, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 3, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 3, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-22T00:13:11.209 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-22T00:13:11.210 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-22T00:13:11.211 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T00:13:11.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-22T00:13:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-22T00:13:11.214 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 24, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-22T00:13:11.215 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4249236, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 4246631, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 24 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 4, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 28, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 24, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 2, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 21, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 19, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 3, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 28, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 25, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-22T00:13:11.217 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 51608, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.218 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.219 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.220 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-22T00:13:11.223 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.224 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.225 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.230 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.231 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 1841, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-22T00:13:11.232 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T00:13:11.233 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T00:13:11.234 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 182037, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 48, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-22T00:13:11.235 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9560, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 184, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 15776, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 44, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-22T00:13:11.236 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 2176, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 2176, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2176, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 42.696988184, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.019621777 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.237 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2176, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2177 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 2176, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 2176, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 1398, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 776, 2024-04-22T00:13:11.238 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-04-22T00:13:11.239 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-22T00:13:11.240 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-22T00:13:11.241 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 101, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 12, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 4, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 4, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-22T00:13:11.242 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T00:13:11.243 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2449, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 2151608, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2449, 2024-04-22T00:13:11.244 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2449, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2151608, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:11.245 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2176, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 54937, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2176, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 54937, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2176, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2176, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2176, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2176, 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:11.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:11.248 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x558d1d4806e0": { 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-22T00:13:11.249 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-22T00:13:11.250 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-22T00:13:12.369 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T00:13:12.370 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-04-22T00:13:12.370 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T00:13:12.420 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T00:13:12.420 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:13:12.421 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T00:13:12.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:12 smithi038 ceph-mon[27972]: pgmap v345: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:12.522 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-22T00:13:12.522 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T00:13:12.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:12 smithi203 ceph-mon[31431]: pgmap v345: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:12.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:12 smithi111 ceph-mon[31543]: pgmap v345: 129 pgs: 65 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 41 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-22T00:13:12.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:13:12.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.824+0000 7f10f3839640 1 -- 172.21.15.38:0/1859959501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10e6a0 msgr2=0x7f10ec11cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:12.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.824+0000 7f10f3839640 1 --2- 172.21.15.38:0/1859959501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10e6a0 0x7f10ec11cf90 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009900 tx=0x7f10dc02bc40 comp rx=0 tx=0).stop 2024-04-22T00:13:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.827+0000 7f10f3839640 1 -- 172.21.15.38:0/1859959501 shutdown_connections 2024-04-22T00:13:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.827+0000 7f10f3839640 1 --2- 172.21.15.38:0/1859959501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10e6a0 0x7f10ec11cf90 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.827+0000 7f10f3839640 1 --2- 172.21.15.38:0/1859959501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec110310 0x7f10ec10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.827+0000 7f10f3839640 1 --2- 172.21.15.38:0/1859959501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec10fea0 0x7f10ec10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.827+0000 7f10f3839640 1 -- 172.21.15.38:0/1859959501 >> 172.21.15.38:0/1859959501 conn(0x7f10ec06c860 msgr2=0x7f10ec06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:12.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.834+0000 7f10f3839640 1 -- 172.21.15.38:0/1859959501 shutdown_connections 2024-04-22T00:13:12.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.835+0000 7f10f3839640 1 -- 172.21.15.38:0/1859959501 wait complete. 2024-04-22T00:13:12.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.835+0000 7f10f3839640 1 Processor -- start 2024-04-22T00:13:12.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 -- start start 2024-04-22T00:13:12.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:12.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec1a7380 0x7f10ec1ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:12.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec1abc40 0x7f10ec1ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:12.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec120a00 con 0x7f10ec10fea0 2024-04-22T00:13:12.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec120880 con 0x7f10ec1abc40 2024-04-22T00:13:12.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.836+0000 7f10f3839640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec120b80 con 0x7f10ec1a7380 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.837+0000 7f10f15d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.837+0000 7f10f15d5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48710/0 (socket says 172.21.15.38:48710) 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.837+0000 7f10f15d5640 1 -- 172.21.15.38:0/962040646 learned_addr learned my addr 172.21.15.38:0/962040646 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.837+0000 7f10f0dd4640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec1a7380 0x7f10ec1ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f1dd6640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec1abc40 0x7f10ec1ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f15d5640 1 -- 172.21.15.38:0/962040646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec1a7380 msgr2=0x7f10ec1ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f15d5640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec1a7380 0x7f10ec1ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:12.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f15d5640 1 -- 172.21.15.38:0/962040646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec1abc40 msgr2=0x7f10ec1ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:12.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f15d5640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec1abc40 0x7f10ec1ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:12.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.838+0000 7f10f15d5640 1 -- 172.21.15.38:0/962040646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec110700 con 0x7f10ec10fea0 2024-04-22T00:13:12.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.839+0000 7f10f15d5640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b4a0 tx=0x7f10e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:12.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.839+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8014070 con 0x7f10ec10fea0 2024-04-22T00:13:12.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.839+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ec1109f0 con 0x7f10ec10fea0 2024-04-22T00:13:12.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.839+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ec110f90 con 0x7f10ec10fea0 2024-04-22T00:13:12.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.840+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e8010af0 con 0x7f10ec10fea0 2024-04-22T00:13:12.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.840+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e801e060 con 0x7f10ec10fea0 2024-04-22T00:13:12.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.842+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d0005290 con 0x7f10ec10fea0 2024-04-22T00:13:12.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.844+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f10e8017070 con 0x7f10ec10fea0 2024-04-22T00:13:12.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.845+0000 7f10e27fc640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 0x7f10d80a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:12.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.845+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f10e80ce8c0 con 0x7f10ec10fea0 2024-04-22T00:13:12.848 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.850+0000 7f10f0dd4640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 0x7f10d80a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:12.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.851+0000 7f10f0dd4640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 0x7f10d80a5300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f10e400a400 tx=0x7f10e400d040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:12.852 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:12.854+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f10e809b1a0 con 0x7f10ec10fea0 2024-04-22T00:13:13.220 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.223+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10d0005850 con 0x7f10ec10fea0 2024-04-22T00:13:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.226+0000 7f10e27fc640 1 -- 172.21.15.38:0/962040646 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+25336 (secure 0 0 0) 0x7f10e809aaa0 con 0x7f10ec10fea0 2024-04-22T00:13:13.224 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:13:13.224 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":101,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:13:06.136993+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 msgr2=0x7f10d80a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 0x7f10d80a5300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f10e400a400 tx=0x7f10e400d040 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 msgr2=0x7f10ec1a6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b4a0 tx=0x7f10e800b960 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 shutdown_connections 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f10d80a2e40 0x7f10d80a5300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10ec1abc40 0x7f10ec1ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10ec1a7380 0x7f10ec1ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.229+0000 7f10f3839640 1 --2- 172.21.15.38:0/962040646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10ec10fea0 0x7f10ec1a6e40 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.230+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 >> 172.21.15.38:0/962040646 conn(0x7f10ec06c860 msgr2=0x7f10ec070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.230+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 shutdown_connections 2024-04-22T00:13:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.230+0000 7f10f3839640 1 -- 172.21.15.38:0/962040646 wait complete. 2024-04-22T00:13:13.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:54:54.123605+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:59:34.016885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:59:35.352517+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:59:38.827876+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-22T00:13:13.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T00:13:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.440+0000 7f2baea1b640 1 -- 172.21.15.38:0/2777719620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba810ff90 msgr2=0x7f2ba810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.440+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2777719620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba810ff90 0x7f2ba810da80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009890 tx=0x7f2b9c02bbb0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.441+0000 7f2baea1b640 1 -- 172.21.15.38:0/2777719620 shutdown_connections 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.441+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2777719620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba811af20 0x7f2ba811d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.441+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2777719620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 0x7f2ba810e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.441+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2777719620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba810ff90 0x7f2ba810da80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.441+0000 7f2baea1b640 1 -- 172.21.15.38:0/2777719620 >> 172.21.15.38:0/2777719620 conn(0x7f2ba806c850 msgr2=0x7f2ba806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.443+0000 7f2baea1b640 1 -- 172.21.15.38:0/2777719620 shutdown_connections 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.443+0000 7f2baea1b640 1 -- 172.21.15.38:0/2777719620 wait complete. 2024-04-22T00:13:13.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 Processor -- start 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 -- start start 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 0x7f2ba811a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 0x7f2ba81135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 0x7f2ba8113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8120970 con 0x7f2ba810dfc0 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81207f0 con 0x7f2ba811af20 2024-04-22T00:13:13.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.444+0000 7f2baea1b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8120af0 con 0x7f2ba810ff90 2024-04-22T00:13:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2ba77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 0x7f2ba81135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2ba77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 0x7f2ba81135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:50730/0 (socket says 172.21.15.38:50730) 2024-04-22T00:13:13.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2ba77fe640 1 -- 172.21.15.38:0/2417866278 learned_addr learned my addr 172.21.15.38:0/2417866278 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2bacfb8640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 0x7f2ba8113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2ba7fff640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 0x7f2ba811a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2bacfb8640 1 -- 172.21.15.38:0/2417866278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 msgr2=0x7f2ba81135e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2bacfb8640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 0x7f2ba81135e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2bacfb8640 1 -- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 msgr2=0x7f2ba811a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.445+0000 7f2bacfb8640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 0x7f2ba811a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.444 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.446+0000 7f2bacfb8640 1 -- 172.21.15.38:0/2417866278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba8114110 con 0x7f2ba811af20 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.446+0000 7f2bacfb8640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 0x7f2ba8113b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000be00 tx=0x7f2ba000bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.447+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0019070 con 0x7f2ba811af20 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.447+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ba0014920 con 0x7f2ba811af20 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.447+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00129e0 con 0x7f2ba811af20 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.447+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba8114400 con 0x7f2ba811af20 2024-04-22T00:13:13.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.447+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba8118360 con 0x7f2ba811af20 2024-04-22T00:13:13.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.449+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b94005290 con 0x7f2ba811af20 2024-04-22T00:13:13.448 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.450+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2ba001c070 con 0x7f2ba811af20 2024-04-22T00:13:13.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.451+0000 7f2ba57fa640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 0x7f2b900a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.452+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f2ba0099120 con 0x7f2ba811af20 2024-04-22T00:13:13.450 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.453+0000 7f2ba7fff640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 0x7f2b900a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.451 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.454+0000 7f2ba7fff640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 0x7f2b900a5340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009860 tx=0x7f2b9c009520 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:13.457 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.460+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2ba009b780 con 0x7f2ba811af20 2024-04-22T00:13:13.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.830+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2b94005850 con 0x7f2ba811af20 2024-04-22T00:13:13.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.831+0000 7f2ba57fa640 1 -- 172.21.15.38:0/2417866278 <== mon.2 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f2ba009b080 con 0x7f2ba811af20 2024-04-22T00:13:13.829 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-04-22T00:13:13.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 msgr2=0x7f2b900a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 0x7f2b900a5340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009860 tx=0x7f2b9c009520 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 msgr2=0x7f2ba8113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 0x7f2ba8113b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000be00 tx=0x7f2ba000bef0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 shutdown_connections 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2b900a2e80 0x7f2b900a5340 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009860 tx=0x7f2b9c009520 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ba811af20 0x7f2ba8113b20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ba810ff90 0x7f2ba81135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 --2- 172.21.15.38:0/2417866278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ba810dfc0 0x7f2ba811a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 >> 172.21.15.38:0/2417866278 conn(0x7f2ba806c850 msgr2=0x7f2ba811c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:13.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.835+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 shutdown_connections 2024-04-22T00:13:13.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.836+0000 7f2baea1b640 1 -- 172.21.15.38:0/2417866278 wait complete. 2024-04-22T00:13:13.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T00:13:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 -- 172.21.15.38:0/2567846640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b14107730 msgr2=0x7f3b14107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/2567846640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b14107730 0x7f3b14107b10 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009930 tx=0x7f3b1002bc30 comp rx=0 tx=0).stop 2024-04-22T00:13:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 -- 172.21.15.38:0/2567846640 shutdown_connections 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/2567846640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b1406a780 0x7f3b1410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/2567846640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 0x7f3b1406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/2567846640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b14107730 0x7f3b14107b10 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 -- 172.21.15.38:0/2567846640 >> 172.21.15.38:0/2567846640 conn(0x7f3b140776a0 msgr2=0x7f3b14077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 -- 172.21.15.38:0/2567846640 shutdown_connections 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.981+0000 7f3b1b81b640 1 -- 172.21.15.38:0/2567846640 wait complete. 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 Processor -- start 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 -- start start 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 0x7f3b141981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 0x7f3b1419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b14111b40 con 0x7f3b1406a780 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141119c0 con 0x7f3b14069dc0 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b1b81b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b14111cc0 con 0x7f3b14107730 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b18db6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b18db6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48012/0 (socket says 172.21.15.38:48012) 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b18db6640 1 -- 172.21.15.38:0/1069055015 learned_addr learned my addr 172.21.15.38:0/1069055015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b195b7640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 0x7f3b141981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.982+0000 7f3b19db8640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 0x7f3b1419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 -- 172.21.15.38:0/1069055015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 msgr2=0x7f3b1419ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 0x7f3b1419ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 -- 172.21.15.38:0/1069055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 msgr2=0x7f3b141981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 0x7f3b141981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 -- 172.21.15.38:0/1069055015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1419d190 con 0x7f3b1406a780 2024-04-22T00:13:13.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b19db8640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 0x7f3b1419ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b18db6640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400d950 tx=0x7f3b0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04017070 con 0x7f3b1406a780 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b04009c30 con 0x7f3b1406a780 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1419d480 con 0x7f3b1406a780 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0400f6b0 con 0x7f3b1406a780 2024-04-22T00:13:13.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.983+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b141a4d10 con 0x7f3b1406a780 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.984+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b14102e60 con 0x7f3b1406a780 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.985+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f3b040109d0 con 0x7f3b1406a780 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.986+0000 7f3b0a7fc640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 0x7f3b000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.986+0000 7f3b195b7640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 0x7f3b000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.986+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f3b040cf490 con 0x7f3b1406a780 2024-04-22T00:13:13.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.986+0000 7f3b195b7640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 0x7f3b000a51c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005ab0 tx=0x7f3b10005fd0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:13.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:13.990+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3b04009de0 con 0x7f3b1406a780 2024-04-22T00:13:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.259+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3b14109c90 con 0x7f3b1406a780 2024-04-22T00:13:14.257 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.260+0000 7f3b0a7fc640 1 -- 172.21.15.38:0/1069055015 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7f3b0409bcc0 con 0x7f3b1406a780 2024-04-22T00:13:14.258 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:13:14.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.265+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 msgr2=0x7f3b000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.265+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 0x7f3b000a51c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005ab0 tx=0x7f3b10005fd0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.265+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 msgr2=0x7f3b14198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.265+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400d950 tx=0x7f3b0400de10 comp rx=0 tx=0).stop 2024-04-22T00:13:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 shutdown_connections 2024-04-22T00:13:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f3b000a2d00 0x7f3b000a51c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b14107730 0x7f3b1419ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b1406a780 0x7f3b14198730 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 --2- 172.21.15.38:0/1069055015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b14069dc0 0x7f3b141981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 >> 172.21.15.38:0/1069055015 conn(0x7f3b140776a0 msgr2=0x7f3b140fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:14.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.268+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 shutdown_connections 2024-04-22T00:13:14.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.269+0000 7f3b1b81b640 1 -- 172.21.15.38:0/1069055015 wait complete. 2024-04-22T00:13:14.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T00:13:14.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:14 smithi038 ceph-mon[27972]: pgmap v346: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:13:14.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/962040646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:14.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:14 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2417866278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:13:14.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 -- 172.21.15.38:0/139114143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8410ff90 msgr2=0x7fdb8410da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 --2- 172.21.15.38:0/139114143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8410ff90 0x7fdb8410da80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009ca0 tx=0x7fdb7c02c120 comp rx=0 tx=0).stop 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 -- 172.21.15.38:0/139114143 shutdown_connections 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 --2- 172.21.15.38:0/139114143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8411af20 0x7fdb8411d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 --2- 172.21.15.38:0/139114143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410dfc0 0x7fdb8410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 --2- 172.21.15.38:0/139114143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8410ff90 0x7fdb8410da80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.471+0000 7fdb88b75640 1 -- 172.21.15.38:0/139114143 >> 172.21.15.38:0/139114143 conn(0x7fdb8406c850 msgr2=0x7fdb8406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.472+0000 7fdb88b75640 1 -- 172.21.15.38:0/139114143 shutdown_connections 2024-04-22T00:13:14.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.472+0000 7fdb88b75640 1 -- 172.21.15.38:0/139114143 wait complete. 2024-04-22T00:13:14.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.473+0000 7fdb88b75640 1 Processor -- start 2024-04-22T00:13:14.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.473+0000 7fdb88b75640 1 -- start start 2024-04-22T00:13:14.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.474+0000 7fdb88b75640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8410dfc0 0x7fdb841a6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:14.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.474+0000 7fdb88b75640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410ff90 0x7fdb841a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:14.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.474+0000 7fdb88b75640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:14.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.474+0000 7fdb88b75640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841208e0 con 0x7fdb8410ff90 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.474+0000 7fdb82d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:46866/0 (socket says 172.21.15.38:46866) 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 -- 172.21.15.38:0/2860856848 learned_addr learned my addr 172.21.15.38:0/2860856848 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84120760 con 0x7fdb8410dfc0 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84120a60 con 0x7fdb8411af20 2024-04-22T00:13:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb81d9b640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410ff90 0x7fdb841a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb8259c640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8410dfc0 0x7fdb841a6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 -- 172.21.15.38:0/2860856848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8410dfc0 msgr2=0x7fdb841a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8410dfc0 0x7fdb841a6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 -- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410ff90 msgr2=0x7fdb841a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410ff90 0x7fdb841a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.475+0000 7fdb82d9d640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb841abe50 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb82d9d640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400d750 tx=0x7fdb7400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74017070 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb841ac140 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb841b39d0 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb74005500 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.476+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74004d90 con 0x7fdb8411af20 2024-04-22T00:13:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.477+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb64005290 con 0x7fdb8411af20 2024-04-22T00:13:14.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.481+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fdb74009bd0 con 0x7fdb8411af20 2024-04-22T00:13:14.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.481+0000 7fdb737fe640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 0x7fdb6c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:14.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.482+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fdb740cef70 con 0x7fdb8411af20 2024-04-22T00:13:14.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.482+0000 7fdb8259c640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 0x7fdb6c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:14.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.482+0000 7fdb8259c640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 0x7fdb6c0a53c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c006010 tx=0x7fdb7c005f80 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:14.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.485+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdb7409b840 con 0x7fdb8411af20 2024-04-22T00:13:14.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:14 smithi203 ceph-mon[31431]: pgmap v346: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:13:14.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/962040646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:14.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:14 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2417866278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:13:14.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:14 smithi111 ceph-mon[31543]: pgmap v346: 129 pgs: 54 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 58 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T00:13:14.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/962040646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:14.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:14 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2417866278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T00:13:14.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.843+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdb64005850 con 0x7fdb8411af20 2024-04-22T00:13:14.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.844+0000 7fdb737fe640 1 -- 172.21.15.38:0/2860856848 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) ==== 100+0+11 (secure 0 0 0) 0x7fdb7409b140 con 0x7fdb8411af20 2024-04-22T00:13:14.841 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-04-22T00:13:14.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 msgr2=0x7fdb6c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 0x7fdb6c0a53c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c006010 tx=0x7fdb7c005f80 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 msgr2=0x7fdb841ab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400d750 tx=0x7fdb7400dc10 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 shutdown_connections 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fdb6c0a2f00 0x7fdb6c0a53c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb8411af20 0x7fdb841ab720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb8410ff90 0x7fdb841a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 --2- 172.21.15.38:0/2860856848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb8410dfc0 0x7fdb841a6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.847+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 >> 172.21.15.38:0/2860856848 conn(0x7fdb8406c850 msgr2=0x7fdb84119a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:14.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.848+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 shutdown_connections 2024-04-22T00:13:14.846 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:14.848+0000 7fdb88b75640 1 -- 172.21.15.38:0/2860856848 wait complete. 2024-04-22T00:13:14.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.055+0000 7f39e209d640 1 -- 172.21.15.38:0/174613609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 msgr2=0x7f39dc10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 --2- 172.21.15.38:0/174613609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc10d4e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009890 tx=0x7f39cc02bb90 comp rx=0 tx=0).stop 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 -- 172.21.15.38:0/174613609 shutdown_connections 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 --2- 172.21.15.38:0/174613609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc10d4e0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 --2- 172.21.15.38:0/174613609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 --2- 172.21.15.38:0/174613609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39dc1066b0 0x7f39dc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 -- 172.21.15.38:0/174613609 >> 172.21.15.38:0/174613609 conn(0x7f39dc077740 msgr2=0x7f39dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.056+0000 7f39e209d640 1 -- 172.21.15.38:0/174613609 shutdown_connections 2024-04-22T00:13:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.057+0000 7f39e209d640 1 -- 172.21.15.38:0/174613609 wait complete. 2024-04-22T00:13:15.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.058+0000 7f39e209d640 1 Processor -- start 2024-04-22T00:13:15.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 -- start start 2024-04-22T00:13:15.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc19d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39dc1066b0 0x7f39dc195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc10f960 con 0x7f39dc100fd0 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc10f7e0 con 0x7f39dc1066b0 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.064+0000 7f39e209d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc10fae0 con 0x7f39dc100610 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.065+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.38:46880/0 (socket says 172.21.15.38:46880) 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 -- 172.21.15.38:0/2762938626 learned_addr learned my addr 172.21.15.38:0/2762938626 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39daffd640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc19d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 -- 172.21.15.38:0/2762938626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39dc1066b0 msgr2=0x7f39dc195e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39dc1066b0 0x7f39dc195e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 -- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 msgr2=0x7f39dc19d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc19d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39db7fe640 1 -- 172.21.15.38:0/2762938626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc196730 con 0x7f39dc100610 2024-04-22T00:13:15.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.066+0000 7f39daffd640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc19d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:13:15.067 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.070+0000 7f39db7fe640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c3a0 tx=0x7f39d400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:15.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.070+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d401b070 con 0x7f39dc100610 2024-04-22T00:13:15.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.070+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39d4003e50 con 0x7f39dc100610 2024-04-22T00:13:15.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.070+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d400b890 con 0x7f39dc100610 2024-04-22T00:13:15.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.071+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39dc196a20 con 0x7f39dc100610 2024-04-22T00:13:15.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.071+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39dc19a980 con 0x7f39dc100610 2024-04-22T00:13:15.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.073+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39dc101d50 con 0x7f39dc100610 2024-04-22T00:13:15.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.074+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f39d401e070 con 0x7f39dc100610 2024-04-22T00:13:15.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.075+0000 7f39d8ff9640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 0x7f39c40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.075+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f39d40cf940 con 0x7f39dc100610 2024-04-22T00:13:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.080+0000 7f39daffd640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 0x7f39c40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.079 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.081+0000 7f39daffd640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 0x7f39c40a53c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39dc197400 tx=0x7f39d0008040 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:15.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.081+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f39d409c110 con 0x7f39dc100610 2024-04-22T00:13:15.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.397+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f39dc19b4a0 con 0x7f39dc100610 2024-04-22T00:13:15.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.398+0000 7f39d8ff9640 1 -- 172.21.15.38:0/2762938626 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v101) ==== 103+0+11 (secure 0 0 0) 0x7f39d409ba10 con 0x7f39dc100610 2024-04-22T00:13:15.396 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 msgr2=0x7f39c40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 0x7f39c40a53c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39dc197400 tx=0x7f39d0008040 comp rx=0 tx=0).stop 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 msgr2=0x7f39dc19cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c3a0 tx=0x7f39d400c860 comp rx=0 tx=0).stop 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 shutdown_connections 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f39c40a2f00 0x7f39c40a53c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39dc1066b0 0x7f39dc195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc100fd0 0x7f39dc19d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.398 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 --2- 172.21.15.38:0/2762938626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39dc100610 0x7f39dc19cbc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 >> 172.21.15.38:0/2762938626 conn(0x7f39dc077740 msgr2=0x7f39dc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:15.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 shutdown_connections 2024-04-22T00:13:15.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.400+0000 7f39e209d640 1 -- 172.21.15.38:0/2762938626 wait complete. 2024-04-22T00:13:15.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T00:13:15.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:15 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1069055015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:15 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2860856848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.515+0000 7f72283af640 1 -- 172.21.15.38:0/3766926990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72201007b0 msgr2=0x7f7220100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.515+0000 7f72283af640 1 --2- 172.21.15.38:0/3766926990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72201007b0 0x7f7220100b90 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f721c009900 tx=0x7f721c02bd60 comp rx=0 tx=0).stop 2024-04-22T00:13:15.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- 172.21.15.38:0/3766926990 shutdown_connections 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- 172.21.15.38:0/3766926990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7220105ad0 0x7f7220105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- 172.21.15.38:0/3766926990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 0x7f7220105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- 172.21.15.38:0/3766926990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72201007b0 0x7f7220100b90 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- 172.21.15.38:0/3766926990 >> 172.21.15.38:0/3766926990 conn(0x7f7220077740 msgr2=0x7f7220077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- 172.21.15.38:0/3766926990 shutdown_connections 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- 172.21.15.38:0/3766926990 wait complete. 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 Processor -- start 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- start start 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72201007b0 0x7f722019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 0x7f722019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7220118140 con 0x7f7220105ad0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7220117fc0 con 0x7f72201010d0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.516+0000 7f72283af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72201182c0 con 0x7f72201007b0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48064/0 (socket says 172.21.15.38:48064) 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 -- 172.21.15.38:0/1401143242 learned_addr learned my addr 172.21.15.38:0/1401143242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722594a640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 0x7f722019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722614b640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72201007b0 0x7f722019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 -- 172.21.15.38:0/1401143242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72201007b0 msgr2=0x7f722019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72201007b0 0x7f722019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 -- 172.21.15.38:0/1401143242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 msgr2=0x7f722019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 0x7f722019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 -- 172.21.15.38:0/1401143242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72201a3670 con 0x7f7220105ad0 2024-04-22T00:13:15.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f722694c640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f721800eaa0 tx=0x7f721800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721800cdb0 con 0x7f7220105ad0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7218005020 con 0x7f7220105ad0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72201a3960 con 0x7f7220105ad0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180116b0 con 0x7f7220105ad0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.517+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72201ab1f0 con 0x7f7220105ad0 2024-04-22T00:13:15.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.518+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7220068560 con 0x7f7220105ad0 2024-04-22T00:13:15.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.519+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7218005560 con 0x7f7220105ad0 2024-04-22T00:13:15.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.520+0000 7f72177fe640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 0x7f720c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:15.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.520+0000 7f722614b640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 0x7f720c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:15.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.520+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f72180cf9d0 con 0x7f7220105ad0 2024-04-22T00:13:15.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.520+0000 7f722614b640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 0x7f720c0a5240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f721c006010 tx=0x7f721c000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:15.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.524+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7218016030 con 0x7f7220105ad0 2024-04-22T00:13:15.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:15 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1069055015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:15 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2860856848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.581 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:15 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1069055015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:15 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2860856848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T00:13:15.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.726+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f72201ab4b0 con 0x7f7220105ad0 2024-04-22T00:13:15.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.727+0000 7f72177fe640 1 -- 172.21.15.38:0/1401143242 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v101) ==== 81+0+940 (secure 0 0 0) 0x7f721809c1a0 con 0x7f7220105ad0 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1468209311 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/2340980806 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/2340980806 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4082720653 2024-04-22T23:52:25.561345+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/914846331 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2416463439 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1866190732 2024-04-22T23:51:57.369607+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2899039909 2024-04-23T00:02:04.818169+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/2824973701 2024-04-22T23:52:25.561345+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1458203717 2024-04-22T23:51:57.369607+0000 2024-04-22T00:13:15.725 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/71250327 2024-04-22T23:51:57.369607+0000 2024-04-22T00:13:15.726 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/71250327 2024-04-22T23:51:57.369607+0000 2024-04-22T00:13:15.726 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/247200712 2024-04-22T23:52:25.561345+0000 2024-04-22T00:13:15.726 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4200150941 2024-04-22T23:51:57.369607+0000 2024-04-22T00:13:15.726 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2769068060 2024-04-22T23:52:25.561345+0000 2024-04-22T00:13:15.726 INFO:teuthology.orchestra.run.smithi038.stderr:listed 16 entries 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 msgr2=0x7f720c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 0x7f720c0a5240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f721c006010 tx=0x7f721c000bc0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 msgr2=0x7f72201a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f721800eaa0 tx=0x7f721800ef60 comp rx=0 tx=0).stop 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 shutdown_connections 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f720c0a2d80 0x7f720c0a5240 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7220105ad0 0x7f72201a2f40 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72201010d0 0x7f722019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 --2- 172.21.15.38:0/1401143242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72201007b0 0x7f722019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:15.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 >> 172.21.15.38:0/1401143242 conn(0x7f7220077740 msgr2=0x7f7220103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:15.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 shutdown_connections 2024-04-22T00:13:15.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:15.729+0000 7f72283af640 1 -- 172.21.15.38:0/1401143242 wait complete. 2024-04-22T00:13:15.738 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T00:13:15.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:13:15.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T00:13:15.840 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:13:15.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:13:15.841 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T00:13:15.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:13:15.895 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-04-22T00:13:15.931 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T00:13:15.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:13:15.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T00:13:15.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T00:13:15.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-04-22T00:13:16.099 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T00:13:16.127 INFO:tasks.cephadm:Teardown begin 2024-04-22T00:13:16.127 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T00:13:16.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:13:16.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 -- 172.21.15.38:0/3223502672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24101560 msgr2=0x7f7c241019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 --2- 172.21.15.38:0/3223502672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24101560 0x7f7c241019e0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009900 tx=0x7f7c1802bc30 comp rx=0 tx=0).stop 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 -- 172.21.15.38:0/3223502672 shutdown_connections 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 --2- 172.21.15.38:0/3223502672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 0x7f7c2410e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 --2- 172.21.15.38:0/3223502672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24101560 0x7f7c241019e0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.233+0000 7f7c28da2640 1 --2- 172.21.15.38:0/3223502672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24107620 0x7f7c24107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 -- 172.21.15.38:0/3223502672 >> 172.21.15.38:0/3223502672 conn(0x7f7c240fd070 msgr2=0x7f7c240ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 -- 172.21.15.38:0/3223502672 shutdown_connections 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 -- 172.21.15.38:0/3223502672 wait complete. 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 Processor -- start 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 -- start start 2024-04-22T00:13:16.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.234+0000 7f7c28da2640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24101560 0x7f7c24197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 0x7f7c241982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c24111950 con 0x7f7c24107620 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7c241117d0 con 0x7f7c24101560 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c24111ad0 con 0x7f7c24101f20 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48086/0 (socket says 172.21.15.38:48086) 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c237fe640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24101560 0x7f7c24197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 -- 172.21.15.38:0/2256337937 learned_addr learned my addr 172.21.15.38:0/2256337937 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c22ffd640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 0x7f7c241982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 -- 172.21.15.38:0/2256337937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 msgr2=0x7f7c241982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 0x7f7c241982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 -- 172.21.15.38:0/2256337937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24101560 msgr2=0x7f7c24197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24101560 0x7f7c24197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 -- 172.21.15.38:0/2256337937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2419cda0 con 0x7f7c24107620 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c23fff640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00eb10 tx=0x7f7c1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c004b20 con 0x7f7c24107620 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c1c004cc0 con 0x7f7c24107620 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2419d090 con 0x7f7c24107620 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c010760 con 0x7f7c24107620 2024-04-22T00:13:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.235+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c241a4920 con 0x7f7c24107620 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.237+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c24068480 con 0x7f7c24107620 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.238+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7c1c005020 con 0x7f7c24107620 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.238+0000 7f7c20ff9640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 0x7f7c0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.239+0000 7f7c237fe640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 0x7f7c0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.239+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f7c1c0cfee0 con 0x7f7c24107620 2024-04-22T00:13:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.239+0000 7f7c237fe640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 0x7f7c0c0a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400bbe0 tx=0x7f7c1400b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.243+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7c1c09c710 con 0x7f7c24107620 2024-04-22T00:13:16.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:16 smithi038 ceph-mon[27972]: pgmap v347: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:13:16.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:16 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2762938626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:13:16.389 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:16 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1401143242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T00:13:16.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.471+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2402d170 con 0x7f7c24107620 2024-04-22T00:13:16.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.472+0000 7f7c20ff9640 1 -- 172.21.15.38:0/2256337937 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11935 (secure 0 0 0) 0x7f7c1c0bc0d0 con 0x7f7c24107620 2024-04-22T00:13:16.470 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:13:16.470 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:12:03.690688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:13:16.470 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 42 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 msgr2=0x7f7c0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 0x7f7c0c0a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400bbe0 tx=0x7f7c1400b390 comp rx=0 tx=0).stop 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 msgr2=0x7f7c2419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00eb10 tx=0x7f7c1c00c2d0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 shutdown_connections 2024-04-22T00:13:16.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7c0c0a2d00 0x7f7c0c0a51c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c24107620 0x7f7c2419c670 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c24101f20 0x7f7c241982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 --2- 172.21.15.38:0/2256337937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c24101560 0x7f7c24197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 >> 172.21.15.38:0/2256337937 conn(0x7f7c240fd070 msgr2=0x7f7c240fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 shutdown_connections 2024-04-22T00:13:16.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.474+0000 7f7c28da2640 1 -- 172.21.15.38:0/2256337937 wait complete. 2024-04-22T00:13:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:16 smithi111 ceph-mon[31543]: pgmap v347: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:13:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:16 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2762938626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:13:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:16 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1401143242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T00:13:16.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T00:13:16.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:16 smithi203 ceph-mon[31431]: pgmap v347: 129 pgs: 21 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 99 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T00:13:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:16 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2762938626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-22T00:13:16.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:16 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1401143242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 -- 172.21.15.38:0/3433811802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c107750 msgr2=0x7fcc4c107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 --2- 172.21.15.38:0/3433811802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c107750 0x7fcc4c107b30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009900 tx=0x7fcc4802bcd0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 -- 172.21.15.38:0/3433811802 shutdown_connections 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 --2- 172.21.15.38:0/3433811802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c102160 0x7fcc4c10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 --2- 172.21.15.38:0/3433811802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 --2- 172.21.15.38:0/3433811802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c107750 0x7fcc4c107b30 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 -- 172.21.15.38:0/3433811802 >> 172.21.15.38:0/3433811802 conn(0x7fcc4c0788d0 msgr2=0x7fcc4c078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 -- 172.21.15.38:0/3433811802 shutdown_connections 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 -- 172.21.15.38:0/3433811802 wait complete. 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.591+0000 7fcc533b8640 1 Processor -- start 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 -- start start 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c107750 0x7fcc4c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c111b10 con 0x7fcc4c102160 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c111990 con 0x7fcc4c107750 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc533b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c111c90 con 0x7fcc4c1017c0 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48100/0 (socket says 172.21.15.38:48100) 2024-04-22T00:13:16.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 -- 172.21.15.38:0/1542065893 learned_addr learned my addr 172.21.15.38:0/1542065893 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc51955640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c107750 0x7fcc4c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc51154640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 -- 172.21.15.38:0/1542065893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 msgr2=0x7fcc4c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 -- 172.21.15.38:0/1542065893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c107750 msgr2=0x7fcc4c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c107750 0x7fcc4c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 -- 172.21.15.38:0/1542065893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c19f1a0 con 0x7fcc4c102160 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc51154640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.592+0000 7fcc50953640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00e9f0 tx=0x7fcc3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.593+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c00cd80 con 0x7fcc4c102160 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.593+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc3c010910 con 0x7fcc4c102160 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.593+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c19f490 con 0x7fcc4c102160 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.593+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c018990 con 0x7fcc4c102160 2024-04-22T00:13:16.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.593+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1a6d20 con 0x7fcc4c102160 2024-04-22T00:13:16.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.594+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c0689b0 con 0x7fcc4c102160 2024-04-22T00:13:16.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.595+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7fcc3c010ac0 con 0x7fcc4c102160 2024-04-22T00:13:16.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.596+0000 7fcc427fc640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 0x7fcc380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.596+0000 7fcc51154640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 0x7fcc380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.596+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fcc3c0cfc40 con 0x7fcc4c102160 2024-04-22T00:13:16.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.596+0000 7fcc51154640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 0x7fcc380a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc4802efd0 tx=0x7fcc48000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.601+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcc3c09c410 con 0x7fcc4c102160 2024-04-22T00:13:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.829+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc4c1a6fc0 con 0x7fcc4c102160 2024-04-22T00:13:16.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.830+0000 7fcc427fc640 1 -- 172.21.15.38:0/1542065893 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11935 (secure 0 0 0) 0x7fcc3c09bd10 con 0x7fcc4c102160 2024-04-22T00:13:16.828 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:13:16.828 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14532,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/425693237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":425693237},{"type":"v1","addr":"172.21.15.38:6835","nonce":425693237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/815557617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":815557617},{"type":"v1","addr":"172.21.15.38:6837","nonce":815557617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:59:36.029248+0000","modified":"2024-04-22T00:12:03.690688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24487,"mds_2":24460,"mds_3":24455,"mds_4":24448},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6839/3894275312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":3894275312},{"type":"v1","addr":"172.21.15.38:6839","nonce":3894275312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14634":{"gid":14634,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6841/3401657425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":3401657425},{"type":"v1","addr":"172.21.15.38:6841","nonce":3401657425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24395":{"gid":24395,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6833/618180230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":618180230},{"type":"v1","addr":"172.21.15.111:6833","nonce":618180230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24409":{"gid":24409,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1279747281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1279747281},{"type":"v1","addr":"172.21.15.203:6833","nonce":1279747281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.111:6835/3450283861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3450283861},{"type":"v1","addr":"172.21.15.111:6835","nonce":3450283861}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6835/521403621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":521403621},{"type":"v1","addr":"172.21.15.203:6835","nonce":521403621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.111:6837/782972998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":782972998},{"type":"v1","addr":"172.21.15.111:6837","nonce":782972998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/1809489419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1809489419},{"type":"v1","addr":"172.21.15.203:6837","nonce":1809489419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/3602809941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3602809941},{"type":"v1","addr":"172.21.15.203:6839","nonce":3602809941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6839/3890542559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":3890542559},{"type":"v1","addr":"172.21.15.111:6839","nonce":3890542559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24455,24448,24460,24500,24487]},"id":1}]} 2024-04-22T00:13:16.828 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 42 2024-04-22T00:13:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 msgr2=0x7fcc380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 0x7fcc380a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc4802efd0 tx=0x7fcc48000bc0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 msgr2=0x7fcc4c19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00e9f0 tx=0x7fcc3c00eeb0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 shutdown_connections 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7fcc380a2d00 0x7fcc380a51c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc4c107750 0x7fcc4c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc4c102160 0x7fcc4c19a6f0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 --2- 172.21.15.38:0/1542065893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc4c1017c0 0x7fcc4c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 >> 172.21.15.38:0/1542065893 conn(0x7fcc4c0788d0 msgr2=0x7fcc4c10cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 shutdown_connections 2024-04-22T00:13:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.832+0000 7fcc533b8640 1 -- 172.21.15.38:0/1542065893 wait complete. 2024-04-22T00:13:16.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T00:13:16.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 -- 172.21.15.38:0/3749104727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a0100cb0 msgr2=0x7f40a01063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3749104727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a0100cb0 0x7f40a01063e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f4094009900 tx=0x7f409402bc30 comp rx=0 tx=0).stop 2024-04-22T00:13:16.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 -- 172.21.15.38:0/3749104727 shutdown_connections 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3749104727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0106ac0 0x7f40a010e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3749104727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a0100cb0 0x7f40a01063e0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3749104727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0100300 0x7f40a01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 -- 172.21.15.38:0/3749104727 >> 172.21.15.38:0/3749104727 conn(0x7f40a0075920 msgr2=0x7f40a0075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 -- 172.21.15.38:0/3749104727 shutdown_connections 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.947+0000 7f40a66f4640 1 -- 172.21.15.38:0/3749104727 wait complete. 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 Processor -- start 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 -- start start 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 0x7f40a0198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0106ac0 0x7f40a01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0111b90 con 0x7f40a019c950 2024-04-22T00:13:16.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0111a10 con 0x7f40a0100300 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a66f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40a0111d10 con 0x7f40a0106ac0 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a4c91640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a4c91640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48116/0 (socket says 172.21.15.38:48116) 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.948+0000 7f40a4c91640 1 -- 172.21.15.38:0/3539953012 learned_addr learned my addr 172.21.15.38:0/3539953012 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409ffff640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 0x7f40a0198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409f7fe640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0106ac0 0x7f40a01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 -- 172.21.15.38:0/3539953012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0106ac0 msgr2=0x7f40a01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0106ac0 0x7f40a01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 -- 172.21.15.38:0/3539953012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 msgr2=0x7f40a0198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 0x7f40a0198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 -- 172.21.15.38:0/3539953012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a019d440 con 0x7f40a019c950 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409ffff640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 0x7f40a0198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:13:16.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a4c91640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4098009fd0 tx=0x7f409800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098017070 con 0x7f40a019c950 2024-04-22T00:13:16.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a019d730 con 0x7f40a019c950 2024-04-22T00:13:16.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4098003e50 con 0x7f40a019c950 2024-04-22T00:13:16.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40980106c0 con 0x7f40a019c950 2024-04-22T00:13:16.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.949+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a01a4ea0 con 0x7f40a019c950 2024-04-22T00:13:16.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.950+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a00689b0 con 0x7f40a019c950 2024-04-22T00:13:16.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.952+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f4098003970 con 0x7f40a019c950 2024-04-22T00:13:16.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.952+0000 7f409d7fa640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 0x7f40880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:16.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.952+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f40980cf5b0 con 0x7f40a019c950 2024-04-22T00:13:16.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.952+0000 7f409ffff640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 0x7f40880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:16.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.953+0000 7f409ffff640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 0x7f40880a51c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4090005fd0 tx=0x7f409000b5f0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:16.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:16.956+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f409809bde0 con 0x7f40a019c950 2024-04-22T00:13:17.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.142+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40a01993e0 con 0x7f40a019c950 2024-04-22T00:13:17.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.144+0000 7f409d7fa640 1 -- 172.21.15.38:0/3539953012 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+25336 (secure 0 0 0) 0x7f409809b6e0 con 0x7f40a019c950 2024-04-22T00:13:17.142 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-22T00:13:17.142 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":101,"fsid":"eab6f840-0039-11ef-bc93-c7b262605968","created":"2024-04-21T23:51:31.159714+0000","modified":"2024-04-22T00:13:06.136993+0000","last_up_change":"2024-04-21T23:58:47.421786+0000","last_in_change":"2024-04-21T23:58:26.889709+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:54:54.123605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:59:34.016885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:59:35.352517+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:59:38.827876+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"41c96fae-afbf-4345-b838-bfdab30012d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6803","nonce":1038840841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6805","nonce":1038840841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6809","nonce":1038840841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1038840841},{"type":"v1","addr":"172.21.15.38:6807","nonce":1038840841}]},"public_addr":"172.21.15.38:6803/1038840841","cluster_addr":"172.21.15.38:6805/1038840841","heartbeat_back_addr":"172.21.15.38:6809/1038840841","heartbeat_front_addr":"172.21.15.38:6807/1038840841","state":["exists","up"]},{"osd":1,"uuid":"87fe4b77-db92-4f21-b74f-76c1aec76f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6801","nonce":4079345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6803","nonce":4079345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6807","nonce":4079345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":4079345941},{"type":"v1","addr":"172.21.15.111:6805","nonce":4079345941}]},"public_addr":"172.21.15.111:6801/4079345941","cluster_addr":"172.21.15.111:6803/4079345941","heartbeat_back_addr":"172.21.15.111:6807/4079345941","heartbeat_front_addr":"172.21.15.111:6805/4079345941","state":["exists","up"]},{"osd":2,"uuid":"d2e0ed53-ee4a-432b-9147-1e4df2da6dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6801","nonce":2981966976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6803","nonce":2981966976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6807","nonce":2981966976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2981966976},{"type":"v1","addr":"172.21.15.203:6805","nonce":2981966976}]},"public_addr":"172.21.15.203:6801/2981966976","cluster_addr":"172.21.15.203:6803/2981966976","heartbeat_back_addr":"172.21.15.203:6807/2981966976","heartbeat_front_addr":"172.21.15.203:6805/2981966976","state":["exists","up"]},{"osd":3,"uuid":"28f56df4-c518-4ecc-bbfa-51b25a0104d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6811","nonce":1951374002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6813","nonce":1951374002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6817","nonce":1951374002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":1951374002},{"type":"v1","addr":"172.21.15.38:6815","nonce":1951374002}]},"public_addr":"172.21.15.38:6811/1951374002","cluster_addr":"172.21.15.38:6813/1951374002","heartbeat_back_addr":"172.21.15.38:6817/1951374002","heartbeat_front_addr":"172.21.15.38:6815/1951374002","state":["exists","up"]},{"osd":4,"uuid":"cab4f9b6-2a83-4faf-87a1-1f5b6baaae91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6809","nonce":753751366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6811","nonce":753751366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6815","nonce":753751366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":753751366},{"type":"v1","addr":"172.21.15.111:6813","nonce":753751366}]},"public_addr":"172.21.15.111:6809/753751366","cluster_addr":"172.21.15.111:6811/753751366","heartbeat_back_addr":"172.21.15.111:6815/753751366","heartbeat_front_addr":"172.21.15.111:6813/753751366","state":["exists","up"]},{"osd":5,"uuid":"89520131-b2cc-4385-a99f-e77fbde2670b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6809","nonce":1090281315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6811","nonce":1090281315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6815","nonce":1090281315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":1090281315},{"type":"v1","addr":"172.21.15.203:6813","nonce":1090281315}]},"public_addr":"172.21.15.203:6809/1090281315","cluster_addr":"172.21.15.203:6811/1090281315","heartbeat_back_addr":"172.21.15.203:6815/1090281315","heartbeat_front_addr":"172.21.15.203:6813/1090281315","state":["exists","up"]},{"osd":6,"uuid":"fe082d0b-2acc-446e-ba14-845d68b911e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6819","nonce":1747049129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6821","nonce":1747049129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6825","nonce":1747049129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":1747049129},{"type":"v1","addr":"172.21.15.38:6823","nonce":1747049129}]},"public_addr":"172.21.15.38:6819/1747049129","cluster_addr":"172.21.15.38:6821/1747049129","heartbeat_back_addr":"172.21.15.38:6825/1747049129","heartbeat_front_addr":"172.21.15.38:6823/1747049129","state":["exists","up"]},{"osd":7,"uuid":"8bff1d33-4cb9-45ba-8cf4-efc6051da49a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6817","nonce":961343269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6819","nonce":961343269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6823","nonce":961343269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":961343269},{"type":"v1","addr":"172.21.15.111:6821","nonce":961343269}]},"public_addr":"172.21.15.111:6817/961343269","cluster_addr":"172.21.15.111:6819/961343269","heartbeat_back_addr":"172.21.15.111:6823/961343269","heartbeat_front_addr":"172.21.15.111:6821/961343269","state":["exists","up"]},{"osd":8,"uuid":"6d85f1f1-9fed-495f-92b8-cd9f0fad42ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6817","nonce":434241518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6819","nonce":434241518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6823","nonce":434241518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":434241518},{"type":"v1","addr":"172.21.15.203:6821","nonce":434241518}]},"public_addr":"172.21.15.203:6817/434241518","cluster_addr":"172.21.15.203:6819/434241518","heartbeat_back_addr":"172.21.15.203:6823/434241518","heartbeat_front_addr":"172.21.15.203:6821/434241518","state":["exists","up"]},{"osd":9,"uuid":"75dc496d-7396-4c25-94c0-4976235724a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6827","nonce":654111402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6829","nonce":654111402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6833","nonce":654111402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":654111402},{"type":"v1","addr":"172.21.15.38:6831","nonce":654111402}]},"public_addr":"172.21.15.38:6827/654111402","cluster_addr":"172.21.15.38:6829/654111402","heartbeat_back_addr":"172.21.15.38:6833/654111402","heartbeat_front_addr":"172.21.15.38:6831/654111402","state":["exists","up"]},{"osd":10,"uuid":"9465f481-94aa-472a-993a-61d6a1946d0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030828796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030828796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030828796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030828796},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030828796}]},"public_addr":"172.21.15.111:6825/4030828796","cluster_addr":"172.21.15.111:6827/4030828796","heartbeat_back_addr":"172.21.15.111:6831/4030828796","heartbeat_front_addr":"172.21.15.111:6829/4030828796","state":["exists","up"]},{"osd":11,"uuid":"66e605ac-1577-4fcf-861c-cc6278c07699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6825","nonce":3896776627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6827","nonce":3896776627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6831","nonce":3896776627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":3896776627},{"type":"v1","addr":"172.21.15.203:6829","nonce":3896776627}]},"public_addr":"172.21.15.203:6825/3896776627","cluster_addr":"172.21.15.203:6827/3896776627","heartbeat_back_addr":"172.21.15.203:6831/3896776627","heartbeat_front_addr":"172.21.15.203:6829/3896776627","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:02.979324+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:27.191294+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:54:50.801943+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:17.753555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:55:42.815513+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:08.060786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:56:34.187776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:00.464539+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:26.408948+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:57:53.017406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:19.304688+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:58:45.678666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/1468209311":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2340980806":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/4082720653":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/914846331":"2024-04-23T00:02:04.818169+0000","172.21.15.38:0/2416463439":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6801/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1866190732":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2899039909":"2024-04-23T00:02:04.818169+0000","172.21.15.38:6800/2824973701":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/1458203717":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6800/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:6801/71250327":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/247200712":"2024-04-22T23:52:25.561345+0000","172.21.15.38:0/4200150941":"2024-04-22T23:51:57.369607+0000","172.21.15.38:0/2769068060":"2024-04-22T23:52:25.561345+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 msgr2=0x7f40880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 0x7f40880a51c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4090005fd0 tx=0x7f409000b5f0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 msgr2=0x7f40a019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4098009fd0 tx=0x7f409800b260 comp rx=0 tx=0).stop 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 shutdown_connections 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f40880a2d00 0x7f40880a51c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40a019c950 0x7f40a019ce00 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40a0106ac0 0x7f40a01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 --2- 172.21.15.38:0/3539953012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a0100300 0x7f40a0198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 >> 172.21.15.38:0/3539953012 conn(0x7f40a0075920 msgr2=0x7f40a00fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:17.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 shutdown_connections 2024-04-22T00:13:17.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.146+0000 7f40a66f4640 1 -- 172.21.15.38:0/3539953012 wait complete. 2024-04-22T00:13:17.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 -- 172.21.15.38:0/1526367627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 msgr2=0x7f7ef810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 --2- 172.21.15.38:0/1526367627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef810d510 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009ca0 tx=0x7f7ee802c0f0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 -- 172.21.15.38:0/1526367627 shutdown_connections 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 --2- 172.21.15.38:0/1526367627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef810d510 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 --2- 172.21.15.38:0/1526367627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef8101fd0 0x7f7ef8106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 --2- 172.21.15.38:0/1526367627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 0x7f7ef8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.263+0000 7f7effc6f640 1 -- 172.21.15.38:0/1526367627 >> 172.21.15.38:0/1526367627 conn(0x7f7ef8075940 msgr2=0x7f7ef8075d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- 172.21.15.38:0/1526367627 shutdown_connections 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- 172.21.15.38:0/1526367627 wait complete. 2024-04-22T00:13:17.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 Processor -- start 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- start start 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 0x7f7ef8198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef81998b0 0x7f7ef81a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81109f0 con 0x7f7ef8106ae0 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8110870 con 0x7f7ef8101620 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7effc6f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8110b70 con 0x7f7ef81998b0 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.264+0000 7f7efd20a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48126/0 (socket says 172.21.15.38:48126) 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 -- 172.21.15.38:0/2787790359 learned_addr learned my addr 172.21.15.38:0/2787790359 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efda0b640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 0x7f7ef8198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efe20c640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef81998b0 0x7f7ef81a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 -- 172.21.15.38:0/2787790359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef81998b0 msgr2=0x7f7ef81a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef81998b0 0x7f7ef81a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 -- 172.21.15.38:0/2787790359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 msgr2=0x7f7ef8198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 0x7f7ef8198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 -- 172.21.15.38:0/2787790359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef81a4630 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7efd20a640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00d770 tx=0x7f7eec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec017070 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eec009c30 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef81a4920 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec00f620 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.265+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a4e70 con 0x7f7ef8106ae0 2024-04-22T00:13:17.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.266+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef80689b0 con 0x7f7ef8106ae0 2024-04-22T00:13:17.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.267+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f7eec01a070 con 0x7f7ef8106ae0 2024-04-22T00:13:17.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.268+0000 7f7ef6ffd640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 0x7f7ee00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.268+0000 7f7efda0b640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 0x7f7ee00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.269+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f7eec0cf2d0 con 0x7f7ef8106ae0 2024-04-22T00:13:17.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.269+0000 7f7efda0b640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 0x7f7ee00a51c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000bbe0 tx=0x7f7ef000b390 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:17.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.272+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7eec009de0 con 0x7f7ef8106ae0 2024-04-22T00:13:17.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2256337937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1542065893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/3539953012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.497+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7ef8100840 con 0x7f7ef8106ae0 2024-04-22T00:13:17.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2256337937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1542065893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.496 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/3539953012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.500+0000 7f7ef6ffd640 1 -- 172.21.15.38:0/2787790359 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 161+0+0 (secure 0 0 0) 0x7f7eec09bb00 con 0x7f7ef8106ae0 2024-04-22T00:13:17.498 INFO:teuthology.orchestra.run.smithi038.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T00:13:17.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 msgr2=0x7f7ee00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 0x7f7ee00a51c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000bbe0 tx=0x7f7ef000b390 comp rx=0 tx=0).stop 2024-04-22T00:13:17.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 msgr2=0x7f7ef8199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7eec00d770 tx=0x7f7eec00dc30 comp rx=0 tx=0).stop 2024-04-22T00:13:17.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 shutdown_connections 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f7ee00a2d00 0x7f7ee00a51c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef81998b0 0x7f7ef81a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ef8106ae0 0x7f7ef8199370 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 --2- 172.21.15.38:0/2787790359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ef8101620 0x7f7ef8198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 >> 172.21.15.38:0/2787790359 conn(0x7f7ef8075940 msgr2=0x7f7ef80ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 shutdown_connections 2024-04-22T00:13:17.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.504+0000 7f7effc6f640 1 -- 172.21.15.38:0/2787790359 wait complete. 2024-04-22T00:13:17.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T00:13:17.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:17 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2256337937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:17 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1542065893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.534 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:17 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/3539953012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T00:13:17.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 -- 172.21.15.38:0/2052140912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 msgr2=0x7f2eec107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 --2- 172.21.15.38:0/2052140912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec107ae0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009930 tx=0x7f2edc02bc30 comp rx=0 tx=0).stop 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 -- 172.21.15.38:0/2052140912 shutdown_connections 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 --2- 172.21.15.38:0/2052140912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 0x7f2eec10e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 --2- 172.21.15.38:0/2052140912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 0x7f2eec101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 --2- 172.21.15.38:0/2052140912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec107ae0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 -- 172.21.15.38:0/2052140912 >> 172.21.15.38:0/2052140912 conn(0x7f2eec078b90 msgr2=0x7f2eec078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 -- 172.21.15.38:0/2052140912 shutdown_connections 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.624+0000 7f2ef1985640 1 -- 172.21.15.38:0/2052140912 wait complete. 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 Processor -- start 2024-04-22T00:13:17.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 -- start start 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 0x7f2eec198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 0x7f2eec198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec111ad0 con 0x7f2eec107700 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec111950 con 0x7f2eec1020b0 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.625+0000 7f2ef1985640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec111c50 con 0x7f2eec1016f0 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48144/0 (socket says 172.21.15.38:48144) 2024-04-22T00:13:17.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 -- 172.21.15.38:0/1481142640 learned_addr learned my addr 172.21.15.38:0/1481142640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeaffd640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 0x7f2eec198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eea7fc640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 0x7f2eec198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 -- 172.21.15.38:0/1481142640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 msgr2=0x7f2eec198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 0x7f2eec198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 -- 172.21.15.38:0/1481142640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 msgr2=0x7f2eec198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 0x7f2eec198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 -- 172.21.15.38:0/1481142640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec19d040 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2eeb7fe640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400eb10 tx=0x7f2ee400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4004b20 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee4004cc0 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4010760 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec19d330 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.626+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec1a4bc0 con 0x7f2eec107700 2024-04-22T00:13:17.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.627+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec0689b0 con 0x7f2eec107700 2024-04-22T00:13:17.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.629+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f2ee4010900 con 0x7f2eec107700 2024-04-22T00:13:17.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.630+0000 7f2ef0983640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 0x7f2ed40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:17.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.630+0000 7f2eeaffd640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 0x7f2ed40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:17.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.630+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9868+0+0 (secure 0 0 0) 0x7f2ee40d00c0 con 0x7f2eec107700 2024-04-22T00:13:17.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.630+0000 7f2eeaffd640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 0x7f2ed40a5240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2edc006010 tx=0x7f2edc000bc0 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:17.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.633+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2ee40536f0 con 0x7f2eec107700 2024-04-22T00:13:17.769 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a[27968]: 2024-04-22T00:13:17.498+0000 7f0cdb1bc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T00:13:17.863 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:17.865+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2eec02d170 con 0x7f2eec107700 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mds[60987]: -- 172.21.15.111:0/502416092 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee99f11a40 con 0x55ee99ee4800 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: starting mds.e at 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.082 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mds[61608]: -- 172.21.15.111:0/84785309 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a758c09a40 con 0x55a758bdcc00 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: starting mds.b at 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mds[61299]: -- 172.21.15.111:0/4218378699 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555b08d9e4e0 con 0x555b08d74800 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mds[61299]: -- 172.21.15.111:0/4218378699 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b08da1a40 con 0x555b08d74800 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: starting mds.h at 2024-04-22T00:13:18.083 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.084 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.084 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-mds[60666]: -- 172.21.15.111:0/1931469856 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3ba945a40 con 0x55f3ba919000 2024-04-22T00:13:18.084 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:13:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: starting mds.k at 2024-04-22T00:13:18.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.208 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mds[69884]: -- 172.21.15.38:0/2046013940 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e300aeda40 con 0x55e300ac0c00 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: starting mds.a at 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mds[69524]: -- 172.21.15.38:0/538913393 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564fb41484e0 con 0x564fb411f000 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-mds[69524]: -- 172.21.15.38:0/538913393 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fb414ba40 con 0x564fb411f000 2024-04-22T00:13:18.209 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:13:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: starting mds.g at 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: pgmap v348: 129 pgs: 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 118 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 563 B/s wr, 12 op/s 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T00:13:18.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:18 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T00:13:18.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.504+0000 7f2ef0983640 1 -- 172.21.15.38:0/1481142640 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) ==== 104+0+0 (secure 0 0 0) 0x7f2ee409c7a0 con 0x7f2eec107700 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 msgr2=0x7f2ed40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 0x7f2ed40a5240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2edc006010 tx=0x7f2edc000bc0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 msgr2=0x7f2eec19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400eb10 tx=0x7f2ee400c2d0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 shutdown_connections 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f2ed40a2d80 0x7f2ed40a5240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2eec107700 0x7f2eec19c910 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2eec1020b0 0x7f2eec198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 --2- 172.21.15.38:0/1481142640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2eec1016f0 0x7f2eec198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 >> 172.21.15.38:0/1481142640 conn(0x7f2eec078b90 msgr2=0x7f2eec0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:18.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 shutdown_connections 2024-04-22T00:13:18.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.507+0000 7f2ef1985640 1 -- 172.21.15.38:0/1481142640 wait complete. 2024-04-22T00:13:18.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T00:13:18.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: pgmap v348: 129 pgs: 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 118 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 563 B/s wr, 12 op/s 2024-04-22T00:13:18.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T00:13:18.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T00:13:18.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.536 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.537 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mds[62649]: -- 172.21.15.203:0/2305827406 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a283101a40 con 0x55a2830d5000 2024-04-22T00:13:18.537 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: starting mds.i at 2024-04-22T00:13:18.537 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.537 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.538 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mds[62031]: -- 172.21.15.203:0/1602537631 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8a7579a40 con 0x55c8a754c800 2024-04-22T00:13:18.538 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: starting mds.f at 2024-04-22T00:13:18.538 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.538 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.538 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mds[62339]: -- 172.21.15.203:0/3315435386 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cdcd211a40 con 0x55cdcd1e4c00 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: starting mds.c at 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: 2024-04-22T00:13:18.242+0000 7f9788153700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: ignoring --setuser ceph since I am not root 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: ignoring --setgroup ceph since I am not root 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-mds[61720]: -- 172.21.15.203:0/3236597503 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb13b81a40 con 0x55cb13b54800 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: starting mds.l at 2024-04-22T00:13:18.539 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:13:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: 2024-04-22T00:13:18.242+0000 7feec79a0700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-22T00:13:18.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: pgmap v348: 129 pgs: 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 118 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 563 B/s wr, 12 op/s 2024-04-22T00:13:18.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T00:13:18.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T00:13:18.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T00:13:18.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/2787790359' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T00:13:18.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T00:13:18.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T00:13:18.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:18 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T00:13:18.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 -- 172.21.15.38:0/2679194292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 msgr2=0x7f23ec10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:18.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 --2- 172.21.15.38:0/2679194292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec10e5e0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009cd0 tx=0x7f23e402c0f0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 -- 172.21.15.38:0/2679194292 shutdown_connections 2024-04-22T00:13:18.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 --2- 172.21.15.38:0/2679194292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec10e5e0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 --2- 172.21.15.38:0/2679194292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec101710 0x7f23ec101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 --2- 172.21.15.38:0/2679194292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec107720 0x7f23ec107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 -- 172.21.15.38:0/2679194292 >> 172.21.15.38:0/2679194292 conn(0x7f23ec078b90 msgr2=0x7f23ec078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 -- 172.21.15.38:0/2679194292 shutdown_connections 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.638+0000 7f23f265f640 1 -- 172.21.15.38:0/2679194292 wait complete. 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 Processor -- start 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 -- start start 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 0x7f23ec198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 0x7f23ec19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec111b30 con 0x7f23ec1020d0 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1119b0 con 0x7f23ec107720 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f265f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec111cb0 con 0x7f23ec101710 2024-04-22T00:13:18.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48200/0 (socket says 172.21.15.38:48200) 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 -- 172.21.15.38:0/4167227848 learned_addr learned my addr 172.21.15.38:0/4167227848 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23ebfff640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 0x7f23ec198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f0bfc640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 0x7f23ec19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 -- 172.21.15.38:0/4167227848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 msgr2=0x7f23ec198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 0x7f23ec198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 -- 172.21.15.38:0/4167227848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 msgr2=0x7f23ec19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 0x7f23ec19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23eb7fe640 1 -- 172.21.15.38:0/4167227848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec19d000 con 0x7f23ec1020d0 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23ebfff640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 0x7f23ec198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.639+0000 7f23f0bfc640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 0x7f23ec19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:13:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23eb7fe640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00d950 tx=0x7f23dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:18.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc017030 con 0x7f23ec1020d0 2024-04-22T00:13:18.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc004020 con 0x7f23ec1020d0 2024-04-22T00:13:18.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec19d2f0 con 0x7f23ec1020d0 2024-04-22T00:13:18.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0047a0 con 0x7f23ec1020d0 2024-04-22T00:13:18.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.640+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1a4b80 con 0x7f23ec1020d0 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.641+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ec0689b0 con 0x7f23ec1020d0 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.642+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f23dc0041d0 con 0x7f23ec1020d0 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.643+0000 7f23e97fa640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 0x7f23d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.643+0000 7f23ebfff640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 0x7f23d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.643+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9868+0+0 (secure 0 0 0) 0x7f23dc0cf410 con 0x7f23ec1020d0 2024-04-22T00:13:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.643+0000 7f23ebfff640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 0x7f23d80a51c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009f40 tx=0x7f23d4005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:18.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.647+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f23dc09ba90 con 0x7f23ec1020d0 2024-04-22T00:13:18.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:18.833+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f23ec02d170 con 0x7f23ec1020d0 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: Cluster is now healthy 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6832/3261586575,v1:172.21.15.111:6835/3261586575] up:boot 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6833/2557048546,v1:172.21.15.111:6837/2557048546] up:boot 2024-04-22T00:13:19.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6834/1999943419,v1:172.21.15.111:6838/1999943419] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.111:6836/630507318,v1:172.21.15.111:6839/630507318] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6839/1995425639,v1:172.21.15.38:6841/1995425639] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.38:6838/2324465122,v1:172.21.15.38:6840/2324465122] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6833/3727763593,v1:172.21.15.203:6832/3727763593] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6834/4221221003,v1:172.21.15.203:6835/4221221003] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6837/687857943,v1:172.21.15.203:6838/687857943] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: mds.? [v2:172.21.15.203:6836/3656494349,v1:172.21.15.203:6839/3656494349] up:boot 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: fsmap 12 up:standby 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:13:19.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:13:19.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:13:19.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:13:19.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:13:19.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:13:19.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:19 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:19.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T00:13:19.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T00:13:19.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: Cluster is now healthy 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6832/3261586575,v1:172.21.15.111:6835/3261586575] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6833/2557048546,v1:172.21.15.111:6837/2557048546] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6834/1999943419,v1:172.21.15.111:6838/1999943419] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.111:6836/630507318,v1:172.21.15.111:6839/630507318] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6839/1995425639,v1:172.21.15.38:6841/1995425639] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.38:6838/2324465122,v1:172.21.15.38:6840/2324465122] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6833/3727763593,v1:172.21.15.203:6832/3727763593] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6834/4221221003,v1:172.21.15.203:6835/4221221003] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6837/687857943,v1:172.21.15.203:6838/687857943] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: mds.? [v2:172.21.15.203:6836/3656494349,v1:172.21.15.203:6839/3656494349] up:boot 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: fsmap 12 up:standby 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:13:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:13:19.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:19 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:19.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.511+0000 7f23e97fa640 1 -- 172.21.15.38:0/4167227848 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) ==== 181+0+0 (secure 0 0 0) 0x7f23dc09b390 con 0x7f23ec1020d0 2024-04-22T00:13:19.512 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' removed 2024-04-22T00:13:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 msgr2=0x7f23d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 0x7f23d80a51c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009f40 tx=0x7f23d4005f90 comp rx=0 tx=0).stop 2024-04-22T00:13:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 msgr2=0x7f23ec198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:19.513 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00d950 tx=0x7f23dc00de10 comp rx=0 tx=0).stop 2024-04-22T00:13:19.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 shutdown_connections 2024-04-22T00:13:19.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f23d80a2d00 0x7f23d80a51c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23ec107720 0x7f23ec19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23ec1020d0 0x7f23ec198550 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 --2- 172.21.15.38:0/4167227848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23ec101710 0x7f23ec198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 >> 172.21.15.38:0/4167227848 conn(0x7f23ec078b90 msgr2=0x7f23ec0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:19.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.515+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 shutdown_connections 2024-04-22T00:13:19.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.516+0000 7f23f265f640 1 -- 172.21.15.38:0/4167227848 wait complete. 2024-04-22T00:13:19.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: Cluster is now healthy 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1481142640' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6832/3261586575,v1:172.21.15.111:6835/3261586575] up:boot 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6833/2557048546,v1:172.21.15.111:6837/2557048546] up:boot 2024-04-22T00:13:19.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6834/1999943419,v1:172.21.15.111:6838/1999943419] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.111:6836/630507318,v1:172.21.15.111:6839/630507318] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6839/1995425639,v1:172.21.15.38:6841/1995425639] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.38:6838/2324465122,v1:172.21.15.38:6840/2324465122] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6833/3727763593,v1:172.21.15.203:6832/3727763593] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6834/4221221003,v1:172.21.15.203:6835/4221221003] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6837/687857943,v1:172.21.15.203:6838/687857943] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: mds.? [v2:172.21.15.203:6836/3656494349,v1:172.21.15.203:6839/3656494349] up:boot 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: fsmap 12 up:standby 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T00:13:19.583 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T00:13:19.584 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T00:13:19.584 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T00:13:19.584 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T00:13:19.584 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='mgr.15528 172.21.15.38:0/2983842481' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T00:13:19.584 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:19 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.655+0000 7f97ae937640 1 -- 172.21.15.38:0/1986982209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8112720 msgr2=0x7f97a8114b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.655+0000 7f97ae937640 1 --2- 172.21.15.38:0/1986982209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8112720 0x7f97a8114b10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009ca0 tx=0x7f97a002c180 comp rx=0 tx=0).stop 2024-04-22T00:13:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 -- 172.21.15.38:0/1986982209 shutdown_connections 2024-04-22T00:13:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 --2- 172.21.15.38:0/1986982209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8112720 0x7f97a8114b10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 --2- 172.21.15.38:0/1986982209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8074400 0x7f97a81121e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 --2- 172.21.15.38:0/1986982209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8073ae0 0x7f97a8073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 -- 172.21.15.38:0/1986982209 >> 172.21.15.38:0/1986982209 conn(0x7f97a80fd180 msgr2=0x7f97a80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 -- 172.21.15.38:0/1986982209 shutdown_connections 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 -- 172.21.15.38:0/1986982209 wait complete. 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 Processor -- start 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.656+0000 7f97ae937640 1 -- start start 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074400 0x7f97a819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:19.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8112720 0x7f97a81a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81180c0 con 0x7f97a8073ae0 2024-04-22T00:13:19.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8117f40 con 0x7f97a8112720 2024-04-22T00:13:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97ae937640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8118240 con 0x7f97a8074400 2024-04-22T00:13:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:19.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48216/0 (socket says 172.21.15.38:48216) 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 -- 172.21.15.38:0/1537669607 learned_addr learned my addr 172.21.15.38:0/1537669607 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a77fe640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074400 0x7f97a819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97aced4640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8112720 0x7f97a81a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 -- 172.21.15.38:0/1537669607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074400 msgr2=0x7f97a819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074400 0x7f97a819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 -- 172.21.15.38:0/1537669607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8112720 msgr2=0x7f97a81a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8112720 0x7f97a81a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 -- 172.21.15.38:0/1537669607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a81a35f0 con 0x7f97a8073ae0 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a7fff640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f979800d950 tx=0x7f979800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.657+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798017030 con 0x7f97a8073ae0 2024-04-22T00:13:19.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.658+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a81a38e0 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.658+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9798004020 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.658+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97980047a0 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.658+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a81ab170 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.659+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a80689b0 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.660+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f97980041d0 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.660+0000 7f97a57fa640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.661+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9372+0+0 (secure 0 0 0) 0x7f97980cf180 con 0x7f97a8073ae0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.661+0000 7f97a77fe640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:19.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.661+0000 7f97a77fe640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 0x7f97900a51c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f979c005fd0 tx=0x7f979c005850 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:19.677 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.665+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f979809ba50 con 0x7f97a8073ae0 2024-04-22T00:13:19.849 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:19.851+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97a819f950 con 0x7f97a8073ae0 2024-04-22T00:13:20.510 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.512+0000 7f97a57fa640 1 -- 172.21.15.38:0/1537669607 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) ==== 169+0+0 (secure 0 0 0) 0x7f979809b350 con 0x7f97a8073ae0 2024-04-22T00:13:20.510 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' removed 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 msgr2=0x7f97900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 0x7f97900a51c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f979c005fd0 tx=0x7f979c005850 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 msgr2=0x7f97a819e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f979800d950 tx=0x7f979800de10 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 shutdown_connections 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a8112720 0x7f97a81a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97a8074400 0x7f97a819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 --2- 172.21.15.38:0/1537669607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97a8073ae0 0x7f97a819e600 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 >> 172.21.15.38:0/1537669607 conn(0x7f97a80fd180 msgr2=0x7f97a80fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:20.515 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 shutdown_connections 2024-04-22T00:13:20.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.516+0000 7f97ae937640 1 -- 172.21.15.38:0/1537669607 wait complete. 2024-04-22T00:13:20.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-22T00:13:20.606 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:20 smithi111 ceph-mon[31543]: pgmap v350: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T00:13:20.606 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:20 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:20.606 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:20 smithi111 ceph-mon[31543]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T00:13:20.606 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:20 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:20.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- 172.21.15.38:0/1708544056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 msgr2=0x7f75c010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:20.650 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/1708544056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c010e1a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009ca0 tx=0x7f75b802c230 comp rx=0 tx=0).stop 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- 172.21.15.38:0/1708544056 shutdown_connections 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/1708544056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c010e1a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/1708544056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/1708544056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- 172.21.15.38:0/1708544056 >> 172.21.15.38:0/1708544056 conn(0x7f75c0077740 msgr2=0x7f75c0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- 172.21.15.38:0/1708544056 shutdown_connections 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- 172.21.15.38:0/1708544056 wait complete. 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 Processor -- start 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.649+0000 7f75c6d0c640 1 -- start start 2024-04-22T00:13:20.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c01984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0111b00 con 0x7f75c010bd00 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0111980 con 0x7f75c01029c0 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0111c80 con 0x7f75c01032e0 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c4aa8640 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:48236/0 (socket says 172.21.15.38:48236) 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 -- 172.21.15.38:0/826044532 learned_addr learned my addr 172.21.15.38:0/826044532 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-22T00:13:20.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75bffff640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c01984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 -- 172.21.15.38:0/826044532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 msgr2=0x7f75c01984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c01984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 -- 172.21.15.38:0/826044532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 msgr2=0x7f75c0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 -- 172.21.15.38:0/826044532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c019cf90 con 0x7f75c010bd00 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75bffff640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c01984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c4aa8640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c52a9640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f75b802c840 tx=0x7f75b8035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8039040 con 0x7f75c010bd00 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b8044070 con 0x7f75c010bd00 2024-04-22T00:13:20.653 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c019d220 con 0x7f75c010bd00 2024-04-22T00:13:20.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.650+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8004330 con 0x7f75c010bd00 2024-04-22T00:13:20.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.651+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c01a4b10 con 0x7f75c010bd00 2024-04-22T00:13:20.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.652+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c0068560 con 0x7f75c010bd00 2024-04-22T00:13:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.653+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 137964+0+0 (secure 0 0 0) 0x7f75b80044d0 con 0x7f75c010bd00 2024-04-22T00:13:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.654+0000 7f75bdffb640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 0x7f75a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T00:13:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.654+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8842+0+0 (secure 0 0 0) 0x7f75b80f1970 con 0x7f75c010bd00 2024-04-22T00:13:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.654+0000 7f75c4aa8640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 0x7f75a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T00:13:20.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.654+0000 7f75c4aa8640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 0x7f75a80a51c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75b00095f0 tx=0x7f75b0005f90 comp rx=0 tx=0).ready entity=mgr.15528 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T00:13:20.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.658+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f75b80be560 con 0x7f75c010bd00 2024-04-22T00:13:20.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:20 smithi203 ceph-mon[31431]: pgmap v350: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T00:13:20.785 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:20 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:20.785 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:20 smithi203 ceph-mon[31431]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T00:13:20.785 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:20 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:20.844 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:20.846+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f75c0106eb0 con 0x7f75c010bd00 2024-04-22T00:13:20.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:20 smithi038 ceph-mon[27972]: pgmap v350: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-22T00:13:20.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/4167227848' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:20.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:20 smithi038 ceph-mon[27972]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T00:13:20.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:20 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:21.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.526+0000 7f75bdffb640 1 -- 172.21.15.38:0/826044532 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v105) ==== 178+0+0 (secure 0 0 0) 0x7f75b80bde60 con 0x7f75c010bd00 2024-04-22T00:13:21.524 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data_ec' removed 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 msgr2=0x7f75a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 0x7f75a80a51c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75b00095f0 tx=0x7f75b0005f90 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 msgr2=0x7f75c019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f75b802c840 tx=0x7f75b8035d90 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 shutdown_connections 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:6800/2750091046,v1:172.21.15.38:6801/2750091046] conn(0x7f75a80a2d00 0x7f75a80a51c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c010bd00 0x7f75c019c860 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75c01032e0 0x7f75c01984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 --2- 172.21.15.38:0/826044532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c01029c0 0x7f75c0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 >> 172.21.15.38:0/826044532 conn(0x7f75c0077740 msgr2=0x7f75c00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 shutdown_connections 2024-04-22T00:13:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-22T00:13:21.530+0000 7f75c6d0c640 1 -- 172.21.15.38:0/826044532 wait complete. 2024-04-22T00:13:21.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:13:21.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:13:21.741 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:21 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:21.742 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:21 smithi111 ceph-mon[31543]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T00:13:21.742 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:21 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:21.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:13:21.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:21 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:21.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:21 smithi203 ceph-mon[31431]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T00:13:21.778 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:21 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:21.870 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T00:13:21.870 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T00:13:21.946 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T00:13:21.960 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T00:13:21.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:21 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/1537669607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:21.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:21 smithi038 ceph-mon[27972]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T00:13:21.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:21 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T00:13:21.980 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T00:13:21.980 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T00:13:21.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a 2024-04-22T00:13:22.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:22 smithi038 systemd[1]: Stopping Ceph mon.a for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:22 smithi203 ceph-mon[31431]: pgmap v353: 33 pgs: 33 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:13:22.784 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:22 smithi203 ceph-mon[31431]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:22.785 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:22 smithi203 ceph-mon[31431]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T00:13:22.831 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:22 smithi111 ceph-mon[31543]: pgmap v353: 33 pgs: 33 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:13:22.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:22 smithi111 ceph-mon[31543]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:22.832 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:22 smithi111 ceph-mon[31543]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T00:13:22.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:22 smithi038 ceph-mon[27972]: pgmap v353: 33 pgs: 33 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T00:13:22.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:22 smithi038 ceph-mon[27972]: from='client.? 172.21.15.38:0/826044532' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T00:13:22.963 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:22 smithi038 ceph-mon[27972]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T00:13:24.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a[27968]: 2024-04-22T00:13:24.033+0000 7f0ce09c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:13:24.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a[27968]: 2024-04-22T00:13:24.033+0000 7f0ce09c7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T00:13:24.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:24 smithi038 podman[104090]: 2024-04-22 00:13:24.13068401 +0000 UTC m=+1.739933583 container died 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T00:13:29.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:29 smithi038 podman[104090]: 2024-04-22 00:13:29.057586462 +0000 UTC m=+6.666836034 container cleanup 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T00:13:29.463 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:29 smithi038 bash[104090]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a 2024-04-22T00:13:32.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Apr 22 00:13:32 smithi038 podman[104134]: 2024-04-22 00:13:32.429885978 +0000 UTC m=+8.292486022 container remove 0310a999fd7259ec0ea37a66c770c1f64fac67c64535684b0465570a868c677d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T00:13:32.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T00:13:32.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:32.874 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T00:13:32.874 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T00:13:32.875 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.b 2024-04-22T00:13:33.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:32 smithi111 systemd[1]: Stopping Ceph mon.b for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:35.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:34 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b[31538]: 2024-04-22T00:13:34.931+0000 7f9ac797b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:13:35.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:34 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b[31538]: 2024-04-22T00:13:34.931+0000 7f9ac797b700 -1 mon.b@2(electing) e3 *** Got Signal Terminated *** 2024-04-22T00:13:35.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:35 smithi111 podman[67418]: 2024-04-22 00:13:35.143912545 +0000 UTC m=+1.714844179 container died 7fe567889c331b960563b4f3599a39c2645b7327a074e17bb0a846410bd4e469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True) 2024-04-22T00:13:41.082 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:40 smithi111 podman[67418]: 2024-04-22 00:13:40.808646913 +0000 UTC m=+7.379578529 container cleanup 7fe567889c331b960563b4f3599a39c2645b7327a074e17bb0a846410bd4e469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T00:13:41.082 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:40 smithi111 bash[67418]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b 2024-04-22T00:13:43.640 INFO:journalctl@ceph.mon.b.smithi111.stdout:Apr 22 00:13:43 smithi111 podman[67428]: 2024-04-22 00:13:43.385258428 +0000 UTC m=+8.242984671 container remove 7fe567889c331b960563b4f3599a39c2645b7327a074e17bb0a846410bd4e469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:13:43.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T00:13:43.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:43.814 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T00:13:43.814 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T00:13:43.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.c 2024-04-22T00:13:44.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:43 smithi203 systemd[1]: Stopping Ceph mon.c for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:44.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:44 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c[31425]: 2024-04-22T00:13:44.196+0000 7f252b20b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:13:44.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:44 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c[31425]: 2024-04-22T00:13:44.196+0000 7f252b20b700 -1 mon.c@1(leader) e3 *** Got Signal Terminated *** 2024-04-22T00:13:44.535 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:44 smithi203 podman[68140]: 2024-04-22 00:13:44.430927894 +0000 UTC m=+0.340782907 container died c2c78d773b2782f6a53d1ea505da9573431b75dbd995bfe809aefde22782b946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-22T00:13:44.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:44 smithi203 podman[68140]: 2024-04-22 00:13:44.822360606 +0000 UTC m=+0.732215618 container cleanup c2c78d773b2782f6a53d1ea505da9573431b75dbd995bfe809aefde22782b946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-22T00:13:44.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:44 smithi203 bash[68140]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c 2024-04-22T00:13:45.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 22 00:13:45 smithi203 podman[68154]: 2024-04-22 00:13:45.141191754 +0000 UTC m=+0.711338077 container remove c2c78d773b2782f6a53d1ea505da9573431b75dbd995bfe809aefde22782b946 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-22T00:13:45.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T00:13:45.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:45.370 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T00:13:45.370 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T00:13:45.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x 2024-04-22T00:13:45.713 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:45 smithi038 systemd[1]: Stopping Ceph mgr.x for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:48.213 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:47 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x[28197]: 2024-04-22T00:13:47.907+0000 7feb033f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:13:48.213 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:47 smithi038 podman[104299]: 2024-04-22 00:13:47.950323022 +0000 UTC m=+2.005252817 container died aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T00:13:52.963 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:52 smithi038 podman[104299]: 2024-04-22 00:13:52.492364341 +0000 UTC m=+6.547294126 container cleanup aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T00:13:52.963 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:52 smithi038 bash[104299]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x 2024-04-22T00:13:55.343 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Apr 22 00:13:55 smithi038 podman[104313]: 2024-04-22 00:13:55.065079894 +0000 UTC m=+7.021655216 container remove aed10d5afb91cee4613b93b2f104d0ba7cdfbaf832c1cbbce484a322c36d2a2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T00:13:55.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T00:13:55.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:55.393 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T00:13:55.393 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T00:13:55.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.y 2024-04-22T00:13:55.739 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:13:55 smithi111 systemd[1]: Stopping Ceph mgr.y for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:56.044 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:13:55 smithi111 podman[67549]: 2024-04-22 00:13:55.74131856 +0000 UTC m=+0.145008371 container died 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-22T00:13:56.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:13:56 smithi111 podman[67549]: 2024-04-22 00:13:56.146292346 +0000 UTC m=+0.549982164 container cleanup 302af1c9f77c10ceba13ff626e4759a34f14547c5617243a670458e71472bdd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T00:13:56.332 INFO:journalctl@ceph.mgr.y.smithi111.stdout:Apr 22 00:13:56 smithi111 bash[67549]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-y 2024-04-22T00:13:56.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T00:13:56.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:56.562 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T00:13:56.562 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T00:13:56.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.z 2024-04-22T00:13:56.898 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:13:56 smithi203 systemd[1]: Stopping Ceph mgr.z for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:57.211 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:13:56 smithi203 podman[68277]: 2024-04-22 00:13:56.899667624 +0000 UTC m=+0.141512481 container died a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:13:57.534 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:13:57 smithi203 podman[68277]: 2024-04-22 00:13:57.31293878 +0000 UTC m=+0.554783624 container cleanup a76505f5a3477bad380c725c55f59bfff0f2b7a79e64329a572a8eaf58c2ae53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-22T00:13:57.535 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 22 00:13:57 smithi203 bash[68277]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mgr-z 2024-04-22T00:13:57.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T00:13:57.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:13:57.779 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T00:13:57.779 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T00:13:57.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.0 2024-04-22T00:13:58.163 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:13:57 smithi038 systemd[1]: Stopping Ceph osd.0 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:13:58.963 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:13:58 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0[42821]: 2024-04-22T00:13:58.694+0000 7f4008567700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:13:58.964 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:13:58 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0[42821]: 2024-04-22T00:13:58.694+0000 7f4008567700 -1 osd.0 105 *** Got signal Terminated *** 2024-04-22T00:13:58.964 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:13:58 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0[42821]: 2024-04-22T00:13:58.694+0000 7f4008567700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:04.826 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:04 smithi038 podman[104422]: 2024-04-22 00:14:04.533247051 +0000 UTC m=+6.549769677 container died 5e90c03034cbd448bb5fe66acf2362feaf1377146592d40ef776b234b0fc8a13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T00:14:05.160 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:04 smithi038 podman[104422]: 2024-04-22 00:14:04.92767353 +0000 UTC m=+6.944196155 container cleanup 5e90c03034cbd448bb5fe66acf2362feaf1377146592d40ef776b234b0fc8a13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:05.160 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:04 smithi038 bash[104422]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0 2024-04-22T00:14:05.463 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104441]: 2024-04-22 00:14:05.162440529 +0000 UTC m=+0.623460613 container remove 5e90c03034cbd448bb5fe66acf2362feaf1377146592d40ef776b234b0fc8a13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-22T00:14:05.830 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104528]: 2024-04-22 00:14:05.390733183 +0000 UTC m=+0.018566136 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:05.830 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104528]: 2024-04-22 00:14:05.530334005 +0000 UTC m=+0.158166999 container create a44d2c006d92d1431bd2031dc133fc3af69915f03d1cdbb8fe5bf44aea29a8ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:05.831 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104528]: 2024-04-22 00:14:05.833156186 +0000 UTC m=+0.460989146 container init a44d2c006d92d1431bd2031dc133fc3af69915f03d1cdbb8fe5bf44aea29a8ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:06.213 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104528]: 2024-04-22 00:14:05.836487359 +0000 UTC m=+0.464320312 container start a44d2c006d92d1431bd2031dc133fc3af69915f03d1cdbb8fe5bf44aea29a8ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:06.213 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:05 smithi038 podman[104528]: 2024-04-22 00:14:05.933481896 +0000 UTC m=+0.561314853 container attach a44d2c006d92d1431bd2031dc133fc3af69915f03d1cdbb8fe5bf44aea29a8ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2) 2024-04-22T00:14:06.709 INFO:journalctl@ceph.osd.0.smithi038.stdout:Apr 22 00:14:06 smithi038 podman[104528]: 2024-04-22 00:14:06.430264024 +0000 UTC m=+1.058096977 container died a44d2c006d92d1431bd2031dc133fc3af69915f03d1cdbb8fe5bf44aea29a8ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-22T00:14:07.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T00:14:07.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:07.049 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T00:14:07.049 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T00:14:07.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.1 2024-04-22T00:14:07.332 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:07 smithi111 systemd[1]: Stopping Ceph osd.1 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:07.832 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:07 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1[36145]: 2024-04-22T00:14:07.375+0000 7fdd08e1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:07.832 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:07 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1[36145]: 2024-04-22T00:14:07.375+0000 7fdd08e1e700 -1 osd.1 105 *** Got signal Terminated *** 2024-04-22T00:14:07.832 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:07 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1[36145]: 2024-04-22T00:14:07.375+0000 7fdd08e1e700 -1 osd.1 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:13.332 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:13 smithi111 podman[67675]: 2024-04-22 00:14:13.052912324 +0000 UTC m=+5.805441419 container died 4ce29f80d5d27f4ddb8333cee00f3e41abdf5bf4a32135a1dcc3021a32b4dada (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-820a2f8) 2024-04-22T00:14:13.707 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:13 smithi111 podman[67675]: 2024-04-22 00:14:13.458933098 +0000 UTC m=+6.211462188 container cleanup 4ce29f80d5d27f4ddb8333cee00f3e41abdf5bf4a32135a1dcc3021a32b4dada (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T00:14:13.708 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:13 smithi111 bash[67675]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1 2024-04-22T00:14:14.021 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:13 smithi111 podman[67692]: 2024-04-22 00:14:13.710458127 +0000 UTC m=+0.657174693 container remove 4ce29f80d5d27f4ddb8333cee00f3e41abdf5bf4a32135a1dcc3021a32b4dada (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T00:14:14.273 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:13.926387016 +0000 UTC m=+0.018040768 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:14.273 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:14.036566081 +0000 UTC m=+0.128219856 container create a3afe49d0581f2a907fb621e9fbcaae6b33e2009b8559076345f5d71f734d470 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8) 2024-04-22T00:14:14.273 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:14.272399709 +0000 UTC m=+0.364053447 container init a3afe49d0581f2a907fb621e9fbcaae6b33e2009b8559076345f5d71f734d470 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T00:14:14.273 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:14.275626618 +0000 UTC m=+0.367280360 container start a3afe49d0581f2a907fb621e9fbcaae6b33e2009b8559076345f5d71f734d470 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T00:14:14.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:14.381245486 +0000 UTC m=+0.472899303 container attach a3afe49d0581f2a907fb621e9fbcaae6b33e2009b8559076345f5d71f734d470 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T00:14:15.082 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 22 00:14:14 smithi111 podman[67780]: 2024-04-22 00:14:14.821607488 +0000 UTC m=+0.913261227 container died a3afe49d0581f2a907fb621e9fbcaae6b33e2009b8559076345f5d71f734d470 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:15.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T00:14:15.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:15.541 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T00:14:15.541 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T00:14:15.541 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.2 2024-04-22T00:14:15.864 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:15 smithi203 systemd[1]: Stopping Ceph osd.2 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:16.284 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2[36583]: 2024-04-22T00:14:15.866+0000 7f8686293700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:16.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2[36583]: 2024-04-22T00:14:15.866+0000 7f8686293700 -1 osd.2 105 *** Got signal Terminated *** 2024-04-22T00:14:16.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2[36583]: 2024-04-22T00:14:15.866+0000 7f8686293700 -1 osd.2 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:22.016 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:21 smithi203 podman[68400]: 2024-04-22 00:14:21.722167578 +0000 UTC m=+5.984285892 container died 39f2a7f5f7423bfe733cc4df09a496766033993edc96886449d398fdd2ac5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T00:14:22.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:22 smithi203 podman[68400]: 2024-04-22 00:14:22.109174078 +0000 UTC m=+6.371292390 container cleanup 39f2a7f5f7423bfe733cc4df09a496766033993edc96886449d398fdd2ac5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:22.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:22 smithi203 bash[68400]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2 2024-04-22T00:14:22.666 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:22 smithi203 podman[68429]: 2024-04-22 00:14:22.369515397 +0000 UTC m=+0.648697312 container remove 39f2a7f5f7423bfe733cc4df09a496766033993edc96886449d398fdd2ac5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:23.013 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:22 smithi203 podman[68517]: 2024-04-22 00:14:22.572077729 +0000 UTC m=+0.018981505 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:23.013 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:22 smithi203 podman[68517]: 2024-04-22 00:14:22.70869755 +0000 UTC m=+0.155601310 container create 562f7dd3b0a176c780d89b3c2badf763cfcb3a6d8aa1ec985d4ea5a55b315e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD) 2024-04-22T00:14:23.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:23 smithi203 podman[68517]: 2024-04-22 00:14:23.016025206 +0000 UTC m=+0.462928966 container init 562f7dd3b0a176c780d89b3c2badf763cfcb3a6d8aa1ec985d4ea5a55b315e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T00:14:23.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:23 smithi203 podman[68517]: 2024-04-22 00:14:23.01956216 +0000 UTC m=+0.466465925 container start 562f7dd3b0a176c780d89b3c2badf763cfcb3a6d8aa1ec985d4ea5a55b315e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T00:14:23.285 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:23 smithi203 podman[68517]: 2024-04-22 00:14:23.141298715 +0000 UTC m=+0.588202476 container attach 562f7dd3b0a176c780d89b3c2badf763cfcb3a6d8aa1ec985d4ea5a55b315e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T00:14:23.892 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 22 00:14:23 smithi203 podman[68517]: 2024-04-22 00:14:23.57555683 +0000 UTC m=+1.022460610 container died 562f7dd3b0a176c780d89b3c2badf763cfcb3a6d8aa1ec985d4ea5a55b315e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:24.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T00:14:24.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:24.228 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T00:14:24.229 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T00:14:24.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.3 2024-04-22T00:14:24.541 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:24 smithi038 systemd[1]: Stopping Ceph osd.3 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:24.963 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3[49358]: 2024-04-22T00:14:24.542+0000 7f418e220700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:24.963 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3[49358]: 2024-04-22T00:14:24.542+0000 7f418e220700 -1 osd.3 105 *** Got signal Terminated *** 2024-04-22T00:14:24.963 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:24 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3[49358]: 2024-04-22T00:14:24.542+0000 7f418e220700 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:30.691 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:30 smithi038 podman[104644]: 2024-04-22 00:14:30.411758536 +0000 UTC m=+5.983886253 container died e2dae62347159db5f818fb6f0faa7357be9a8975c065c3fa997f1ba865c10bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T00:14:30.963 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:30 smithi038 podman[104644]: 2024-04-22 00:14:30.775560514 +0000 UTC m=+6.347688230 container cleanup e2dae62347159db5f818fb6f0faa7357be9a8975c065c3fa997f1ba865c10bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:30.963 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:30 smithi038 bash[104644]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3 2024-04-22T00:14:31.360 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104660]: 2024-04-22 00:14:31.060229471 +0000 UTC m=+0.643309948 container remove e2dae62347159db5f818fb6f0faa7357be9a8975c065c3fa997f1ba865c10bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T00:14:31.636 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.266757331 +0000 UTC m=+0.018601687 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:31.636 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.377861495 +0000 UTC m=+0.129705853 container create b8557b1e8140313f054416e5ba010a1c93b11bb53b0056631c273107d5b7a7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T00:14:31.944 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.638979562 +0000 UTC m=+0.390823882 container init b8557b1e8140313f054416e5ba010a1c93b11bb53b0056631c273107d5b7a7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:31.944 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.642431569 +0000 UTC m=+0.394275898 container start b8557b1e8140313f054416e5ba010a1c93b11bb53b0056631c273107d5b7a7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8) 2024-04-22T00:14:31.944 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.747456227 +0000 UTC m=+0.499300547 container attach b8557b1e8140313f054416e5ba010a1c93b11bb53b0056631c273107d5b7a7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:32.213 INFO:journalctl@ceph.osd.3.smithi038.stdout:Apr 22 00:14:31 smithi038 podman[104751]: 2024-04-22 00:14:31.9467552 +0000 UTC m=+0.698599529 container died b8557b1e8140313f054416e5ba010a1c93b11bb53b0056631c273107d5b7a7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, ceph=True, GIT_CLEAN=True) 2024-04-22T00:14:32.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T00:14:32.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:32.643 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T00:14:32.643 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T00:14:32.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.4 2024-04-22T00:14:32.953 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:32 smithi111 systemd[1]: Stopping Ceph osd.4 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:33.332 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:32 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4[42805]: 2024-04-22T00:14:32.954+0000 7faf20cda700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:33.332 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:32 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4[42805]: 2024-04-22T00:14:32.954+0000 7faf20cda700 -1 osd.4 105 *** Got signal Terminated *** 2024-04-22T00:14:33.332 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:32 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4[42805]: 2024-04-22T00:14:32.954+0000 7faf20cda700 -1 osd.4 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:39.053 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:38 smithi111 podman[67901]: 2024-04-22 00:14:38.766836797 +0000 UTC m=+5.926409436 container died 13c65d2a7080ade045797d9c04686d85011b66d78beea0618d84177a595c0bda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T00:14:39.332 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:39 smithi111 podman[67901]: 2024-04-22 00:14:39.162785405 +0000 UTC m=+6.322358031 container cleanup 13c65d2a7080ade045797d9c04686d85011b66d78beea0618d84177a595c0bda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T00:14:39.332 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:39 smithi111 bash[67901]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4 2024-04-22T00:14:39.756 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:39 smithi111 podman[67918]: 2024-04-22 00:14:39.464166489 +0000 UTC m=+0.696087490 container remove 13c65d2a7080ade045797d9c04686d85011b66d78beea0618d84177a595c0bda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T00:14:40.023 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:39 smithi111 podman[68013]: 2024-04-22 00:14:39.661392807 +0000 UTC m=+0.017705580 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:40.024 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:39 smithi111 podman[68013]: 2024-04-22 00:14:39.79018095 +0000 UTC m=+0.146493724 container create 7349ccf050eb9cbb31fcd09dcc0400a250ca178438ab0d128981fdbbb157067c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T00:14:40.024 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:40 smithi111 podman[68013]: 2024-04-22 00:14:40.026306199 +0000 UTC m=+0.382618973 container init 7349ccf050eb9cbb31fcd09dcc0400a250ca178438ab0d128981fdbbb157067c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-22T00:14:40.319 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:40 smithi111 podman[68013]: 2024-04-22 00:14:40.029515437 +0000 UTC m=+0.385828211 container start 7349ccf050eb9cbb31fcd09dcc0400a250ca178438ab0d128981fdbbb157067c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:40.319 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:40 smithi111 podman[68013]: 2024-04-22 00:14:40.126288787 +0000 UTC m=+0.482601572 container attach 7349ccf050eb9cbb31fcd09dcc0400a250ca178438ab0d128981fdbbb157067c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-22T00:14:40.582 INFO:journalctl@ceph.osd.4.smithi111.stdout:Apr 22 00:14:40 smithi111 podman[68013]: 2024-04-22 00:14:40.322048918 +0000 UTC m=+0.678361697 container died 7349ccf050eb9cbb31fcd09dcc0400a250ca178438ab0d128981fdbbb157067c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T00:14:40.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T00:14:41.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:41.014 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T00:14:41.014 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T00:14:41.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.5 2024-04-22T00:14:41.329 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:41 smithi203 systemd[1]: Stopping Ceph osd.5 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:41.785 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5[43412]: 2024-04-22T00:14:41.330+0000 7fa145ec0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:41.785 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5[43412]: 2024-04-22T00:14:41.330+0000 7fa145ec0700 -1 osd.5 105 *** Got signal Terminated *** 2024-04-22T00:14:41.785 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:41 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5[43412]: 2024-04-22T00:14:41.330+0000 7fa145ec0700 -1 osd.5 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:47.621 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:47 smithi203 podman[68622]: 2024-04-22 00:14:47.335131372 +0000 UTC m=+6.126631487 container died 7cf9b807c3099de08b0c4a19bcd7883ce4b025eb07a25cb649b6be9c77da78cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T00:14:48.015 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:47 smithi203 podman[68622]: 2024-04-22 00:14:47.714812925 +0000 UTC m=+6.506313039 container cleanup 7cf9b807c3099de08b0c4a19bcd7883ce4b025eb07a25cb649b6be9c77da78cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:48.015 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:47 smithi203 bash[68622]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5 2024-04-22T00:14:48.285 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68639]: 2024-04-22 00:14:48.017295655 +0000 UTC m=+0.681160484 container remove 7cf9b807c3099de08b0c4a19bcd7883ce4b025eb07a25cb649b6be9c77da78cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:48.593 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.214950695 +0000 UTC m=+0.018143857 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:48.594 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.351746224 +0000 UTC m=+0.154939353 container create 313bb9d4f0669d79ad0023afc8cc4ba20aa400b9b633e93de7343cfaca0172be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T00:14:48.594 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.596316414 +0000 UTC m=+0.399509543 container init 313bb9d4f0669d79ad0023afc8cc4ba20aa400b9b633e93de7343cfaca0172be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:48.883 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.599353605 +0000 UTC m=+0.402546735 container start 313bb9d4f0669d79ad0023afc8cc4ba20aa400b9b633e93de7343cfaca0172be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:48.883 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.696741876 +0000 UTC m=+0.499935020 container attach 313bb9d4f0669d79ad0023afc8cc4ba20aa400b9b633e93de7343cfaca0172be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:14:49.213 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 22 00:14:48 smithi203 podman[68733]: 2024-04-22 00:14:48.885909337 +0000 UTC m=+0.689102465 container died 313bb9d4f0669d79ad0023afc8cc4ba20aa400b9b633e93de7343cfaca0172be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416) 2024-04-22T00:14:49.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T00:14:49.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:49.599 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T00:14:49.599 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T00:14:49.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.6 2024-04-22T00:14:49.927 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:49 smithi038 systemd[1]: Stopping Ceph osd.6 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:50.202 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:49 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6[56355]: 2024-04-22T00:14:49.928+0000 7fea4460f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:50.202 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:49 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6[56355]: 2024-04-22T00:14:49.928+0000 7fea4460f700 -1 osd.6 105 *** Got signal Terminated *** 2024-04-22T00:14:50.202 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:49 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6[56355]: 2024-04-22T00:14:49.928+0000 7fea4460f700 -1 osd.6 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:14:56.179 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:55 smithi038 podman[104858]: 2024-04-22 00:14:55.786275475 +0000 UTC m=+5.984339438 container died 8d0e4445b07a1861f665ddaaecc404db75025ef46cd3d730955d8b0cb65b4865 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:56.463 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:56 smithi038 podman[104858]: 2024-04-22 00:14:56.270267546 +0000 UTC m=+6.468331500 container cleanup 8d0e4445b07a1861f665ddaaecc404db75025ef46cd3d730955d8b0cb65b4865 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T00:14:56.463 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:56 smithi038 bash[104858]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6 2024-04-22T00:14:56.895 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:56 smithi038 podman[104888]: 2024-04-22 00:14:56.546981719 +0000 UTC m=+0.759610461 container remove 8d0e4445b07a1861f665ddaaecc404db75025ef46cd3d730955d8b0cb65b4865 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:14:57.147 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:56 smithi038 podman[104975]: 2024-04-22 00:14:56.80058237 +0000 UTC m=+0.018509919 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:14:57.148 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:56 smithi038 podman[104975]: 2024-04-22 00:14:56.914397505 +0000 UTC m=+0.132325027 container create ac40ff561dd120ab58bd5c78ec4b8eabc4209a201757c23d8c5d6f8c1b31d19f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T00:14:57.456 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:57 smithi038 podman[104975]: 2024-04-22 00:14:57.150333325 +0000 UTC m=+0.368260846 container init ac40ff561dd120ab58bd5c78ec4b8eabc4209a201757c23d8c5d6f8c1b31d19f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD) 2024-04-22T00:14:57.456 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:57 smithi038 podman[104975]: 2024-04-22 00:14:57.153856258 +0000 UTC m=+0.371783788 container start ac40ff561dd120ab58bd5c78ec4b8eabc4209a201757c23d8c5d6f8c1b31d19f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:14:57.456 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:57 smithi038 podman[104975]: 2024-04-22 00:14:57.258770941 +0000 UTC m=+0.476698472 container attach ac40ff561dd120ab58bd5c78ec4b8eabc4209a201757c23d8c5d6f8c1b31d19f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T00:14:57.713 INFO:journalctl@ceph.osd.6.smithi038.stdout:Apr 22 00:14:57 smithi038 podman[104975]: 2024-04-22 00:14:57.458598023 +0000 UTC m=+0.676525559 container died ac40ff561dd120ab58bd5c78ec4b8eabc4209a201757c23d8c5d6f8c1b31d19f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T00:14:58.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T00:14:58.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:14:58.301 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T00:14:58.301 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T00:14:58.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.7 2024-04-22T00:14:58.582 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:14:58 smithi111 systemd[1]: Stopping Ceph osd.7 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:14:59.082 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:14:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7[49513]: 2024-04-22T00:14:58.606+0000 7f4b48c42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:14:59.082 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:14:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7[49513]: 2024-04-22T00:14:58.606+0000 7f4b48c42700 -1 osd.7 105 *** Got signal Terminated *** 2024-04-22T00:14:59.082 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:14:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7[49513]: 2024-04-22T00:14:58.606+0000 7f4b48c42700 -1 osd.7 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:15:04.775 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:04 smithi111 podman[68118]: 2024-04-22 00:15:04.515159143 +0000 UTC m=+6.020281936 container died 05bf97d34aeaa6e480c04126cd34dc06d7fec934946e47b25c202821c05c2971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T00:15:05.082 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:04 smithi111 podman[68118]: 2024-04-22 00:15:04.873204158 +0000 UTC m=+6.378326951 container cleanup 05bf97d34aeaa6e480c04126cd34dc06d7fec934946e47b25c202821c05c2971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-22T00:15:05.082 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:04 smithi111 bash[68118]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7 2024-04-22T00:15:05.520 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68147]: 2024-04-22 00:15:05.216212718 +0000 UTC m=+0.696209048 container remove 05bf97d34aeaa6e480c04126cd34dc06d7fec934946e47b25c202821c05c2971 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T00:15:05.808 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68238]: 2024-04-22 00:15:05.425465332 +0000 UTC m=+0.017501922 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:15:05.808 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68238]: 2024-04-22 00:15:05.550126416 +0000 UTC m=+0.142162997 container create ce2e82fdd632e840c5aca4f68b2d0f1ea58582c252dc4a5027c2245d46ec468f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:15:06.083 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68238]: 2024-04-22 00:15:05.810873775 +0000 UTC m=+0.402910357 container init ce2e82fdd632e840c5aca4f68b2d0f1ea58582c252dc4a5027c2245d46ec468f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-820a2f8) 2024-04-22T00:15:06.083 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68238]: 2024-04-22 00:15:05.814689322 +0000 UTC m=+0.406725905 container start ce2e82fdd632e840c5aca4f68b2d0f1ea58582c252dc4a5027c2245d46ec468f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T00:15:06.083 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:05 smithi111 podman[68238]: 2024-04-22 00:15:05.877631684 +0000 UTC m=+0.469668288 container attach ce2e82fdd632e840c5aca4f68b2d0f1ea58582c252dc4a5027c2245d46ec468f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-22T00:15:06.378 INFO:journalctl@ceph.osd.7.smithi111.stdout:Apr 22 00:15:06 smithi111 podman[68238]: 2024-04-22 00:15:06.106980131 +0000 UTC m=+0.699016722 container died ce2e82fdd632e840c5aca4f68b2d0f1ea58582c252dc4a5027c2245d46ec468f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T00:15:06.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T00:15:06.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:06.724 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T00:15:06.724 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T00:15:06.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.8 2024-04-22T00:15:07.035 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:06 smithi203 systemd[1]: Stopping Ceph osd.8 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:07.332 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:07 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:07.070+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:07.463 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:07 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:07.344+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:07.535 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:07 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8[50287]: 2024-04-22T00:15:07.037+0000 7fbe7c9d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:07.535 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:07 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8[50287]: 2024-04-22T00:15:07.037+0000 7fbe7c9d9700 -1 osd.8 105 *** Got signal Terminated *** 2024-04-22T00:15:07.535 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:07 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8[50287]: 2024-04-22T00:15:07.037+0000 7fbe7c9d9700 -1 osd.8 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:15:08.539 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:08 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:08.069+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:08.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:08 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:08.096+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:08.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:08 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:08.305+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:09.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:09 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:09.043+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:09.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:09 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:09.092+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:09.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:09 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:09.352+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:10.332 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:10 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:10.068+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:10.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:10 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:10.046+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:10.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:10 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:10.329+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:11.332 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:11 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:11.055+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:11.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:11 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:11.037+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:11.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:11 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:11.316+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:12.332 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:12 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:12.082+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:12.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:12 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:12.085+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:12.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:12 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:12.340+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:12.994 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:12 smithi203 podman[68856]: 2024-04-22 00:15:12.687991563 +0000 UTC m=+5.765742838 container died 84de3f49b305715d6414bea0dc85eebf2c1f2dd449e0486dcdf0c9e2a655e499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-22T00:15:13.285 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:13 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:13.107+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:13.285 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68856]: 2024-04-22 00:15:13.096264441 +0000 UTC m=+6.174015712 container cleanup 84de3f49b305715d6414bea0dc85eebf2c1f2dd449e0486dcdf0c9e2a655e499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T00:15:13.285 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 bash[68856]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8 2024-04-22T00:15:13.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:13 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:13.089+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:13.664 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68873]: 2024-04-22 00:15:13.331387413 +0000 UTC m=+0.645616629 container remove 84de3f49b305715d6414bea0dc85eebf2c1f2dd449e0486dcdf0c9e2a655e499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T00:15:13.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:13 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:13.339+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:13.924 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68962]: 2024-04-22 00:15:13.569822768 +0000 UTC m=+0.018670418 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:15:13.924 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68962]: 2024-04-22 00:15:13.690886096 +0000 UTC m=+0.139733765 container create f737a3a71fb0c06d11867539be224a1fc5b2105f5b23f8ad7086c14c27e90254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:13.925 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68962]: 2024-04-22 00:15:13.926817201 +0000 UTC m=+0.375664834 container init f737a3a71fb0c06d11867539be224a1fc5b2105f5b23f8ad7086c14c27e90254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T00:15:14.222 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:14 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:14.154+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:14.222 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:14 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:14.154+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:14.223 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:13 smithi203 podman[68962]: 2024-04-22 00:15:13.930122848 +0000 UTC m=+0.378970481 container start f737a3a71fb0c06d11867539be224a1fc5b2105f5b23f8ad7086c14c27e90254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-22T00:15:14.223 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:14 smithi203 podman[68962]: 2024-04-22 00:15:14.010188048 +0000 UTC m=+0.459035688 container attach f737a3a71fb0c06d11867539be224a1fc5b2105f5b23f8ad7086c14c27e90254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T00:15:14.494 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 22 00:15:14 smithi203 podman[68962]: 2024-04-22 00:15:14.224916763 +0000 UTC m=+0.673764402 container died f737a3a71fb0c06d11867539be224a1fc5b2105f5b23f8ad7086c14c27e90254 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:15:14.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:14 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:14.119+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:14.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:14 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:14.361+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:14.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T00:15:14.923 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:14.923 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T00:15:14.924 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T00:15:14.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.9 2024-04-22T00:15:15.213 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:14 smithi038 systemd[1]: Stopping Ceph osd.9 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:15.434 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:15.158+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:15.434 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:15.158+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:15.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:15 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:15.147+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:15.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:15 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:15.147+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:15.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:15 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:15.313+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:15.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:15 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:15.356+0000 7fc988222700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:15.714 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:15 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:15.356+0000 7fc988222700 -1 osd.9 105 *** Got signal Terminated *** 2024-04-22T00:15:15.714 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:15 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:15.356+0000 7fc988222700 -1 osd.9 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:15:16.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:16 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:16.208+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:16.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:16 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:16.208+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:16.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:16 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:16.194+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:16.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:16 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:16.194+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:16.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:16 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:16.316+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:16.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:16 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:16.316+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.984916+0000 front 2024-04-22T00:14:11.984836+0000 (oldest deadline 2024-04-22T00:15:15.484897+0000) 2024-04-22T00:15:17.436 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:17 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:17.158+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:17.437 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:17 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:17.158+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:17.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:17.216+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:17.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:17 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:17.216+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:17.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:17.335+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:17.714 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:17 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:17.335+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.984916+0000 front 2024-04-22T00:14:11.984836+0000 (oldest deadline 2024-04-22T00:15:15.484897+0000) 2024-04-22T00:15:18.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:18.113+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:18.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:18 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:18.113+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:18.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:18 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:18.201+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:18.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:18 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:18.201+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:18.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:18.301+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:18.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:18 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:18.301+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.984916+0000 front 2024-04-22T00:14:11.984836+0000 (oldest deadline 2024-04-22T00:15:15.484897+0000) 2024-04-22T00:15:19.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:19 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:19.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:19.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:19 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:19.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:19.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:19 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:19.221+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:19.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:19 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:19.221+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:19.615 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:19 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:19.273+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:19.615 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:19 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:19.273+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.984916+0000 front 2024-04-22T00:14:11.984836+0000 (oldest deadline 2024-04-22T00:15:15.484897+0000) 2024-04-22T00:15:20.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:20 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:20.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:20.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:20 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:20.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:20.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:20 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:20.198+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:20.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:20 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:20.198+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:20.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:20 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:20.299+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:03.183696+0000 front 2024-04-22T00:14:03.183528+0000 (oldest deadline 2024-04-22T00:15:07.283772+0000) 2024-04-22T00:15:20.714 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:20 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9[63229]: 2024-04-22T00:15:20.299+0000 7fc984816700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.984916+0000 front 2024-04-22T00:14:11.984836+0000 (oldest deadline 2024-04-22T00:15:15.484897+0000) 2024-04-22T00:15:21.431 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:20 smithi038 podman[105081]: 2024-04-22 00:15:20.979573897 +0000 UTC m=+5.851198409 container died a4c607ca3e0940196518477e4e99865c54dc52d242475c55d326a39beba93312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T00:15:21.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:21 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:21.136+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:21.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:21 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:21.136+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:21.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:21 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:21.195+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:21.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:21 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:21.195+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:21.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:21 smithi038 podman[105081]: 2024-04-22 00:15:21.532322009 +0000 UTC m=+6.403946521 container cleanup a4c607ca3e0940196518477e4e99865c54dc52d242475c55d326a39beba93312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T00:15:21.713 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:21 smithi038 bash[105081]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9 2024-04-22T00:15:22.155 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:21 smithi038 podman[105098]: 2024-04-22 00:15:21.825161363 +0000 UTC m=+0.838601657 container remove a4c607ca3e0940196518477e4e99865c54dc52d242475c55d326a39beba93312 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.061056035 +0000 UTC m=+0.018671465 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.167765105 +0000 UTC m=+0.125380533 container create 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.40349335 +0000 UTC m=+0.361108777 container init 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T00:15:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.407135679 +0000 UTC m=+0.364751107 container start 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T00:15:22.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:22 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:22.171+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:22.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:22 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:22.171+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:22.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:22 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:22.155+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:22.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:22 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:22.155+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:22.963 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.497540704 +0000 UTC m=+0.455156143 container attach 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:22.964 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:22 smithi038 podman[105188]: 2024-04-22 00:15:22.712839772 +0000 UTC m=+0.670455206 container died 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T00:15:23.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T00:15:23.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:23 smithi038 podman[105214]: 2024-04-22 00:15:23.206870188 +0000 UTC m=+0.488035588 container remove 35e7d409ccc25516e4744e399d532c2e4a66cc913a549e1fcfb21d3d7d3aef87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T00:15:23.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:23 smithi038 systemd[1]: ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.9.service: Deactivated successfully. 2024-04-22T00:15:23.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:23 smithi038 systemd[1]: Stopped Ceph osd.9 for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-22T00:15:23.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Apr 22 00:15:23 smithi038 systemd[1]: ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.9.service: Consumed 10min 38.813s CPU time. 2024-04-22T00:15:23.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:23.267 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T00:15:23.267 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T00:15:23.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.10 2024-04-22T00:15:23.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:23 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:23.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:23.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:23 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:23.149+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:23.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:23.186+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:23.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:23.186+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:23.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 systemd[1]: Stopping Ceph osd.10 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:24.082 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:23.590+0000 7f3e71f7d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:24.082 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:23.590+0000 7f3e71f7d700 -1 osd.10 105 *** Got signal Terminated *** 2024-04-22T00:15:24.082 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:23 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:23.590+0000 7f3e71f7d700 -1 osd.10 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:15:24.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:24 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:24.181+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:24.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:24 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:24.181+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:24.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:24 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:24.181+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:24.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:24 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:24.158+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:24.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:24 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:24.158+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:24.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:24 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:24.158+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.020745+0000 front 2024-04-22T00:14:18.020677+0000 (oldest deadline 2024-04-22T00:15:23.920651+0000) 2024-04-22T00:15:25.285 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:25 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:25.160+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:25.285 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:25 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:25.160+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:25.285 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:25 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:25.160+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:25.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:25 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:25.117+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:25.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:25 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:25.117+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:25.583 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:25 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:25.117+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.020745+0000 front 2024-04-22T00:14:18.020677+0000 (oldest deadline 2024-04-22T00:15:23.920651+0000) 2024-04-22T00:15:26.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:26 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:26.192+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:26.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:26 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:26.192+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:26.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:26 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:26.192+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:26.583 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:26 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:26.130+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:26.583 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:26 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:26.130+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:26.583 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:26 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:26.130+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.020745+0000 front 2024-04-22T00:14:18.020677+0000 (oldest deadline 2024-04-22T00:15:23.920651+0000) 2024-04-22T00:15:27.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:27 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:27.186+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:27.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:27 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:27.186+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:27.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:27 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:27.186+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:27.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:27.108+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:27.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:27.108+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:27.582 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:27 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:27.108+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.020745+0000 front 2024-04-22T00:14:18.020677+0000 (oldest deadline 2024-04-22T00:15:23.920651+0000) 2024-04-22T00:15:28.483 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:28 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:28.102+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:14:01.618777+0000 front 2024-04-22T00:14:01.618805+0000 (oldest deadline 2024-04-22T00:15:06.318671+0000) 2024-04-22T00:15:28.484 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:28 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:28.102+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:11.619580+0000 front 2024-04-22T00:14:11.619721+0000 (oldest deadline 2024-04-22T00:15:15.119721+0000) 2024-04-22T00:15:28.484 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:28 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10[56390]: 2024-04-22T00:15:28.102+0000 7f3e6e571700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.020745+0000 front 2024-04-22T00:14:18.020677+0000 (oldest deadline 2024-04-22T00:15:23.920651+0000) 2024-04-22T00:15:28.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:28 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:28.163+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:28.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:28 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:28.163+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:28.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:28 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:28.163+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:29.522 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:29 smithi111 podman[68342]: 2024-04-22 00:15:29.213211081 +0000 UTC m=+5.754818879 container died 8e83acdf498f7f01d58d07719565630789d9a9db5e6ebb6db580ddd717a9b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T00:15:29.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:29 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:29.200+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:29.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:29 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:29.200+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:29.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:29 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:29.200+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:29.832 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:29 smithi111 podman[68342]: 2024-04-22 00:15:29.524479477 +0000 UTC m=+6.066087219 container cleanup 8e83acdf498f7f01d58d07719565630789d9a9db5e6ebb6db580ddd717a9b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T00:15:29.832 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:29 smithi111 bash[68342]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10 2024-04-22T00:15:29.832 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:29 smithi111 podman[68359]: 2024-04-22 00:15:29.725535508 +0000 UTC m=+0.504113041 container remove 8e83acdf498f7f01d58d07719565630789d9a9db5e6ebb6db580ddd717a9b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T00:15:30.333 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:30.043221884 +0000 UTC m=+0.115098135 container create 1bcf53d0bcfb1011fba08dd5b7643184140a0fa0fdcb5065560a2b3879b5352f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T00:15:30.333 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:29.946425062 +0000 UTC m=+0.018301320 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:15:30.333 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:30.262214878 +0000 UTC m=+0.334091133 container init 1bcf53d0bcfb1011fba08dd5b7643184140a0fa0fdcb5065560a2b3879b5352f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-22T00:15:30.333 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:30.265231431 +0000 UTC m=+0.337107683 container start 1bcf53d0bcfb1011fba08dd5b7643184140a0fa0fdcb5065560a2b3879b5352f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-04-22T00:15:30.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:30 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:30.184+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:30.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:30 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:30.184+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:30.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:30 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:30.184+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:30.832 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:30.337062496 +0000 UTC m=+0.408938749 container attach 1bcf53d0bcfb1011fba08dd5b7643184140a0fa0fdcb5065560a2b3879b5352f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:30.833 INFO:journalctl@ceph.osd.10.smithi111.stdout:Apr 22 00:15:30 smithi111 podman[68452]: 2024-04-22 00:15:30.552510222 +0000 UTC m=+0.624386474 container died 1bcf53d0bcfb1011fba08dd5b7643184140a0fa0fdcb5065560a2b3879b5352f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T00:15:31.121 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T00:15:31.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:31.159 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T00:15:31.160 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T00:15:31.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.11 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.193+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.193+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.193+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 systemd[1]: Stopping Ceph osd.11 for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.484+0000 7f8a71245700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.484+0000 7f8a71245700 -1 osd.11 105 *** Got signal Terminated *** 2024-04-22T00:15:31.484 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:31 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:31.484+0000 7f8a71245700 -1 osd.11 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T00:15:32.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:32 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:32.172+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:32.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:32 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:32.172+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:32.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:32 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:32.172+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:33.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:33 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:33.154+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:33.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:33 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:33.154+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:33.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:33 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:33.154+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:34.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:34 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:34.150+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:34.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:34 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:34.150+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:34.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:34 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:34.150+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:35.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:35 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:35.130+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:35.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:35 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:35.130+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:35.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:35 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:35.130+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:36.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:36 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:36.180+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-04-22T00:13:59.014685+0000 front 2024-04-22T00:13:59.014901+0000 (oldest deadline 2024-04-22T00:15:07.815259+0000) 2024-04-22T00:15:36.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:36 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:36.180+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.111:6804 osd.1 since back 2024-04-22T00:14:07.815729+0000 front 2024-04-22T00:14:07.815744+0000 (oldest deadline 2024-04-22T00:15:13.715710+0000) 2024-04-22T00:15:36.535 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:36 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11[56903]: 2024-04-22T00:15:36.180+0000 7f8a6d839700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-22T00:14:18.917401+0000 front 2024-04-22T00:14:18.917179+0000 (oldest deadline 2024-04-22T00:15:23.617388+0000) 2024-04-22T00:15:37.402 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:37 smithi203 podman[69080]: 2024-04-22 00:15:37.135950108 +0000 UTC m=+5.782351019 container died 6bebf3c17c4d734461311c70155ffde0e1303225c5764fd285635fac4817f520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T00:15:37.785 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:37 smithi203 podman[69080]: 2024-04-22 00:15:37.49467193 +0000 UTC m=+6.141072831 container cleanup 6bebf3c17c4d734461311c70155ffde0e1303225c5764fd285635fac4817f520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T00:15:37.785 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:37 smithi203 bash[69080]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11 2024-04-22T00:15:38.062 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:37 smithi203 podman[69100]: 2024-04-22 00:15:37.788432396 +0000 UTC m=+0.643380074 container remove 6bebf3c17c4d734461311c70155ffde0e1303225c5764fd285635fac4817f520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-22T00:15:38.365 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:38.064362609 +0000 UTC m=+0.094389206 container create 48bff4058951865510d065ed7e630a1ce1146d8d08c5c1e7676be294f3367509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0) 2024-04-22T00:15:38.366 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:37.988127712 +0000 UTC m=+0.018154329 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-22T00:15:38.366 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:38.259999252 +0000 UTC m=+0.290025859 container init 48bff4058951865510d065ed7e630a1ce1146d8d08c5c1e7676be294f3367509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-22T00:15:38.366 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:38.263354076 +0000 UTC m=+0.293380683 container start 48bff4058951865510d065ed7e630a1ce1146d8d08c5c1e7676be294f3367509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-22T00:15:38.366 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:38.367850479 +0000 UTC m=+0.397877077 container attach 48bff4058951865510d065ed7e630a1ce1146d8d08c5c1e7676be294f3367509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True) 2024-04-22T00:15:38.785 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 22 00:15:38 smithi203 podman[69187]: 2024-04-22 00:15:38.54849739 +0000 UTC m=+0.578523989 container died 48bff4058951865510d065ed7e630a1ce1146d8d08c5c1e7676be294f3367509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:39.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T00:15:39.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:39.168 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T00:15:39.168 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T00:15:39.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.a 2024-04-22T00:15:39.463 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:39 smithi038 systemd[1]: Stopping Ceph mds.a for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:39.963 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:39 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: 2024-04-22T00:15:39.508+0000 7f5a43f6c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:39.963 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:39 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a[69880]: 2024-04-22T00:15:39.508+0000 7f5a43f6c700 -1 mds.a *** got signal Terminated *** 2024-04-22T00:15:42.049 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:41 smithi038 podman[105294]: 2024-04-22 00:15:41.797980467 +0000 UTC m=+2.434620613 container died 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T00:15:42.324 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 podman[105294]: 2024-04-22 00:15:42.141785709 +0000 UTC m=+2.778425859 container cleanup 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T00:15:42.324 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 bash[105294]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a 2024-04-22T00:15:42.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T00:15:42.577 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 podman[105308]: 2024-04-22 00:15:42.326302851 +0000 UTC m=+0.527093347 container remove 0ad9c3a25426ac9e90ef51bc831e600189c3ae97f3c9723cf645291df54f9045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:15:42.577 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 systemd[1]: ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.a.service: Deactivated successfully. 2024-04-22T00:15:42.577 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 systemd[1]: Stopped Ceph mds.a for eab6f840-0039-11ef-bc93-c7b262605968. 2024-04-22T00:15:42.578 INFO:journalctl@ceph.mds.a.smithi038.stdout:Apr 22 00:15:42 smithi038 systemd[1]: ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.178s CPU time. 2024-04-22T00:15:42.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:42.591 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T00:15:42.591 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T00:15:42.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.d 2024-04-22T00:15:42.910 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:42 smithi038 systemd[1]: Stopping Ceph mds.d for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:43.213 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d[68888]: 2024-04-22T00:15:42.910+0000 7fbb92508700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:43.214 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:42 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d[68888]: 2024-04-22T00:15:42.910+0000 7fbb92508700 -1 mds.d *** got signal Terminated *** 2024-04-22T00:15:44.293 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:43 smithi038 podman[105414]: 2024-04-22 00:15:43.998398728 +0000 UTC m=+1.185248398 container died 1e801e466c4e507b545bed087905bef29ff0b4e7478c9206557bd9f89d2e2922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T00:15:44.293 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:44 smithi038 podman[105414]: 2024-04-22 00:15:44.295616593 +0000 UTC m=+1.482466263 container cleanup 1e801e466c4e507b545bed087905bef29ff0b4e7478c9206557bd9f89d2e2922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-22T00:15:44.711 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:44 smithi038 bash[105414]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d 2024-04-22T00:15:44.712 INFO:journalctl@ceph.mds.d.smithi038.stdout:Apr 22 00:15:44 smithi038 podman[105428]: 2024-04-22 00:15:44.513831382 +0000 UTC m=+0.513727841 container remove 1e801e466c4e507b545bed087905bef29ff0b4e7478c9206557bd9f89d2e2922 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-d, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-22T00:15:44.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T00:15:44.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:44.762 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T00:15:44.762 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T00:15:44.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.g 2024-04-22T00:15:45.080 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:44 smithi038 systemd[1]: Stopping Ceph mds.g for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:45.463 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:45 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: 2024-04-22T00:15:45.081+0000 7f23a18bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:45.464 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:45 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g[69518]: 2024-04-22T00:15:45.081+0000 7f23a18bc700 -1 mds.g *** got signal Terminated *** 2024-04-22T00:15:49.463 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:49 smithi038 podman[105538]: 2024-04-22 00:15:49.08899841 +0000 UTC m=+4.102864903 container died 7e1a5fbaf2963811e71ccfd97611019cca9cef5c0d05ddf85908f571d374eb97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:49.923 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:49 smithi038 podman[105538]: 2024-04-22 00:15:49.683154181 +0000 UTC m=+4.697020601 container cleanup 7e1a5fbaf2963811e71ccfd97611019cca9cef5c0d05ddf85908f571d374eb97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T00:15:49.923 INFO:journalctl@ceph.mds.g.smithi038.stdout:Apr 22 00:15:49 smithi038 bash[105538]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-g 2024-04-22T00:15:50.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T00:15:50.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:50.104 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T00:15:50.104 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T00:15:50.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.j 2024-04-22T00:15:50.452 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:50 smithi038 systemd[1]: Stopping Ceph mds.j for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:50.713 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j[68539]: 2024-04-22T00:15:50.453+0000 7f4059ea1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:50.714 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:50 smithi038 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j[68539]: 2024-04-22T00:15:50.453+0000 7f4059ea1700 -1 mds.j *** got signal Terminated *** 2024-04-22T00:15:53.535 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:53 smithi038 podman[105661]: 2024-04-22 00:15:53.252985398 +0000 UTC m=+2.929681214 container died 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T00:15:53.853 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:53 smithi038 podman[105661]: 2024-04-22 00:15:53.637224415 +0000 UTC m=+3.313920235 container cleanup 212f5685aa77480e4c03de35c54412bdae22b52bf50acb4ebb1521dc292ff32e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-22T00:15:53.853 INFO:journalctl@ceph.mds.j.smithi038.stdout:Apr 22 00:15:53 smithi038 bash[105661]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-j 2024-04-22T00:15:53.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T00:15:54.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:54.016 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T00:15:54.016 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T00:15:54.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.b 2024-04-22T00:15:54.291 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:54 smithi111 systemd[1]: Stopping Ceph mds.b for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:54.291 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:54 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:15:54.292+0000 7f38e93ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:54.292 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:54 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b[61604]: 2024-04-22T00:15:54.292+0000 7f38e93ac700 -1 mds.b *** got signal Terminated *** 2024-04-22T00:15:58.049 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:57 smithi111 podman[68558]: 2024-04-22 00:15:57.797828992 +0000 UTC m=+3.594032521 container died bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T00:15:58.332 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:58 smithi111 podman[68558]: 2024-04-22 00:15:58.150685894 +0000 UTC m=+3.946889453 container cleanup bd03537b71aac0a090c4583ed6c0053ca4f122e3f28fd1d85281e53b466eafb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T00:15:58.332 INFO:journalctl@ceph.mds.b.smithi111.stdout:Apr 22 00:15:58 smithi111 bash[68558]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-b 2024-04-22T00:15:58.565 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T00:15:58.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:15:58.598 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T00:15:58.599 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T00:15:58.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.e 2024-04-22T00:15:58.832 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:15:58 smithi111 systemd[1]: Stopping Ceph mds.e for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:15:59.332 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:15:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: 2024-04-22T00:15:58.945+0000 7f88138ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:15:59.332 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:15:58 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e[60983]: 2024-04-22T00:15:58.945+0000 7f88138ed700 -1 mds.e *** got signal Terminated *** 2024-04-22T00:16:02.083 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:16:01 smithi111 podman[68679]: 2024-04-22 00:16:01.799087493 +0000 UTC m=+2.983701878 container died e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-820a2f8) 2024-04-22T00:16:02.405 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:16:02 smithi111 podman[68679]: 2024-04-22 00:16:02.104324127 +0000 UTC m=+3.288938514 container cleanup e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T00:16:02.406 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:16:02 smithi111 bash[68679]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e 2024-04-22T00:16:02.406 INFO:journalctl@ceph.mds.e.smithi111.stdout:Apr 22 00:16:02 smithi111 podman[68705]: 2024-04-22 00:16:02.313978693 +0000 UTC m=+0.515860658 container remove e1b5c303d49d50ad9bd6dc9769d0c07b31d111fb953fcac49dcb61dfcdf2e999 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T00:16:02.421 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T00:16:02.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:02.454 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T00:16:02.455 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T00:16:02.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.h 2024-04-22T00:16:02.790 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:02 smithi111 systemd[1]: Stopping Ceph mds.h for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:03.082 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: 2024-04-22T00:16:02.791+0000 7f871b6e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:03.082 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:02 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h[61295]: 2024-04-22T00:16:02.791+0000 7f871b6e9700 -1 mds.h *** got signal Terminated *** 2024-04-22T00:16:06.082 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:05 smithi111 podman[68811]: 2024-04-22 00:16:05.792247293 +0000 UTC m=+3.104287063 container died e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T00:16:06.407 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:06 smithi111 podman[68811]: 2024-04-22 00:16:06.174860509 +0000 UTC m=+3.486900325 container cleanup e40b0aeba82c9ad0d3a856fc9ed79f23e69b47a02518112ce148dbec07302457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8) 2024-04-22T00:16:06.407 INFO:journalctl@ceph.mds.h.smithi111.stdout:Apr 22 00:16:06 smithi111 bash[68811]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-h 2024-04-22T00:16:06.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T00:16:06.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:06.563 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T00:16:06.563 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T00:16:06.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.k 2024-04-22T00:16:06.832 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:06 smithi111 systemd[1]: Stopping Ceph mds.k for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:07.332 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:06 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: 2024-04-22T00:16:06.869+0000 7fb439bdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:07.333 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:06 smithi111 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k[60662]: 2024-04-22T00:16:06.869+0000 7fb439bdd700 -1 mds.k *** got signal Terminated *** 2024-04-22T00:16:10.070 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:09 smithi111 podman[68931]: 2024-04-22 00:16:09.799355968 +0000 UTC m=+3.021588029 container died 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-22T00:16:10.332 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:10 smithi111 podman[68931]: 2024-04-22 00:16:10.145058456 +0000 UTC m=+3.367290517 container cleanup 33c4e449aa43761eff14e70b52a47fa2946e567e8df82a33d50ee0f77a81ac76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T00:16:10.332 INFO:journalctl@ceph.mds.k.smithi111.stdout:Apr 22 00:16:10 smithi111 bash[68931]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-k 2024-04-22T00:16:10.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T00:16:10.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:10.604 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T00:16:10.604 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T00:16:10.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.c 2024-04-22T00:16:10.895 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:10 smithi203 systemd[1]: Stopping Ceph mds.c for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:11.160 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:10 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: 2024-04-22T00:16:10.896+0000 7f978614f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:11.161 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:10 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c[62335]: 2024-04-22T00:16:10.896+0000 7f978614f700 -1 mds.c *** got signal Terminated *** 2024-04-22T00:16:14.535 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:14 smithi203 podman[69292]: 2024-04-22 00:16:14.260944999 +0000 UTC m=+3.468351686 container died 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T00:16:14.929 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:14 smithi203 podman[69292]: 2024-04-22 00:16:14.638750148 +0000 UTC m=+3.846156825 container cleanup 03bebce1483c724ef54ee9735b9f6fd64fb75403a5ae277ce3f608668535f7d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T00:16:15.044 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 22 00:16:14 smithi203 bash[69292]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-c 2024-04-22T00:16:15.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T00:16:15.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:15.092 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T00:16:15.092 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T00:16:15.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.f 2024-04-22T00:16:15.285 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:15 smithi203 systemd[1]: Stopping Ceph mds.f for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:15.785 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: 2024-04-22T00:16:15.408+0000 7f5c249aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:15.785 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:15 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f[62027]: 2024-04-22T00:16:15.408+0000 7f5c249aa700 -1 mds.f *** got signal Terminated *** 2024-04-22T00:16:18.533 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:18 smithi203 podman[69425]: 2024-04-22 00:16:18.269748204 +0000 UTC m=+2.962218243 container died bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T00:16:18.785 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:18 smithi203 podman[69425]: 2024-04-22 00:16:18.609566031 +0000 UTC m=+3.302036071 container cleanup bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T00:16:18.786 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:18 smithi203 bash[69425]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f 2024-04-22T00:16:18.786 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 22 00:16:18 smithi203 podman[69439]: 2024-04-22 00:16:18.768970604 +0000 UTC m=+0.490873251 container remove bbb4bd77686c7f9f37aee5abb864854ddba6c7096f8c52099fb01846c3beef77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T00:16:18.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T00:16:19.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:19.009 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T00:16:19.009 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T00:16:19.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.i 2024-04-22T00:16:19.285 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:19 smithi203 systemd[1]: Stopping Ceph mds.i for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:19.785 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:19 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: 2024-04-22T00:16:19.337+0000 7f6b14dcd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:19.785 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:19 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i[62645]: 2024-04-22T00:16:19.337+0000 7f6b14dcd700 -1 mds.i *** got signal Terminated *** 2024-04-22T00:16:22.535 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:22 smithi203 podman[69545]: 2024-04-22 00:16:22.26673607 +0000 UTC m=+3.028541767 container died 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True) 2024-04-22T00:16:22.896 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:22 smithi203 podman[69545]: 2024-04-22 00:16:22.638469673 +0000 UTC m=+3.400275371 container cleanup 93786033400e6174e93e6a684e31279db431e8d361492bdb055344faea97e4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T00:16:22.896 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 22 00:16:22 smithi203 bash[69545]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-i 2024-04-22T00:16:23.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T00:16:23.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:23.098 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T00:16:23.098 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T00:16:23.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.l 2024-04-22T00:16:23.415 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:23 smithi203 systemd[1]: Stopping Ceph mds.l for eab6f840-0039-11ef-bc93-c7b262605968... 2024-04-22T00:16:23.785 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:23 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: 2024-04-22T00:16:23.417+0000 7feec599c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T00:16:23.785 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:23 smithi203 ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l[61716]: 2024-04-22T00:16:23.417+0000 7feec599c700 -1 mds.l *** got signal Terminated *** 2024-04-22T00:16:26.532 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:26 smithi203 podman[69667]: 2024-04-22 00:16:26.269035587 +0000 UTC m=+2.952175504 container died ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T00:16:26.785 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:26 smithi203 podman[69667]: 2024-04-22 00:16:26.609090927 +0000 UTC m=+3.292230840 container cleanup ce1211fe9b598a42223f53be3868d56305e4207e3b7a832eacec0f95ce604c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T00:16:26.785 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 22 00:16:26 smithi203 bash[69667]: ceph-eab6f840-0039-11ef-bc93-c7b262605968-mds-l 2024-04-22T00:16:27.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eab6f840-0039-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T00:16:27.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T00:16:27.092 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T00:16:27.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T00:16:27.292 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:16:29.941 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T00:16:30.132 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:16:32.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T00:16:32.857 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:16:35.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:16:35.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:16:35.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T00:16:35.430 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T00:16:35.433 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi038/crash 2024-04-22T00:16:35.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash -- . 2024-04-22T00:16:35.505 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T00:16:35.505 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-22T00:16:35.508 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi111/crash 2024-04-22T00:16:35.509 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash -- . 2024-04-22T00:16:35.557 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T00:16:35.557 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-04-22T00:16:35.560 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi203/crash 2024-04-22T00:16:35.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash -- . 2024-04-22T00:16:35.594 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/eab6f840-0039-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T00:16:35.594 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-04-22T00:16:35.595 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T00:16:35.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T00:16:35.682 INFO:tasks.cephadm:Compressing logs... 2024-04-22T00:16:35.682 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T00:16:35.685 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T00:16:35.688 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T00:16:35.709 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T00:16:35.709 INFO:teuthology.orchestra.run.smithi038.stderr:: No such file or directory 2024-04-22T00:16:35.709 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T00:16:35.714 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr: 86.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T00:16:35.715 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T00:16:35.724 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T00:16:35.725 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:: No such file or directory 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T00:16:35.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T00:16:35.727 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T00:16:35.727 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T00:16:35.727 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T00:16:35.727 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T00:16:35.762 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.k.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T00:16:35.800 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-22T00:16:35.827 INFO:teuthology.orchestra.run.smithi111.stderr: 84.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T00:16:35.863 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.l.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T00:16:35.875 INFO:teuthology.orchestra.run.smithi038.stderr: 84.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T00:16:35.886 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T00:16:35.914 INFO:teuthology.orchestra.run.smithi203.stderr: 84.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T00:16:35.927 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T00:16:35.992 INFO:teuthology.orchestra.run.smithi203.stderr: 92.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T00:16:36.003 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T00:16:36.009 INFO:teuthology.orchestra.run.smithi111.stderr: 90.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T00:16:36.070 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T00:16:36.085 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T00:16:36.134 INFO:teuthology.orchestra.run.smithi111.stderr: 91.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T00:16:36.144 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-22T00:16:36.153 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T00:16:36.154 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T00:16:36.353 INFO:teuthology.orchestra.run.smithi038.stderr: 94.0% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T00:16:36.535 INFO:teuthology.orchestra.run.smithi203.stderr: 90.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T00:16:36.638 INFO:teuthology.orchestra.run.smithi203.stderr: 93.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T00:16:36.708 INFO:teuthology.orchestra.run.smithi203.stderr: 86.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T00:16:36.926 INFO:teuthology.orchestra.run.smithi203.stderr: 93.3% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T00:16:36.936 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T00:16:37.519 INFO:teuthology.orchestra.run.smithi111.stderr: 86.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T00:16:37.590 INFO:teuthology.orchestra.run.smithi203.stderr: 94.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T00:16:38.019 INFO:teuthology.orchestra.run.smithi111.stderr: 93.2% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-22T00:16:38.146 INFO:teuthology.orchestra.run.smithi203.stderr: 93.2% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T00:16:38.921 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T00:16:39.033 INFO:teuthology.orchestra.run.smithi038.stderr: 90.3% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T00:16:42.858 INFO:teuthology.orchestra.run.smithi111.stderr: 93.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T00:16:43.259 INFO:teuthology.orchestra.run.smithi111.stderr: 91.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T00:16:56.318 INFO:teuthology.orchestra.run.smithi038.stderr: 90.5% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T00:17:00.457 INFO:teuthology.orchestra.run.smithi111.stderr: 95.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T00:32:50.189 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T00:32:56.265 INFO:teuthology.orchestra.run.smithi203.stderr: 92.5% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T00:33:14.446 INFO:teuthology.orchestra.run.smithi111.stderr: 92.9% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T00:35:41.204 INFO:teuthology.orchestra.run.smithi111.stderr: 92.6% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T00:37:52.854 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T00:37:56.674 INFO:teuthology.orchestra.run.smithi111.stderr: 92.2% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T00:37:56.676 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-22T00:37:56.676 INFO:teuthology.orchestra.run.smithi111.stderr:real 21m20.971s 2024-04-22T00:37:56.677 INFO:teuthology.orchestra.run.smithi111.stderr:user 8m22.098s 2024-04-22T00:37:56.677 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m21.659s 2024-04-22T00:38:00.512 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T00:39:08.430 INFO:teuthology.orchestra.run.smithi203.stderr: 92.1% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T00:39:08.432 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-22T00:39:08.432 INFO:teuthology.orchestra.run.smithi203.stderr:real 22m32.727s 2024-04-22T00:39:08.432 INFO:teuthology.orchestra.run.smithi203.stderr:user 8m55.218s 2024-04-22T00:39:08.432 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m23.151s 2024-04-22T00:40:21.455 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T00:41:15.493 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T00:41:47.486 INFO:teuthology.orchestra.run.smithi038.stderr: 92.3% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T00:42:19.741 INFO:teuthology.orchestra.run.smithi038.stderr: 91.4% -- replaced with /var/log/ceph/eab6f840-0039-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T00:42:19.795 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-22T00:42:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:real 25m44.094s 2024-04-22T00:42:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:user 11m31.094s 2024-04-22T00:42:19.795 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m27.433s 2024-04-22T00:42:19.796 INFO:tasks.cephadm:Archiving logs... 2024-04-22T00:42:19.797 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi038/log 2024-04-22T00:42:19.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T00:44:04.433 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi111/log 2024-04-22T00:44:04.434 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T00:45:23.300 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi203/log 2024-04-22T00:45:23.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T00:46:50.296 INFO:tasks.cephadm:Removing cluster... 2024-04-22T00:46:50.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force 2024-04-22T00:46:50.859 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:46:51.748 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force 2024-04-22T00:46:52.326 INFO:teuthology.orchestra.run.smithi111.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:46:53.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eab6f840-0039-11ef-bc93-c7b262605968 --force 2024-04-22T00:46:53.525 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: eab6f840-0039-11ef-bc93-c7b262605968 2024-04-22T00:46:54.317 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T00:46:54.318 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T00:46:54.338 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T00:46:54.359 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T00:46:54.374 INFO:tasks.cephadm:Teardown complete 2024-04-22T00:46:54.386 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T00:46:54.407 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T00:46:54.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T00:46:54.410 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T00:46:54.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T00:46:54.613 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T00:46:54.625 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T00:46:54.625 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:46:54.628 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:46:54.631 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T00:46:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:46:54.644 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:46:54.645 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-22T00:46:54.673 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-22T00:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 +17us[ +17us] +/- 77ms 2024-04-22T00:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 +466us[ +490us] +/- 75ms 2024-04-22T00:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -389us[ -389us] +/- 69ms 2024-04-22T00:46:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +2820ns[ +19us] +/- 77ms 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 +456us[ +472us] +/- 75ms 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -406us[ -406us] +/- 69ms 2024-04-22T00:46:54.675 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 -32us[ -23us] +/- 77ms 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 +424us[ +433us] +/- 75ms 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -431us[ -431us] +/- 69ms 2024-04-22T00:46:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T00:46:54.725 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T00:46:54.736 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T00:46:54.736 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T00:46:54.746 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T00:46:54.915 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T00:46:55.012 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T00:46:55.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:46:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:55.140 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:46:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:55.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T00:46:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:55.198 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T00:46:55.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T00:46:55.273 INFO:teuthology.task.internal:Duration was 3779.195558 seconds 2024-04-22T00:46:55.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T00:46:55.300 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T00:46:55.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T00:46:55.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T00:46:55.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T00:46:55.348 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:46:55.367 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:46:55.401 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T00:46:55.832 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T00:46:55.832 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-22T00:46:55.832 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T00:46:55.871 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-22T00:46:55.871 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T00:46:55.900 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-22T00:46:55.900 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T00:46:55.936 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T00:46:55.936 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T00:46:55.939 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T00:46:55.943 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T00:46:56.002 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T00:46:56.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T00:46:56.005 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T00:46:56.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T00:46:56.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T00:46:56.435 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T00:46:56.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T00:46:56.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T00:46:56.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T00:46:56.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T00:46:56.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.554 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-22T00:46:56.588 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-22T00:46:56.591 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-22T00:46:56.613 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:56.628 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:56.642 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T00:46:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T00:46:56.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T00:46:56.679 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T00:46:56.681 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi038 2024-04-22T00:46:56.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T00:46:56.724 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi111 2024-04-22T00:46:56.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T00:46:56.760 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667643/remote/smithi203 2024-04-22T00:46:56.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T00:46:56.796 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T00:46:56.796 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T00:46:56.799 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T00:46:56.801 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T00:46:56.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T00:46:56.859 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T00:46:56.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T00:46:56.885 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T00:46:56.886 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T00:46:56.888 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T00:46:56.890 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T00:46:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 00:46 /home/ubuntu/cephtest 2024-04-22T00:46:56.902 INFO:teuthology.orchestra.run.smithi111.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 00:46 /home/ubuntu/cephtest 2024-04-22T00:46:56.904 INFO:teuthology.orchestra.run.smithi203.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 00:46 /home/ubuntu/cephtest 2024-04-22T00:46:56.905 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T00:46:56.917 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T00:46:56.944 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 3779.1955575942993 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-22T00:46:56.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T00:46:57.029 INFO:teuthology.run:pass